Local Observation Based Reactive Temporal Logic Planning of Human-Robot Systems

Abstract

Human-robot collaboration plays an important role in intelligent manufacturing. However, the main challenge is how the robot can make online reactive changes to the plan based on the observed human behavior to ensure the completion of user-defined tasks. Such a challenge is further exacerbated if eye-in-hand manipulation is considered since the local field of the camera view cannot capture global observations. Different from existing planning approaches that separate the perception and planning modules, and make strong assumptions about perception abilities, we develop a framework of real-time local reactive planning that enables the robot to quickly adapt its actions if necessary through its limited perception of surroundings using an eye-in-hand camera. Specifically, we develop a locally observable transition system (LOTS) and interpretably express the task using linear temporal logic (LTL). To improve the grasping performance using local visual perception, we propose a high-resolution grasp network (HRG-Net) that achieves state-of-the-art results on multiple datasets (99.50% in Cornell and 97.50% in Jacquard and 96% in Graspnet-1Billion) for the task. A physical experiment using a 7DoF Franka Emika Panda robot demonstrates the effectiveness of the reactive planning framework. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —Intelligent manufacturing often requires the human operator to work collaboratively with the robot in a shared workspace. Due to possible (assistive or non-assistive) interference of human operators, it is highly desired that the robot can perceive human behaviors and react properly to ensure task accomplishment. Hence, this work is particularly motivated to develop a reactive planning framework that relies on real-time local visual perception (i.e., eye-in-hand camera) to quickly react to its dynamic surroundings and replan its motion when necessary. In future work, rather than using the observed human behavior, we will investigate how to predict human intentions to further improve human-robot collaboration.