Mobile Robotic Manipulation for Search-and-Fetch Tasks by Integrating Human-Robot Interaction

Abstract

The increasing labor shortages in commercial and industrial sectors have driven the development of mobile manipulation robots, which combine the mobility of mobile bases with the dexterity of robotic arms to automate complex tasks. However, integrating heterogeneous subsystems, such as perception, navigation, and manipulation, into a cohesive workflow remains an open challenge. This paper presents a unified mobile manipulation system capable of autonomous search-and-fetch operations by seamlessly integrating real-time mapping, path planning, object detection, and grasping. To enhance human-robot interaction, the system incorporates multimodal control via face recognition, gesture recognition, and voice commands, allowing intuitive user-directed operation. Control Lyapunov Functions and Control Barrier Functions are integrated in a hierarchical motion controller to mitigate tracking error and establish safety constraint. Meanwhile, a hybrid learning-optimization grasping algorithm is implemented to generate robust and precise grasping poses for target object manipulation by leveraging convolutional neural networks and gripper kinematic constraints. Experimental results demonstrate the system's effectiveness for indoor search-and-fetch applications, validating its potential to address labor shortages through reliable automation.