Enhancing Household Service Robots with a Dual-Arm Mobile Manipulator and Multimodal Large Language Models

Abstract

This research aims to enhance the practicality of household service robots by integrating multimodal large language models (MLLMs). Utilizing a dual-arm mobile manipulator platform, our focus is on advancing autonomous mobile manipulation in home environments. By leveraging the advanced text understanding and visual processing capabilities inherent in MLLMs, the proposed system adeptly converts user dialogues and environmental visual inputs into precise robotic plans. Among the suite of tested models, GPT-4o mini and Gemini 1.5 flash perform exceptionally well in generating accurate robotic plans. These results underscore a significant enhancement in the robot's ability to manipulate objects within complex home environments, thereby contributing to more seamless human-robot interactions and markedly improved task execution. We anticipate that our findings will inspire future advances in the field of human-robot interaction for home robots.