Integration of Multilingual NLP with 7-DOF Robotic Manipulation for Enhanced Domestic Tasks

Abstract

This work explores the integration of multilingual natural language processing (NLP) with robotic manipulation for household tasks, focusing on beverage preparation using a 7-DOF robotic arm, the Franka Emika Panda. By leveraging Large Language Models (LLMs) such as GPT-4, we developed a novel framework that converts English and German instructions into actionable robotic commands through iterative prompt refinement and a few-shot learning strategy in a Python-based environment. The system was successfully integrated with the robot's control platform, achieving an overall task accuracy of 83% and demonstrating a minor 3% gap between English and German inputs. While no false negatives were observed, challenges were identified in the sequencing of complex operations. A proof-of-concept demonstration, including the preparation of coffee and tea, showcased the potential of this approach for enhancing intuitive human-robot interactions in domestic settings and highlighted areas for further optimisation. This research demonstrates the feasibility of LLM-driven robotics for practical household applications, paving the way for more sophisticated human-robot collaborations.