Enhancing Robustness in Language-Driven Robotics: A Modular Approach to Failure Reduction

Abstract

Recent advances in large language models (LLMs) have led to significant progress in robotics, enabling embodied agents to understand and execute open-ended tasks. However, existing LLM-based approaches face limitations in grounding their outputs within the physical environment and aligning with the capabilities of the robot. While fine-tuning is an attractive approach to addressing these issues, the required data can be expensive to collect, especially when using very large language models. Smaller language models, while more computationally efficient, are less robust in task planning and execution, leading to a difficult trade-off between performance and tractability. In this paper, we present a novel, modular architecture designed to enhance the robustness of locally-executable LLMs in the context of robotics by addressing these grounding and alignment issues. We formalize the task planning problem within a goal-conditioned POMDP framework, identify key failure modes in LLM-driven planning, and propose targeted design principles to mitigate these issues. Our architecture introduces an "expected outcomes" module to prevent mischaracterization of subgoals and a feedback mechanism to enable real-time error recovery. Experimental results, both in simulation and on physical robots, demonstrate that our approach leads to significant improvements in success rates for pick-and-place and manipulation tasks, surpassing baselines using larger models. Through hardware experiments, we also demonstrate how our architecture can be run efficiently and locally. This work highlights the potential of smaller, locally-executable LLMs in robotics and provides a scalable, efficient solution for robust task execution and data collection.<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>