Instruction-Following Long-Horizon Manipulation by LLM-Empowered Symbolic Planner

Abstract

One of the grand goals in robotics is to develop a versatile robot capable of executing a wide array of tasks based on language instructions. These tasks often involve long-horizon manipulation due to the complexity of the real-world tasks and the high-dimensional action space of the dual-arm mobile robots. This work proposes a generalizable framework that combines a Large Language Model, a Visual-language Model, and symbolic planning to address long-horizon bimanual mobile manipulation tasks. We use the Large Language Model as the commonsense reasoner to break down the task into a set of subgoals and generate goal-oriented state queries. Subsequently, we fine-tune a Visual-language Model to obtain answers to queries. Finally, we propose a Planning Domain Definition Language translator that translates the subgoal and the goal-oriented states into a symbolic planning problem that can be solved by search-based planners. The experiments show that the proposed method has generalization ability in novel objects and instruction. It outperforms the baseline methods across all the configurations. Moreover, our approach exhibits competence in increasingly complex tasks, encompassing those that demand upward of 10 planning steps.