Grounding Large Language Models for Robot Task Planning Using Closed‐Loop State Feedback

Abstract

Planning algorithms break complex problems into sequential steps for robots. Recent work employs Large Language Models (LLMs) to generate robot policies directly from natural language in simulation and real‐world settings. Models such as GPT‐5 generalize to unseen tasks but often hallucinate because they lack sufficient environmental grounding; supplying state feedback improves robustness. We introduce a task‐planning method that uses two LLMs—one for high‐level planning and one for low‐level control—thereby raising task success rates and goal‐condition recall. Our algorithm, BrainBody‐LLM, is inspired by the human neural system, dividing planning hierarchically across the two LLMs and closing the loop with feedback that learns from simulator errors to fix execution failures. Implemented with GPT‐5, BrainBody‐LLM improves task‐oriented success in the VirtualHome environment by 17% over competitive baselines. We also evaluate seven complex tasks in a realistic physics simulator and on a Franka Research 3 robotic arm, comparing our approach with other state‐of‐the‐art LLM planners. Results show that recent LLMs can use raw simulator or controller errors to revise plans, yielding more reliable robotic task execution. Project resources are available here .