An AR-assisted Human-Robot Interaction System for Improving LLM-based Robot Control

Abstract

The LLM-based robot control system enables robots to understand and execute high-level language instructions. However, the motion plans generated by the LLM-based robot control system are not always reliable. To allow operators to monitor the working state of the robot after issuing task instructions in high-level language, and demonstrate the robot’s motion plan safely, we designed an AR-assisted human-robot interaction system called SeeIt. This system displays the working state of a robotic arm and predicts its motion trajectory regarding LLM output. Ambiguities in high-level language instructions and perception misunderstanding about the surrounding environment may lead to erroneous motion plans. The AR interaction system allows operators to choose among potential running trajectories and to adjust erroneous motion plans, leveraging human decision-making to enhance the system’s intelligence. A case study was conducted to evaluate the usability and performance of this system. We expected that SeeIt will increase the interaction experience between operator and robot, improving the task performance of the LLM-based robot control system.