Agreeing to Interact in Human-Robot Interaction using Large Language Models and Vision Language Models

Abstract

In human-robot interaction (HRI), the beginning of an interaction is often complex. Whether the robot should communicate with the human is dependent on several situational factors (e.g., the current human’s activity, urgency of the interaction, etc.). We test whether large language models (LLM) and vision language models (VLM) can provide solutions to this problem. We compare four different system-design patterns using LLMs and VLMs, and test on a test set containing 84 human-robot situations. The test set mixes several publicly available datasets and also includes situations where the appropriate action to take is open-ended. Our results using the GPT-4o and Phi-3 Vision model indicate that LLMs and VLMs are capable of handling interaction beginnings when the desired actions are clear. The design using direct image input scored an 89% accuracy on the test set. Of the designs using indirect input, a combined text about human activity and gaze performed best with a 90% accuracy. However, challenges remain in the open-ended situations where the model must choose the priority between the human and robot situation. The design using direct image input mostly prioritized the robot situation, whereas the design with best performance using indirect input mostly prioritized the human situation. Such one-sided behavior could be crucial for practical HRI applications.