Towards Proactive Social Robots: Distilling Visual Knowledge from Large Vision-Language Models

Abstract

Proactive behaviors are crucial for enhancing user engagement in Human-Robot Interaction (HRI), yet their implementation requires robots to generate context-aware utterances grounded in perceptual input. While recent advances in Vision-Language Models (VLMs) and Large Language Models (LLMs) have significantly improved multimodal understanding and generation capabilities, their integration into social robotics remains limited. This is primarily due to the challenges of producing coherent, human-like dialogue under real-time and computational constraints, as well as the high cost and effort associated with collecting large-scale annotated HRI datasets. To address these challenges, we propose a novel training pipeline that employs Knowledge Distillation to automatically generate task-specific annotations from pre-trained VLMs and LLMs. The method processes video data from the MuMMER and AVDIAR datasets using a VLM to generate descriptive captions, which are subsequently refined by an LLM to produce an augmented set of context-aware sentences. In addition, we present a lightweight architecture based on LLaVA-One Vision, optimized for real-time sentence generation in social robotics contexts. We evaluate the proposed system both quantitatively and qualitatively through a user study, and further assess its deployment feasibility on embedded platforms. Results demonstrate the model's capability to produce syntactically correct and semantically appropriate utterances with minimal computational overhead, high-lighting its potential for deployment in real-world social robotics applications.