Exploring Foundation Models in Detecting Concerning Daily Functioning in Psychotherapeutic Context Based on Images from Smart Home Devices

Abstract

The surge of cyber-physical systems (CPS) and smart home devices (e.g., vacuum robots and pet cameras) equipped in U.S. households opens up the potential to screen the day-to-day functioning of individuals in the smart home environment and to provide precautionary assistance to individuals who may need psychotherapies. Meanwhile, recent advances in foundation models (FMs) enable the large language models (LLMs) and vision-language models (VLMs) to have strong reasoning capabilities even in complex scenarios. In this paper, we investigate the integration of FMs and photos taken from the perspective of vacuum robots to screen the behaviors indicative of mental state that need further attention from therapists and health care practitioners. Specifically, we explore the possibility of using VLMs and LLM-based reasoners to accurately detect two of the most concerning behaviors at home: smoking- and drinking-alone. Compared to existing methods based on object detection, we demonstrate that the integration of LLMs and VLMs can significantly enhance detection accuracy, especially in complex home environments with ambiguous patterns and distinguishing concerning events from benign events. We showcase the potential of employing FMs in CPS to discern nuanced insights into day-to-day functioning behaviors in the psychotherapeutic contexts.