Multimodal system in human‒machine interaction: A systematic review and bibliometric analysis of design approaches and user feedback

Abstract

Multimodal systems significantly enhance human–machine interaction (HMI) by integrating various feedback modalities such as auditory, tactile, gestural, and haptic signals, thereby improving responsiveness, intuitiveness, and overall user experience. These systems enable more natural communication between users and machines. However, current implementations still face substantial challenges, including the real-time processing of multimodal feedback, effective cognitive workload management, and the seamless integration of diverse sensory inputs. This study presents a comprehensive systematic literature review (SLR) and bibliometric analysis of 70 peer-reviewed articles published between 2019 and 2024, sourced from the Scopus database. The PRISMA framework guided the article selection process, while Biblioshiny was used to generate thematic maps and visualize research trends. The analysis identified five major thematic clusters: (1) machine learning-enhanced multimodal interfaces, (2) emotion and neurophysiological state detection, (3) deep learning-based multimodal systems, (4) human–robot interaction, and (5) cognitive workload adaptability. The distribution of research across application domains reveals a strong focus on societal contexts (64.28%), followed by medical (24.29%) and transportation (11.43%) sectors. Although AI-driven feedback systems are becoming more prevalent, issues such as latency, algorithmic bias, and lack of personalization still hinder optimal user interaction. This study contributes to the understanding of the current research landscape in multimodal HMI, identifies prevailing trends and gaps, and offers insights into future directions. Specifically, it calls for further research on real-time adaptive interfaces, personalized and context-aware multimodal feedback, and the integration of ethical frameworks in the development of AI-powered HMI systems. These directions are crucial to building more intelligent, responsive, and human-centric interactive systems.