An AI‐Enabled All‐In‐One Visual, Proximity, and Tactile Perception Multimodal Sensor

Abstract

Interactive robotic tasks heavily rely on comprehensive environment perception. Particularly, visual, proximity, and tactile perception are essential sensing modalities providing comprehensive information in such scenarios. However, integrating different sensors poses several challenges, including increased volume and cost, difficulties with signal synchronization and alignment, and multisensor cross‐interference or signal disruption. To tackle these challenges, we propose the vision‐proximity‐tactility sensor (VPTS), an AI‐enabled, all‐in‐one multimodal sensor designed for holistic perception through efficient collaboration and information transfer between modalities, enabling complex, long‐sequence robotic interactions. Facilitated by a transparent membrane patterned with ultraviolet (UV)‐excited fluorescent markers, VPTS utilizes a focus‐adjustable monocular camera to achieve visual, proximity, and tactile perception modalities in a time‐division mode. It switches modalities by toggling UV light, camera focus, and corresponding dedicated three deep learning models. VPTS achieves a benchmark score ( F 1, that comprehensively measures both precision and recall) of 0.9733 in visual perception, 5.098 mm mean absolute error in proximity estimation, and 0.653 mN root–mean square error in force sensing. Real‐world experiments, such as a computer music game involving up to 28 consecutive subtasks, show a cohesive pipeline where modalities collaboratively support such long‐sequence manipulations, verifying VPTS's effectiveness for intricate, multimodal interactive tasks.