Liquid classification in robotic fingers with multimodal tactile sensor system

Abstract

Recent robotics advancements have accelerated the development of intelligent manufacturing systems. Therein, object recognition with visual sensing plays a critical role. However, liquid classification in containers remains an untapped region due to the optical transparency of liquids and the similarity of their tactile properties. Here, we introduce a multimodal tactile sensor system on robotic fingers with machine learning for classifying liquids in bottles. Specifically, the sensor system integrates perceptions of thermal conduction and frequency response, inspired by human thermal and vibrational receptors. Additionally, a convolutional neural network with dual parallel structure is employed to process the multimodal input. The multimodal system demonstrates two different classification tasks: classifying water volumes and liquid types. The proposed model achieves high classification accuracy in both tasks and enables real-time operation. This biologically inspired approach offers a solution for liquid classification using static contact, with broad application in robotic perception.