ViTaSCOPE: Visuo-tactile Implicit Representation for In-hand Pose and Extrinsic Contact Estimation

Abstract

Fig. 1: ViTaSCOPE: Visuo-Tactile Simultaneous Contact and Object Pose Estimation.We present an object-centric neural implicit representation that enables simultaneous in-hand object pose and extrinsic contact estimation from vision and highresolution tactile sensing.Our method is trained entirely in simulation and zero-shot transferred to the real world.In the example above, the tool grasped by the robot makes extrinsic contact with the table.ViTaSCOPE is able to infer the in-hand object pose and register the extrinsic contact (patch highlighted in blue) to the 3D geometry from partial real-world observations.Abstract-Mastering dexterous, contact-rich object manipulation demands precise estimation of both in-hand object poses and external contact locations-tasks particularly challenging due to partial and noisy observations.We present ViTaSCOPE: Visuo-Tactile Simultaneous Contact and Object Pose Estimation, an object-centric neural implicit representation that fuses vision and high-resolution tactile feedback.By representing objects as signed distance fields and distributed tactile feedback as neural shear fields, ViTaSCOPE accurately localizes objects and registers extrinsic contacts onto their 3D geometry as contact fields.Our method enables seamless reasoning over complementary visuotactile cues by leveraging simulation for scalable training and zero-shot transfers to the real-world by bridging the sim-to-real gap.We evaluate our method through comprehensive simulated and real-world experiments, demonstrating its capabilities in dexterous manipulation scenarios.