VR-Hand-in-Hand: Using Virtual Reality (VR) Hand Tracking For Hand-Object Data Annotation

Abstract

We demonstrate VR-Hand-in-Hand, a virtual reality (VR) environment that enables annotators to conduct ground truth annotation of complex hand-object interaction datasets using intuitive hand movements via hand-tracking built into hands-free VR head mounted devices (HMDs). Hand-object datasets annotated with articulated hand pose are of importance for tasks such artificial intelligence (AI)-driven hand pose detection and informing robots on where to grasp for safe human-robot interaction (HRI). Though markerless capture provides the benefit of natural unhindered interaction in comparison to markers, self-occlusions and obscuration of the hand by objects makes at-scale annotation of markerless datasets difficult to perform. VR-Hand-in-Hand eases hand-object annotation task by immersing annotators in an intuitive environment, enabling them to embody captured subjects, align their hand with the subjects’ hand like a glove, and save out the joint points forming the hand skeleton. By providing an intuitive annotation environment, VR-Hand-in-Hand will greatly expand markerless capture of multi-person hand interactions, benefiting human collaborative behavior study in fields such as psychology, physical therapy, and HRI.