Personalized Video-Based Hand Taxonomy Using Egocentric Video in the Wild

Abstract

OBJECTIVE: Hand function is central to inter- actions with our environment. Developing a comprehen- sive model of hand grasps in naturalistic environments is crucial across various disciplines, including robotics, ergonomics, and rehabilitation. Creating such a taxonomy poses challenges due to the significant variation in grasping strategies that individuals may employ. For instance, individuals with impaired hands, such as those with spinal cord injuries (SCI), may develop unique grasps not used by unimpaired individuals. These grasping techniques may differ from person to person, influenced by variable senso- rimotor impairment, creating a need for personalized meth- ods of analysis. METHOD: This study aimed to automatically identify the dominant distinct hand grasps for each indi- vidual without reliance on a priori taxonomies, by applying semantic clustering to egocentric video. Egocentric video recordings collected in the homes of 19 individual with cervical SCI were used to cluster grasping actions with semantic significance. A deep learning model integrating posture and appearance data was employed to create a personalized hand taxonomy. RESULTS: Quantitative analysis reveals a cluster purity of 67.6% ± 24.2% with 18.0% ± 21.8% redundancy. Qualitative assessment revealed meaningful clusters in video content. DISCUSSION: This methodology provides a flexible and effective strategy to analyze hand function in the wild, with applications in clinical assess- ment and in-depth characterization of human-environment interactions in a variety of contexts.