A Lightweight Spatio-Temporal Skeleton Attention Transformer for Long-Distance Gesture Recognition in UAV Control

摘要

Gesture recognition constitutes a core modality for human–robot interaction (HRI). Current studies primarily address short-distance scenarios, whereas long-distance outdoor scenarios remain significantly underexplored. The principal technical challenges arise from three key challenges: First, the predominant separated spatio-temporal processing approach in current network architectures and attention mechanisms. 1) introduces redundant structural complexity and computational demands, severely constraining unmanned aerial vehicle (UAV) platform implementation. 2) Inherently limits the model’s ability to learn weakly-correlated spatio-temporal relationships. 3) Additionally, existing HRI systems demonstrate inherent deficiencies in accurate hand localization and skeleton estimation under long-distance conditions. To address these issues, we propose a lightweight spatio-temporal skeleton (STS) attention transformer for long-distance gesture recognition in UAV control. First, we designed and implemented a lightweight STS Transformer to unify spatial-temporal feature processing while maintaining structural simplicity. Second, we developed an STS attention employing coupled-decoupled operators on skeleton sequences to extract weakly-correlated features. Subsequently, we developed a two-stage RGB+D framework for hand detection and skeleton estimation, achieving capability at extended ranges up to 5 m under UAV onboard camera view and 10 m in third-party camera view. Finally, extensive experiments on two datasets and real-world experiments were conducted to demonstrate the effectiveness of our network in skeleton-based gesture recognition, showing a clear advantage over the state-of-the-art (SOTA) methods.