Wearable Sensors-Based Hand Gesture Recognition for Human–Robot Collaboration in Construction

Abstract

The development of robotic machines has shown the potential to promote automation in construction. One of the critical enablers of human–robot collaboration is a user-friendly interface to support their interactions. Compared with other interfaces, hand gesture is an effective communication channel on construction sites. This article proposes a system for recognition of construction workers’ hand gestures using wearable sensors on fingers. The system starts with synchronizing, normalizing, and smoothing finger motions. Then, the motion data are extracted through a sliding window and fed into an enhanced fully convolutional neural network (FCN) for the hand gesture recognition. The system was tested through a system validation test and achieved the precision and recall of 85.7% and 93.8%, respectively. A pilot study demonstrated the use of the proposed system to interact with a robotic dump truck. The system was further compared with vision-based recognition methods to quantitatively and qualitatively assess their relative benefits and limitations.