Development and Mobile Deployment of a Stair Recognition System for Human-Robot Locomotion

Abstract

Abstract Environment sensing and recognition can improve the safety and autonomy of human-robot locomotion, especially during transitions between environmental states such as walking to and from stairs. However, accurate and real-time perception on edge devices with limited computational resources is an open problem. Here we present the development and mobile deployment of StairNet - a vision-based automated stair recognition system powered by deep learning. Building on ExoNet - the largest open-source dataset of egocentric images of real-world walking environments - we designed a new dataset specifically for stair recognition with over 515,000 images. We then developed a lightweight and efficient convolutional neural network for image classification, which accurately predicted complex stair environments with 98.4% accuracy. We also studied different model compression and optimization methods and deployed our system on several mobile devices running a custom-designed iOS application with onboard accelerators using CPU, GPU, and/or NPU backend computing. Of the designs that we tested, our highest performing system showed negligible reductions in classification accuracy due to the model conversion for mobile deployment and achieved an inference time of 2.75 ms on an iPhone 11. The high speed and accuracy of the StairNet system on edge devices opens new opportunities for autonomous control and planning of robotic prosthetic legs, exoskeletons, and other assistive technologies for human locomotion.