AI-Powered Smart Glasses for Sensing and Recognition of Human-Robot Walking Environments

摘要

Environment sensing and recognition can allow hu-mans and/or robotic systems to dynamically adapt to different walking terrains. However., fast yet accurate visual perception is challenging., especially on embedded systems with limited computational resources. The purpose of this study was to develop and prototype a new pair of integrated AI-powered smart glasses for onboard sensing and recognition of human-robot walking en-vironments with high accuracy and low latency. Our system in-cludes a Raspberry Pi Pico micro controller and an ArduCam low-power camera., both of which interface with commercial eye-glass frames via 3D-printed mounts that we custom-designed. We trained and optimized a lightweight and efficient convolutional neural network using a MobileN etVI backbone to classify real-world walking terrains as either indoor surfaces., outdoor surfaces (grass and dirt)., or outdoor surfaces (paved) using over 62,500 egocentric images that we adapted and manually labelled from the Meta Eg04D dataset. We compiled and deployed our deep learning model using TensorFlow Lite micro and post-training quantization to create a minimized byte array model of size 0.31MB. Our system was able to accurately classify complex walking environments with 93.6% accuracy and an embedded inference speed of 1.5 seconds during online experiments. These AI-powered smart glasses open new opportunities for visual per-ception of human-robot walking environments where real-time embedded computing is desired. Future research will focus on improving the onboard inference speed and further miniaturization of the mechatronic components.