A low-hardware-overhead, high-energy-efficiency, and end-to-end CNN-based feature extraction accelerator for mobile visual SLAM

Abstract

Feature extraction is a crucial step of visual simultaneous localization and mapping (SLAM). The newly emerging SuperPoint feature extraction method based on Convolutional Neural Network (CNN), shows significant accuracy improvement compared to traditional methods. However, due to the high computational complexity, it is difficult to apply SuperPoint on real-time mobile devices limited by battery power and hardware resources, such as small and micro mobile robots. In this paper, a low-hardware-overhead, high-energy-efficiency, and end-to-end CNN-based feature extraction accelerator is proposed. Several design strategies including a lightweight SuperPoint network with an optimum reduced number of filters, a dedicated end-to-end fully pipelined hardware architecture, a descriptor selective generator, and a speed control scheme are proposed to reduce hardware overhead and improve energy efficiency while maintaining comparable accuracy. FPGA evaluation results show that the FPGA hardware resources have been reduced by between 39.5% and $89.0 \%$ of the existing designs. Besides, the proposed accelerator can process 20 -fps $480 \times 640$ images at 200 MHz, with an energy efficiency of $58.2 \mathrm{~mJ} /$ frame.