Lightweight and Optimization Acceleration Methods for Vision Transformer: A Review

摘要

With the rapid development of technologies such as smart home, smart medical and autonomous driving, lightweight networks play an important role in promoting the application of deep learning technology on mobile and embedded terminals. The Transformer-based model changes the architecture of traditional neural networks, and performs outstanding in many fields such as natural language processing and computer vision. However, the huge computing cost and the increasing network scale have increased the demand for storage, running memory and computing, which hinders their widespread deployment on various hardware devices, such as mobile phones, robots and Internet of Things (IoT) devices. Therefore, compression method for Transformer-based models need to be explored so that they can be applied more widely on mobile devices. In this paper, we focus on the lightweight and optimization acceleration methods on Vision Transformer in recent years, and summarize them as quantization, knowledge distillation, pruning and adjusting network structure, the innovations, merits and demerits of these approaches have been compared and reviewed. Through this survey, it is hoped to provide useful help for the current compression method for Vision Transformer, and also expected to point out a direction for future research in this field.