A 94Hz Inference and 7.4mJ/Epoch Fine-Tune Edge SoC for Diffusion-Based Robot Manipulation with Speculation and Disturbance Enhancement

Abstract

We present an edge SoC for diffusion-transformer-based action generation (DiTAG) in robot manipulation featuring both low-latency inference and high-fidelity on-device finetune. The substantial challenges of their edge acceleration are overcome by a speculative parallel inference and disturbanceenhanced low-bit fine-tune architecture. The 28nm prototype integrating the quad-core accelerator with CPU shows 10.6ms inference, <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$36.8 \times$</tex> better than an edge GPU, with 7.88TOPS/W system energy efficiency, and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$7.4 \text{mJ} /$</tex> epoch on-device finetune with minimal accuracy loss at normal voltage.