Design Exploration of Fault-Tolerant Deep Neural Networks Using Posit Number Representation System

Abstract

The applications of deep neural networks (DNNs) in different safety-critical systems (such as autonomous vehicles and robotics) are experiencing emerging growth due to their high accuracy and potential for solving complex problems. However, a single failure in the hardware performing these DNN models can lead to irreparable results. Thus, improving the resilience of these models to transient faults (i.e., soft errors) has been of great interest in recent years. However, the traditional hardware redundancy techniques (such as TMR) are not cost-efficient due to their high resource overheads. In this article, we explored the potential of leveraging the posit number representation system for composing the DNN models, to achieve higher fault tolerance compared to the conventional fixed-point and IEEE 754 32-bit floating-point (FLOAT) number representation-based DNNs, without incurring significant overheads of the traditional hardware redundancy techniques. Posit numbers are composed of four fields, including a sign bit, regime value, exponent, and fraction bits, where the regime value does not exist in the FLOAT numbers. Our explorations are performed at the model, layer, and bitwise levels to determine the most appropriate posit format (e.g., the bit width of different posit fields) for composing the fault-tolerant DNN models. We studied the different posit, fixed-point, and FLOAT-based LeNet-5 and ResNet-50 networks trained with the MNIST and CIFAR-10 datasets, respectively. We then proposed a hardware-level method for error detection and correction of posit-based DNN models. Based on the results, the fault tolerance of the posit-based DNNs outperforms the FLOAT-based models, at each of the three investigated levels, where a 32-bit posit-based DNN achieved up to 15% more classification accuracy than the FLOAT-based one, for the LeNet-5 network. Also, in comparison with the fixed-point models, the posit-based networks showed up to 23% higher accuracy. Moreover, the enhanced 8-bit posit-based DNN that employed the proposed error detection and correction method results in, up to 11% and 41% higher classification accuracy than the unprotected posit and conventional FLOAT-based DNNs, respectively.