🤖 machine learning

Neural Network Training with Approximate Logarithmic Computations

This paper proposes an end-to-end deep neural network training and inference scheme using approximate logarithmic number system computations with fixed-point arithmetic, demonstrating that 16-bit log-domain processing achieves classification accuracy within approximately 1% of floating-point baselines while eliminating the need for hardware multipliers.

Original authors: Arnab Sanyal, Peter A. Beerel, Keith M. Chugg

Published 2019-10-22

📖 4 min read☕ Coffee break read

Original authors: Arnab Sanyal, Peter A. Beerel, Keith M. Chugg

✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern world, computers have become remarkably good at recognizing patterns, from identifying a cat in a photograph to understanding spoken words. This ability comes from a type of software called a neural network, which is designed to mimic the way the human brain processes information by connecting many simple processing units together. To make these networks smart, they must be trained using vast amounts of data, a process that requires the computer to perform billions of complex calculations. Traditionally, these calculations rely heavily on multiplication, an operation that is mathematically straightforward but requires significant energy and specialized hardware to perform quickly. As we try to run these intelligent systems on smaller, battery-powered devices like smartphones or sensors, the heavy demand for multiplication becomes a bottleneck, draining power and slowing down the learning process.

A team of researchers at the University of Southern California has proposed a different way to handle these calculations, one that could make training these networks much more efficient. Instead of performing the standard multiplication required by traditional methods, they suggest converting all the numbers into a logarithmic format. In this system, the difficult task of multiplying two numbers is replaced by the much simpler task of adding them together. While this conversion sounds promising, it introduces a new challenge: adding numbers in this logarithmic format is not as simple as standard addition and usually requires complex look-up tables or approximations to work correctly. The researchers set out to see if they could simplify these additions even further, using rough approximations that are easy for hardware to build, without ruining the accuracy of the final result.

The researchers developed a complete system where the entire training and testing process of a neural network happens within this logarithmic world. They replaced the standard multiplication steps with addition and handled the tricky parts of logarithmic addition using two specific shortcuts. One shortcut uses a small table of pre-calculated values, similar to a cheat sheet, while the other uses a simple bit-shifting technique, which is a fast way for computers to multiply by powers of two. To test their idea, they built a digital simulation of a neural network and trained it on several well-known image datasets, including pictures of handwritten digits and clothing items. They compared their new logarithmic method against the standard, high-precision floating-point method that is currently used in most powerful computers.

The results were surprisingly encouraging. When the researchers used a 16-bit representation for their numbers, the logarithmic method achieved classification accuracy that was within about one percent of the standard floating-point method. This means that despite using simplified math and approximations, the network learned to recognize images almost as well as the traditional system. They found that even with a very small look-up table containing only twenty entries, the system performed well, and in many cases, the simple bit-shifting shortcut was sufficient to maintain high accuracy. The study suggests that for many common tasks, a 16-bit fixed-point representation in the logarithmic domain is enough to approach the performance of much more complex floating-point computations.

This work demonstrates that it is possible to train deep neural networks without relying on expensive multiplication hardware, provided that the addition operations are handled with these specific approximations. The researchers showed that the degradation in accuracy is minimal, suggesting that future hardware designs could potentially eliminate multipliers entirely for training and inference tasks. This would lead to devices that are smaller, faster, and more energy-efficient, capable of learning and adapting directly on the edge without needing to send data back to a large server. While the study focused on specific types of networks and datasets, the findings indicate a viable path toward making advanced artificial intelligence more accessible and practical for everyday devices.

Technical Summary: Neural Network Training with Approximate Logarithmic Computations

Problem Statement
The high computational complexity associated with training deep neural networks (DNNs) restricts the feasibility of online and real-time training on edge devices. While Graphics Processing Units (GPUs) provide massive hardware multiply-accumulate (MAC) accelerators, they are general-purpose and may not utilize hardware efficiently for DNN processing. Custom hardware accelerators offer a path to improved speed and energy efficiency through co-design, but the dominance of multiplier circuitry in MAC units remains a significant bottleneck. Existing complexity reduction methods often focus on sparsity, pruning, or quantization, but there is a need for approaches that fundamentally reduce the complexity of MAC units themselves.

Methodology
This paper proposes an end-to-end training and inference scheme for DNNs that eliminates multiplications by performing all computations in the log-domain using a Logarithmic Number System (LNS). In LNS, a real number $v$ is represented by the logarithm of its absolute value ( $V = \log_2 |v|$ ) and its sign ( $s_v$ ). This transformation converts linear-domain multiplications into log-domain additions.

The primary challenge in LNS is the log-domain addition operation, which requires calculating complex $\Delta$ terms ( $\Delta_+$ and $\Delta_-$ ) to handle the sum of logarithms. To address this, the authors propose low-complexity approximations inspired by hardware-friendly techniques:

Look-Up Tables (LUTs): Pre-computed tables for $\Delta_\pm(d)$ where $d$ is the difference between log-values.
Bit-Shift Approximations: Generalizing a previous bit-shift method to handle signed arithmetic. These approximations are shown to be special cases of LUTs with specific resolutions.

The training procedure is implemented entirely in the log-domain using fixed-point data representations. The authors adapt standard DNN components to this domain:

Activation Functions: The leaky-ReLU is translated to a log-leaky ReLU (llReLU).
Weight Initialization: Symmetric distributions are converted to the log-domain, where the sign becomes a Bernoulli variable.
Soft-max: The soft-max layer and cross-entropy cost are reformulated for log-domain operations.
Fixed-Point Analysis: The paper analyzes the bit-width requirements, noting that while worst-case analysis suggests log-domain representations require more bits than linear-domain equivalents, empirical results suggest they can be comparable.

Key Contributions

End-to-End Log-Domain Training: The authors demonstrate a complete training and inference pipeline for DNNs using approximate LNS computations, removing the need for multipliers.
Generalized Bit-Shift Approximations: The paper extends existing bit-shift approximations to handle signed arithmetic and establishes their equivalence to specific LUT configurations.
Fixed-Point Implementation: A comprehensive framework for implementing DNN training with fixed-point LNS, including specific adaptations for activation functions, weight initialization, and soft-max layers.

Experimental Results
The authors evaluated the proposed method on four datasets (MNIST, Fashion-MNIST, EMNIST-Digits, and EMNIST-Letters) using a Multi-Layer Perceptron (MLP) architecture. They compared 12-bit and 16-bit fixed-point implementations in both the linear domain and the log-domain (using both LUT and bit-shift approximations) against floating-point baselines.

Accuracy: The 16-bit log-based training achieved classification accuracy within approximately 1% of the equivalent floating-point baselines across the tested datasets.
Approximation Sensitivity: The log-domain soft-max layer was found to be more sensitive to approximation errors than other operations, requiring a higher-resolution LUT (640 elements) compared to the standard operations (20 elements).
Bit-Shift Performance: Simple bit-shift approximations, which can be viewed as equivalent to smaller LUTs, provided good classification performance in many cases, though LUTs generally yielded slightly better results.
Precision: 16-bit fixed-point representations in the log-domain were sufficient to approach the accuracy of floating-point computations, with 12-bit implementations showing slightly higher degradation.

Significance and Claims
The paper claims that the proposed approach allows for hardware implementations of DNN training and inference without multipliers, potentially significantly reducing implementation complexity and energy consumption. The authors conclude that:

Approximating log-domain addition using a max operation, an adder, and a LUT-based approximation for the $\Delta$ -term results in only modest degradation in classification accuracy compared to linear processing.
16-bit fixed-point representations are sufficient to achieve performance comparable to floating-point computations.
A LUT of size 20 is sufficient for most operations, and simple bit-shift approximations are viable alternatives in many scenarios.

The authors maintain a modest stance regarding practical deployment, noting that for the approach to be desirable, the circuit implementation complexity of the approximate log-domain adder must be significantly lower than that of a multiplier. They identify future research directions as applying the method to larger convolutional neural networks and co-optimizing $\Delta$ -term approximations for both accuracy and hardware complexity.

More like this