🔬 physics

Thermodynamic cost of inference and learning in physical neural networks

This paper demonstrates that while quasi-static inference in physical neural networks incurs no thermodynamic cost, learning parameters carries an irreducible energy expense, establishing that the fundamental thermodynamic price of neural computation is determined by memory storage rather than arithmetic processing.

Original authors: Alexei Tkachenko

Published 2026-08-18

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Alexei Tkachenko

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Every time a computer solves a problem or learns a new pattern, it burns energy. We see this in the massive power plants that feed our data centers and the heat radiating from our laptops. For decades, scientists have wondered how much of this energy cost is simply a flaw in our current technology, and how much is a fundamental law of nature that no machine can ever escape. The standard answer for digital computers comes from a rule called Landauer's principle. It states that if a machine erases a piece of information, it must release a tiny, unavoidable amount of heat. This rule applies to the binary switches inside our processors, where information is stored as distinct, stable states like "on" or "off." However, many researchers are now building a different kind of computer. These are physical neural networks, devices made of light, mechanical parts, or electronic circuits that compute by relaxing into a state of balance, much like a ball rolling down a hill to find the lowest point. Because these machines do not rely on flipping rigid switches, the old rules about erasing bits might not apply to them in the same way. The question is whether these new machines can compute and learn with almost no energy at all, or if they too are bound by a hard physical limit.

A researcher at Brookhaven National Laboratory has now mapped out exactly how much energy these physical networks require. They did not build a physical device in a lab; instead, they created a detailed mathematical model that treats a neural network as a system of springs and weights. In their model, the connections between the layers of the network are not rigid instructions but elastic constraints, like springs that pull the system toward a specific shape. When the network is given an input, it simply relaxes into the shape that satisfies all the springs at once. This approach allowed the scientist to calculate the energy cost of two distinct processes: inference, which is the act of solving a problem, and learning, which is the act of training the machine to get better at solving problems. Their findings reveal a surprising split in the physics of these machines. They discovered that the act of thinking, or inference, can theoretically be done with zero energy if the machine moves slowly enough. The cost only appears when the machine is forced to move quickly. In contrast, the act of learning carries a permanent energy price tag that cannot be avoided, no matter how slowly the machine operates.

The researcher found that for inference, the energy cost is not determined by the number of connections or "weights" inside the network, as it is in digital computers. Instead, the cost depends on the width of the network, or how many neurons are active at once. If the machine is allowed to relax slowly, moving from one input to the next at a leisurely pace, it requires no work at all. This is because the machine is always in a state of equilibrium, sliding along a smooth valley of energy without ever having to climb a hill or erase a memory. The only time energy is spent is when the machine is pushed to move faster than its natural speed of relaxation. In this fast regime, the energy required is proportional to the distance the machine must travel through its state space, divided by the time it takes. The simulations showed that even at the fastest useful speed, the energy cost is remarkably low, roughly equivalent to the thermal energy of the environment for every dimension of the widest layer in the network. This means the cost scales with the size of the network's width, not its total complexity. For a network with a million active neurons, the energy cost is about a million times the tiny thermal energy of a single particle, a figure that is billions of times smaller than what current digital hardware consumes.

Learning, however, tells a different story. When the machine is trained, both the input and the desired output are forced onto the system at the same time. This creates a conflict, or strain, within the springs of the network, because the current settings cannot satisfy both the input and the target simultaneously. To learn, the machine must adjust its internal weights to relieve this strain. The researcher found that this process of adjusting the weights carries an irreducible energy cost. Unlike inference, this cost does not disappear even if the training is done infinitely slowly. Every time a weight is changed, a small amount of energy is dissipated as heat, roughly a few times the thermal energy of the environment per parameter. This cost is incurred once for the entire set of parameters that define the machine's knowledge. It does not depend on how many examples the machine sees during training, but rather on the number of weights it has to store. This stands in sharp contrast to digital training, where the machine must erase and rewrite information for every single example it sees, leading to a massive energy bill that scales with the size of the dataset.

The researcher tested these ideas by simulating a small network trained to recognize handwritten digits, a standard task in machine learning. They watched how the machine behaved as they switched between different numbers and as it adjusted its weights to improve its accuracy. The results matched their theoretical predictions perfectly. When the machine was asked to infer a digit quickly, the energy it consumed rose sharply, and its accuracy dropped as the work it did became comparable to the random jiggling of heat in the system. But when the machine was allowed to move slowly, it solved the problem with almost no energy, and its accuracy remained high. During training, they observed that the energy spent on adjusting the weights followed a precise pattern, confirming that a small, fixed amount of heat is released for every bit of learning that occurs. The simulations showed that once the machine had learned the task, the energy required to maintain that knowledge was negligible, but the energy required to write that knowledge in the first place was a fundamental, unavoidable cost.

These findings suggest that the thermodynamic price of a neural network is set by its memory rather than its arithmetic. The act of computing a result is nearly free, limited only by how fast the machine is forced to move. The act of learning, however, requires paying a small but permanent fee to write the parameters into the machine's physical structure. This distinction offers a new perspective on the future of computing. While digital machines are bound by the cost of erasing bits for every operation, physical machines could potentially perform complex calculations with an energy efficiency that is orders of magnitude better. The remaining gap between current hardware and these theoretical limits is not due to a flaw in the laws of physics, but rather to the inefficiencies of our current materials and designs. The study provides a clear roadmap for what is possible, showing that if we can build machines that truly relax into their solutions, the energy cost of intelligence could be reduced to the bare minimum allowed by the universe.

Technical Summary: Thermodynamic Cost of Inference and Learning in Physical Neural Networks

Problem Statement
The rapid expansion of deep neural networks has led to escalating energy demands for both training and inference. While current hardware inefficiencies are well-documented, a fundamental question remains: what energetic costs are intrinsic to neural computation itself, independent of specific device implementations? For digital hardware, the standard thermodynamic benchmark is Landauer's principle, which assigns a minimum dissipation of $k_B T \ln 2$ per erased bit for logically irreversible operations. However, this bound applies to metastable, multi-stable information-bearing states and does not directly constrain continuous analogue computation or physical systems that operate via relaxation and equilibration rather than clocked digital gates. The paper seeks to determine the thermodynamic limits of energy-based physical neural networks, specifically addressing how much energy is required for inference and learning when the network is modeled as a physical system governed by a Hamiltonian.

Methodology
The author proposes a theoretical framework that embeds a generic feedforward neural network into a physical Hamiltonian.

Hamiltonian Embedding: A standard feedforward network with layers $x_n$ , weights $W_n$ , and biases $b_n$ is mapped to an elastic energy function:
$H(x; \theta) = \frac{\kappa}{2} \sum_{n=1}^L \|x_n - f_n(W_n x_{n-1} + b_n)\|^2$
Here, each layer relation is treated as an elastic compatibility constraint with stiffness $\kappa$ . The "feedforward state" corresponds to the zero-energy manifold ( $H=0$ ). Unlike Hopfield networks where states are attractors, this construction creates a highly degenerate ground-state manifold where every input selects a unique, zero-energy compatible state.
Dynamics: The system evolves via overdamped Langevin dynamics ( $\dot{x}_n = -\mu \nabla_{x_n} H + \text{noise}$ ).
Inference vs. Training:
- Inference: Modeled as the relaxation of hidden and output variables to the zero-energy manifold under a fixed input.
- Training: Modeled as an overconstrained process where both input and target output are clamped, creating strain (loss) that drives parameter updates via mechanical force balance (backpropagation).
Thermodynamic Bounds: The paper derives exact bounds for the work required to drive the system between thermal states. It utilizes the Wasserstein-2 distance ( $W_2$ ) for optimal transport and relaxes this to an information-geometric bound involving the Kullback-Leibler (KL) divergence and Fisher-Rao distance.
Numerical Validation: Simulations were performed on a minimal physical network (architecture 64 $\to$ 20 $\to$ 10) trained on the handwritten digits dataset (MNIST). The simulations tested work dissipation, relaxation times, and accuracy under varying protocol speeds and temperatures.

Key Contributions and Results

Reversible Inference in the Quasi-Static Limit:
The paper proves that the equilibrium free energy of the embedded network is independent of the input and all parameters (weights and biases) at all temperatures. Consequently, the change in free energy ( $\Delta F$ ) for any input transition is exactly zero. In the quasi-static limit (infinitely slow operation), inference requires zero work. This contrasts with digital systems where computation often involves erasing bits; here, the computation is a monostable, isostatic response along a zero-energy manifold, avoiding the Landauer cost entirely.
Finite-Speed Inference Cost:
For finite-speed operation, the work required is bounded below by the squared Wasserstein-2 distance between the initial and final thermal states, divided by the protocol duration $\tau$ .
- The bound is relaxed to an entropic measure: $E_{inf} \ge d k_B T \Gamma_{inf} \frac{\tau_L}{\tau}$ , where $d$ is the width of the widest layer, $\tau_L$ is the diffusive timescale, and $\Gamma_{inf}$ is a fidelity factor related to signal-to-noise ratio.
- At the fastest usable speed (where $\tau \approx \tau_{inf}$ ), the fidelity and relaxation time cancel, yielding a cost of approximately $k_B T$ per dimension of the widest layer.
- Crucially, this cost scales with the width of the network (number of neurons), not the number of parameters (synapses). Simulations confirmed that the measured work saturates the transport bound to within 4%.
- Accuracy collapses when the dissipated work falls below the thermal scale ( $k_B T$ ), confirming that reliability is determined by the ratio of work to thermal distinguishability.
Irreducible Cost of Learning:
Learning is fundamentally different from inference. Because training requires changing the parameters (weights and biases), it carries an irreducible cost that survives the quasi-static limit.
- The total training cost consists of a data-presentation term (which vanishes with slow presentation) and a parameter-annealing term.
- The parameter-annealing cost is approximately a few $k_B T$ per parameter for the entire training run. This cost is incurred once to establish the parameter configuration, unlike digital training which pays an erasure cost for every data presentation.
- The cost scales with the number of parameters ( $N_\theta$ ) but is independent of the dataset size in the quasi-static limit (unlike digital benchmarks which scale with $N_\theta \times \text{Dataset Size}$ ).
Comparison with Digital Benchmarks:
The paper compares these physical bounds to the Landauer limit for irreversible digital hardware (16-bit arithmetic).
- Inference: The physical cost ( $\sim d k_B T$ ) is orders of magnitude lower than the digital Landauer benchmark ( $\sim N_\theta k_B T$ ) because physical networks perform multiplications via geometry rather than discrete operations.
- Training: The physical cost is a one-time parameter update cost, whereas digital training incurs erasure costs for every token in the dataset.
- Table 1 in the paper estimates that physical inference could be $\sim 10^5$ times more efficient than the Landauer benchmark and $\sim 10^{13}$ times more efficient than current hardware estimates, while physical training offers a similar advantage over digital training due to the absence of per-token erasure costs.

Significance and Claims
The paper claims to establish the fundamental thermodynamic price of neural computation, distinguishing between the costs of arithmetic (inference) and memory (learning).

Inference: The thermodynamic cost is set by the network's width and the speed of operation, not its parameter count. In the ideal limit, computation itself is free; the cost arises only from the finite speed required to separate the signal from thermal noise.
Learning: The cost is dominated by the memory aspect—writing the parameters. This cost is irreducible even in the slow limit but is amortized over the entire dataset, making it potentially much cheaper than digital training which pays per example.
Physical vs. Digital: The work suggests that physical neural networks can theoretically undercut digital implementations not merely through engineering improvements, but because they avoid the fundamental Landauer cost associated with erasing information during computation. The "thermodynamic price" of a neural network is thus associated more with its memory (learning) than its arithmetic (inference).

The author notes that these bounds apply to ideal substrates; real-world costs (data conversion, control, non-ideal materials) currently dominate. Furthermore, the model assumes an affine-plus-activation structure and does not yet address attention mechanisms or the costs of storing weights after training. Nevertheless, the results provide a rigorous thermodynamic baseline for evaluating future neuromorphic and physical computing architectures.

Technical Summary: Thermodynamic Cost of Inference and Learning in Physical Neural Networks

More like this