🤖 machine learning

The Equilibrium Is the Initialization: Lazy Identity Collapse in Physics-Structured Deep Equilibrium Reasoning

This paper reveals that physics-structured Deep Equilibrium Models often suffer from "lazy identity collapse," where the implicit solver performs no actual computation by converging to its initialization regardless of input difficulty, rendering the complex architecture ineffective compared to simple baselines.

Original authors: Joyjeet Singh

Published 2026-07-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Joyjeet Singh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've built a super-smart, physics-powered robot designed to solve tricky puzzles. You programmed it with a special "thinking loop" that was supposed to work like this: when a puzzle is easy, the robot takes a quick glance and answers. When a puzzle is hard, the robot is supposed to spin its gears, run a complex simulation, and use that extra effort to find a deeper, better answer. It's like a calculator that decides to do extra math only when the numbers get scary.

You built this robot using a fancy "Port-Hamiltonian" design (a way of borrowing rules from physics to keep the robot stable) and gave it a "learned initialization" (a smart starting point). You trained it on two very different tasks: one was checking if a story makes logical sense (ProofWriter), and the other was figuring out if you can walk from point A to point B on a map (Graph Reachability).

Here is the twist, and it's a big one: The robot never actually thought.

The "Lazy" Robot

In almost every single experiment (18 out of 19 runs), the robot's "thinking loop" turned out to be a giant, silent trick. The robot was supposed to start at a point, run its physics simulation, and end up at a new, smarter point. But instead, it just stayed exactly where it started.

Think of it like this: You tell a student, "Read this chapter, think about it, and then tell me the answer." The student opens the book, reads the first sentence, closes the book, and says, "The answer is the first sentence." They didn't do the work; they just copied the starting line.

In the paper's data, the robot's final answer was identical to its starting point to a precision of 10⁻⁶ (that's six zeros after the decimal point!). It was a perfect copy.

The "Zero Effort" Score

The most shocking part is that this lazy robot got the exact same score as a robot that didn't even try to think.

The Full Robot: Used the fancy physics loop. Score: 61.80% (on the map task) or ~91.77% (on the logic task).
The "Skip" Robot: The researchers told the robot, "Hey, just skip the thinking loop entirely. Take the starting point and send it straight to the answer." Score: 61.80% and ~91.77%.

The difference was 0.00 percentage points. The fancy physics engine contributed absolutely nothing. In fact, a simple two-layer "MLP" (a basic, non-robotic brain) did just as well, or even better, without needing the complex loop at all.

Why Didn't It Think?

You might guess that the robot was "stuck" because it was pulled back to its starting point by a safety rope (called an "anchor"). The authors tested this. They cut the rope. They even trained the robot to start in random, noisy places.

Result: It still didn't think.

When they removed the rope, the robot still stayed put.
When they forced it to start in a noisy place, the robot just copied that noise and the answer part of the brain learned to ignore the noise completely.

The real reason the robot was lazy was Gradient Starvation. This is a fancy way of saying the robot's "brain" realized that the easiest way to get a good grade was to ignore the hard work. The task was so simple that the robot could solve it just by looking at the clues directly, without needing the loop. Since the loop didn't help, the robot's learning algorithm decided, "Why bother? Let's just stay still." It found a stable, lazy solution and refused to move.

The One Time It "Thought" (And It Was Bad)

There was one single experiment (Seed 3 on the logic task) where the robot did move. But it didn't move to a smart answer; it went crazy.

The robot's state drifted 171.43 units away from where it started.
It wasn't solving the puzzle; it was generating pure noise.
When the researchers removed this "crazy" part, the robot's score actually improved from 81.77% to 83.23%.

So, the only time the robot tried to do something different, it made things worse.

The "Effort" Meter Was Broken

If the robot was actually thinking, you'd expect it to spend more time (more "solver iterations") on harder puzzles.

The Reality: The robot spent the exact same amount of time on easy puzzles and hard puzzles.
The Data: The correlation between puzzle difficulty and time spent was 0.009. That is basically zero.
The Cap: The robot hit the maximum allowed time limit (300 iterations) for 99.9% of the puzzles, regardless of how hard they were. It wasn't adapting; it was just hitting the wall every time.

The "Aha!" Moment: How to Catch a Lazy Robot

The authors realized that standard tests were lying to them. Usually, if you want to see if a part of a robot is useful, you turn it off (zero it out) and see if the score drops. But that didn't work here because turning off the "thinking" part also turned off the "starting" part, confusing the results.

Instead, they invented a new test called Substitution:

Take the robot's starting point.
Skip the thinking loop entirely.
Send the start point straight to the answer.
Compare the score.

If the score is the same, the robot was doing nothing. This test proved, with 0.00 difference, that the robot was a "no-op" (a no-operation).

The Bottom Line

The paper concludes that this specific design—a physics-structured deep equilibrium model with a learned start point—failed to do any "reasoning." It didn't fail because it was broken; it failed because it found a lazy shortcut that worked just fine.

Did it solve the problem? Yes, but only by copying the input, not by thinking.
Did the physics help? No. The stability of the physics actually made it easier for the robot to stay lazy.
Is this a breakthrough? No. It's a "negative result" that warns us: just because a model looks complex and uses fancy physics doesn't mean it's actually doing the hard work.

The authors ran all these tests on a single free Google Colab GPU, taking only 2–6 minutes per run. They released all their code and logs so anyone can check. The lesson? If you build a complex thinking machine, you have to check if it's actually thinking, or if it's just pretending to be busy while copying its homework.

Technical Summary: The Equilibrium Is the Initialization

Problem Statement

Deep Equilibrium Models (DEQs) and implicit recurrent architectures promise input-adaptive computation, where harder problems trigger more solver iterations, and the solved equilibrium encodes the result of genuine iterative inference. Physics-structured parameterizations (e.g., port-Hamiltonian dynamics) are often proposed to ensure stable, energy-controlled latent dynamics suitable for such implicit solving.

This paper investigates a specific failure mode in this paradigm: Lazy Identity Collapse. The authors report that in a port-Hamiltonian DEQ with a learned initialization, the "solved" equilibrium is numerically identical to the solver's start point. Consequently, the implicit computation is a silent no-op (no operation), contributing zero to the final accuracy and failing to allocate computational effort based on problem difficulty.

Methodology

The study employs a controlled, reproducible experimental design on two reasoning tasks:

ProofWriter Entailment: Binary entailment on frozen DeBERTa embeddings, stratified by proof depth.
BFS-Verified Graph Reachability: Synthetic $k$ -hop reachability on 16-node graphs, labeled by Breadth-First Search (BFS) to provide a ground-truth difficulty signal.

Model Architecture:

Dynamics: A damped, anchored, symplectic-Euler-style update on phase-space variables $z=(q, p)$ .
Initialization: A learned, goal-conditioned feed-forward network maps context to an initial state $z_0$ .
Solver: Anderson acceleration (m=5, cap 300 iterations) solves for the fixed point $h^*$ .
Controls: A two-layer MLP baseline trained on identical inputs serves as a positive control to establish the upper bound of performance without implicit layers.

Diagnostic Protocol:
The authors introduce a four-test protocol to audit implicit computation, moving beyond standard accuracy metrics:

Substitution Test (T1): Bypass the solver entirely by feeding the start point $z_0$ directly to the decoder.
Representation Probe (T2): Train a fresh classifier on the frozen equilibrium output $h^*$ to check if it carries task signal.
Iteration-Difficulty Correlation (T3): Correlate solver iteration counts with ground-truth task difficulty.
Forced Displacement (T4): Start the solver from a displaced point at inference to test if the learned vector field contains structure away from the initialization.

Key Results

1. The Equilibrium is the Initialization

Across 19 training runs (18 on graph tasks, 5 on ProofWriter), the displacement between the solved equilibrium $h^*$ and the start point $z_0$ is numerically zero ( $\approx 10^{-7}$ ).

Bypassing the Solver: Feeding $z_0$ directly to the decoder results in 0.00 percentage point (pp) accuracy loss compared to the full pipeline in 18 of 19 runs.
Anchor Irrelevance: Removing the explicit anchoring term (which pulls the state toward $z_0$ ) changes nothing. Retraining with noise-decoupled starts ( $z_0 + \epsilon$ ) results in the solver converging to the noisy start, with the decoder learning to ignore the channel.

2. Two Degenerate Regimes

The paper identifies two distinct failure modes:

Lazy Identity Collapse (Majority): The solver performs a near-identity map. The equilibrium carries no task signal beyond what is copied from the start. The decoder relies entirely on the direct context pathway.
Solver Blow-Up (Minority): In one ProofWriter seed, the solver diverges ( $\|h^* - z_0\| \approx 171$ ). The decoder co-adapts to this high-variance noise. Zeroing $h^*$ in this regime improves accuracy, confirming the "signal" was actually noise riding on divergence.

3. Failure of Standard Ablations

The authors demonstrate that the standard "zeroing ablation" (setting $h^* = 0$ ) is confounded.

Zeroing removes both the computation and the input pathway (the start point $z_0$ ) if the decoder relies on the identity copy.
This leads to wildly seed-dependent accuracy drops (e.g., +44 pp drop in one seed, 0 in another), falsely suggesting the DEQ is sometimes critical.
The Substitution Test correctly isolates the computation, showing a stable 0.00 pp gap.

4. Gradient Starvation

The mechanism driving this collapse is gradient starvation.

Both tasks are solvable via a direct context pathway (evidenced by the two-layer MLP baseline matching or beating the full DEQ).
Because the loss can be minimized without the implicit layer, the optimizer routes gradients through the direct path.
The implicit layer receives no incentive to move away from the initialization. In the graph pipeline, the initializer is gradient-isolated; in ProofWriter, the initializer trains, but the dynamics on top of it remain an identity.

5. Iteration Counts Ignore Difficulty

Solver iteration counts show zero correlation ( $r = 0.009$ ) with ground-truth difficulty (BFS distance). 99.9% of solves hit the iteration cap regardless of task complexity, indicating the model does not adaptively allocate compute.

Significance and Claims

The paper does not claim that all DEQs or physics-structured models fail. Instead, it offers a cautionary, design-specific negative result with the following implications:

Silent Failure: Standard training telemetry (convergence, accuracy, solver iterations) can appear healthy while the implicit computation is entirely non-functional.
The Substitution Test is Essential: The authors propose the Substitution Test (T1) as a mandatory control for any paper claiming implicit or recurrent reasoning. If bypassing the module changes nothing, the module is not computing.
Gradient Incentives > Architecture: Merely adding an iterative module or stabilizing structure (like port-Hamiltonian constraints) does not guarantee computation. If a task is solvable via a shortcut (direct pathway), the optimizer will exploit that shortcut, rendering the expensive iterative path a stable "do-nothing" optimum.
Reproducibility: The entire study is fully reproducible, with code and raw logs released. All experiments ran on a single free Colab GPU, demonstrating that rigorous negative results can be established cheaply and quickly.

The authors conclude that for the specific architecture studied (port-Hamiltonian DEQ with learned initialization), the equilibrium is simply the initialization, and the system functions as a "careful copy machine" rather than a reasoner.