🤖 machine learning

When Can Depth Replace Precision? A Resource Theory of Quantized Neural Computation

This paper establishes a resource theory quantifying when and how increasing the depth of low-bit quantized neural networks can compensate for reduced numerical precision, deriving exact structural limits, convergence rates, and execution-dependent penalties that determine the feasibility of replacing precision with depth under specific operational constraints.

Original authors: Mojtaba Soltanalian

Published 2026-07-28

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Mojtaba Soltanalian

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Great Trade-Off: Why More Steps Can't Always Fix a Crude Map

Imagine you are trying to draw a perfect picture of a mountain. You have two tools: a super-fine, expensive pen that draws with incredible precision, and a cheap, chunky crayon that only has a few colors and makes thick, blocky lines. Usually, if you want a better picture with the cheap crayon, you just have to work harder. You might try to draw the mountain by taking thousands of tiny, careful steps, hoping that if you stack enough small, blocky moves together, they will eventually look like the smooth curve of the real mountain. This is the basic idea behind "quantized neural networks," a field of computer science where we try to make artificial intelligence run on simpler, cheaper hardware by using fewer numbers (low precision) to do the math.

For a long time, researchers believed that if you just added enough "depth" (more layers or more steps) to a low-precision AI, it could eventually mimic the performance of a high-precision one. It was like thinking, "If I take enough tiny, clumsy steps, I can walk exactly like a graceful dancer." But this paper asks a crucial question: Is there a limit to how well clumsy steps can mimic grace? The authors, led by Mojtaba Soltanalian, treat this not just as a coding problem, but as a physics problem. They ask: What is the absolute best a low-bit system can ever do, no matter how deep it gets? And does the way the computer actually does the math (its "arithmetic") change the answer?

The Paper's Big Discovery: The "Structural Floor"

The paper's main finding is that there is a hard limit, which the authors call a "structural floor." Think of this floor like the bottom of a swimming pool. If you are trying to dive deep, you can keep swimming down, but once you hit the bottom, you can't go any deeper, no matter how hard you kick. In the world of AI, this floor represents the distance between the perfect answer you want and the best possible answer your specific set of low-precision tools can ever reach.

The authors prove that for a given set of low-bit tools (a "dictionary" of operations), if the target you are trying to hit doesn't fit perfectly into the shape of those tools, you will hit this floor. No amount of adding more layers (depth) can remove it. It's like trying to build a perfect circle using only square Lego bricks; no matter how many bricks you use, you will always have jagged edges. The paper shows that this "jaggedness" is a permanent feature of the tools you chose, not a failure of the builder.

However, the paper also finds that if your target does fit the shape of your tools, then adding depth does help. In this case, the error (the mistake) shrinks as you add more steps, following a predictable rule: if you double the depth, you roughly halve the error. But this only works if the target is "coherent," meaning the low-bit steps are actually refining a single, smooth path rather than just randomly jumbling together.

The "Freezing" Trap: When More Steps Make Things Worse

One of the most playful and surprising parts of the paper is what happens when you actually run the math on a real computer. The authors show that simply adding more steps can sometimes make the AI worse, not better. They describe a scenario called "full-state write-back."

Imagine you are walking across a room, taking tiny steps. But every time you take a step, you have to stop and write your exact position on a piece of paper that only has a grid with big squares. If your step is smaller than the grid squares, the paper can't see it! It just writes down "you are still here." If you take a million tiny steps, but the paper can't see any of them because they are too small for the grid, you end up standing still. The authors prove that if your computer's "grid" (its precision) is too coarse, adding more depth can actually freeze the AI's progress. The tiny updates get rounded away and disappear.

To fix this, the paper suggests a clever trick called "increment error feedback." Instead of writing down your full position every time, you write down how much you moved and keep a little "carry" note of the tiny bits that were too small to write down. You add that note to the next step. This way, the tiny bits don't disappear; they pile up until they are big enough to be seen. The paper proves that with this method, the AI can keep improving as it gets deeper, avoiding the "freezing" trap.

The Rules of the Game: It's Not Just About "Bits"

The paper argues that we need to stop thinking about AI precision just as a number of "bits" (like 4-bit or 8-bit). Instead, we need to think of it as a resource theory. Just like you have a budget for money, you have a budget for:

Depth: How many steps you take.
Metadata: The "instruction manual" or codebook that tells the AI which tools to use.
Arithmetic: How the computer actually handles the math (does it round up, round down, or keep a carry note?).

The authors show that you can't just swap one for the other freely. If you have a bad instruction manual (metadata), adding more depth won't help. If your computer's math is too "clumsy" (bad arithmetic), adding depth might freeze the system. They provide a set of formulas and "certificates" that let engineers check before they start training an AI whether it is even possible to reach their goal. It's like checking a map before you start a hike to see if the destination is even reachable with the gear you have.

The Verdict: What Works and What Doesn't

The paper is very clear about what it has proved and what it hasn't.

Proven: They have mathematically proved that there is a "structural floor" for certain types of low-bit systems. They proved that "full-state write-back" can freeze progress, while "error feedback" can save it. They proved that for specific, simple targets, you need a depth that grows linearly with the precision you want to match.
Simulated/Measured: They ran experiments on real AI models (like DistilBERT) and found that the theory holds up. When they tried to refine a model that was already "coherent," adding depth worked. When they tried to refine a model that wasn't, it failed, just as the theory predicted.
Not Proven: They do not claim that any AI can be made to work with low precision. They explicitly rule out the idea that you can just throw more depth at a broken system and fix it. If the "floor" is too high, no amount of training will get you to the target.

In short, this paper tells us that "more depth" is not a magic wand. It is a powerful tool, but only if you have the right map, the right compass, and a way to keep track of the tiny steps so they don't get lost. If you ignore the rules of the game, you might end up taking a million steps and going nowhere at all.

Technical Summary: A Resource Theory of Quantized Neural Computation

1. Problem Statement

The paper addresses a fundamental architectural question in quantized neural networks: When can additional low-bit residual computation (depth) replace missing numerical precision for a fixed input–output map?

Traditional quantization views precision loss as a local approximation problem (e.g., replacing weights with low-bit values). This paper reframes the problem as a target-specific resource theory. It asks:

What is the infinite-depth limit of a low-bit system over a fixed horizon?
How quickly is this limit approached as depth increases?
Which numerical or routing resources (metadata, arithmetic semantics, schedule) must scale for this limit to survive actual execution?

The core challenge is distinguishing between structural representability (what the library of operations can theoretically express) and implementation feasibility (whether finite arithmetic and routing preserve that expression).

2. Methodology and Framework

The paper models a depth- $D$ low-bit student as a pure schedule selecting fields from a declared low-bit operation library $\mathcal{G}_b$ over a fixed horizon $T$ . It introduces a relaxed control framework where a probability vector over the library fields is allowed at each instant, defining a closed relaxed reachable set $\mathcal{R}_{\Omega, \text{rel}}$ .

Key Modeling Components:

Lifted State Space: The state is lifted to a Banach space $Z = C(\Xi; \mathbb{R}^q)$ , representing a complete input-indexed map rather than a single trajectory. This prevents choosing different schedules for different inputs unless explicitly declared.
Structural Floor: The distance from the target map $F^\star$ to the closed relaxed reachable set, denoted $E_{\Omega, \infty}(F^\star)$ . This is the exact asymptotic limit; no increase in depth can reduce the error below this floor for a given library.
Resource Decomposition: The total error is decomposed into three distinct radii:
1. Synthesis Error ( $C_{\text{syn}}/D$ ): The cost of approximating the relaxed set with a finite pure schedule.
2. Metadata Error ( $C_{\text{meta}} 2^{-s/m}$ ): The cost of describing a learned dictionary with $s$ bits.
3. Arithmetic/Execution Error ( $A_D$ ): The loss due to finite precision, accumulation, saturation, and routing.

Theoretical Tools:

Balanced Switching (Online Simplex Rounding): A lemma ensuring that a pure schedule can track a relaxed control with a bounded prefix discrepancy, leading to first-order convergence rates.
Hölder Regularity: Extends convergence rates to $O(D^{-\vartheta} + D^{-1})$ when field dependence is $\vartheta$ -Hölder rather than bounded variation.
Hybrid Systems Theory: Analyzes route-changing networks (e.g., Mixture-of-Experts) using transversality conditions and small-gain theorems to bound state errors caused by routing mismatches.
Primal-Dual Certification: Uses HJB subsolutions, support witnesses, and occupation measures to provide rigorous lower and upper bounds on the structural floor before training.

3. Key Contributions and Results

A. The Structural Floor is the Decisive Invariant

The paper proves that for a declared dictionary family, the error of a finite-depth system converges to the distance from the target to the closed relaxed reachable set.

Result: If the structural floor is positive, no optimizer or depth increase can remove it. If it is zero, the target is structurally representable.
Implication: Feasibility is determined by the geometry of the operation library relative to the target, not merely by nominal bit width.

B. The First-Order Depth Price is Necessary

The paper establishes that under specific conditions, the error decays exactly as $O(D^{-1})$ .

Fixed-Teacher Converse: A constructed binary residual recursion (and its neural realizations in ReLU MLPs and non-uniform attention) demonstrates that a fixed target requires a $D^{-1}$ error rate.
Matching Law: For a coherent high-precision comparator with error $\Theta(L^{-1})$ and a low-bit system with a zero structural floor, accuracy matching requires $D = \Theta(L)$ .

C. Execution Arithmetic Can Reverse the Benefit of Depth

A critical finding is that "depth replaces precision" is false without a specific execution model.

Full-State Write-Back: If every microstep writes the complete state to a fixed grid, the error envelope becomes $E \approx E_\infty + C/D + D\rho_z$ . This creates a U-shaped envelope with a finite optimal depth; deeper networks can freeze residual updates and perform worse.
Increment Error Feedback: Storing quantization residuals in a carry register allows the error to telescope. This preserves the $O(D^{-1})$ law and admits an exact integer realization on a common lattice with bounded register widths.

D. Metadata and Depth are Distinct Resources

Learned dictionaries (codebooks) and schedule depth are separate resources.

Allocation Law: Under a total budget, logarithmic metadata growth ( $s \sim \log D$ ) is sufficient to keep codebook discretization commensurate with first-order synthesis depth.
Packing Converse: Information-theoretic bounds show that approximating a family of targets requires paying for both schedule labels and global metadata.

E. Routing Requires Hybrid Stability

For networks with hard routing (e.g., MoE), simple margin arguments are insufficient.

Transversality: Route changes must be isolated and transversal.
Small-Gain Condition: The paper derives a hybrid stability condition where the product of state-error-to-event-width conversion and event-width-to-state-forcing conversion must be less than one to ensure first-order convergence.

F. Certification Before Training

The paper proposes a workflow to decide feasibility before training:

Bracket the Floor: Use primal (feasible trajectory) and dual (HJB/SOS witness) certificates to bound the structural floor.
Decision Rule:
- Feasible: If Upper Bound + Implementation Radius $\le$ Target Tolerance.
- Impossible: If Lower Bound - Implementation Radius > Target Tolerance.
- Unresolved: If the interval overlaps the tolerance (intentional outcome indicating need for better bounds).

4. Experimental and Verified Evidence

The paper distinguishes between exact proofs, exhaustive finite checks, controlled simulations, and pretrained diagnostics:

Exact Witnesses: Verified rational certificates for soft-threshold networks and attention systems confirm the existence of positive structural floors for incompatible targets and zero floors for compatible ones.
Fixed-Teacher Converse: Exhaustive enumeration of schedules up to depth 12 confirms the exact $D^{-1}$ asymptote and the non-zero limiting constant for specific attention and MLP architectures.
Pretrained Diagnostics: A study on DistilBERT (SST-2) shows that subdividing a trained block without quantization moves away from the original map (incoherent), whereas refining a coherent target reduces error. This validates the necessity of target coherence.
Optimization Gaps: Experiments show that local search (coordinate descent) can miss the global optimum by orders of magnitude, proving that representability (existence of a schedule) is distinct from trainability.

5. Significance and Claims

The paper claims to provide a unified, target-specific resource theory that separates structural limits from implementation details. Its significance lies in:

Redefining "Depth Replaces Precision": It clarifies that this trade-off is conditional on the operation library, the execution semantics (specifically error feedback vs. write-back), and the routing model. It is not a universal law.
Structural vs. Optimization: It rigorously separates the question of what can be represented (structural floor) from what can be found (optimization), preventing failed training from being misinterpreted as structural impossibility.
Actionable Certification: It provides a mathematical framework to certify feasibility or impossibility before expensive training, using primal-dual bounds and explicit arithmetic radii.
Execution-Aware Design: It highlights that standard quantization metrics (bit width) are insufficient; the semantics of accumulation (carry vs. saturation) and routing (transversality) are the true determinants of whether depth can compensate for low precision.

The paper explicitly states it does not claim that arbitrary layer insertion preserves maps, that parameter-bit savings imply latency/energy savings, or that a failed optimizer proves non-representability. It focuses on the structural and arithmetic boundaries of quantized residual computation.