🔢 mathematics

Capability from Access Structure, Not Scale: Lower Bounds and Pre-Registered Tests for Hybrid Sequence Models

This paper challenges the notion that scaling alone guarantees capability convergence by proposing the Capability Convergence Hypothesis, which argues that true capability requires a specific hybrid architecture combining a compressive state channel and a verbatim-index channel, a claim supported by theoretical lower bounds and pre-registered experimental results.

Original authors: Wenhui Chen, Jianlin Chen, Ziyao Lin, Chi Man Vong

Published 2026-07-17

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Wenhui Chen, Jianlin Chen, Ziyao Lin, Chi Man Vong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Great AI Race: Why Bigger Isn't Always Better

Imagine you are building a library to store the entire history of the universe. For a long time, scientists believed that if you just kept making the library bigger and the librarians smarter, they would eventually agree on exactly where every single book belonged. This idea, known as the Platonic Representation Hypothesis, suggests that as AI models grow massive, they all start to "think" in the same way, converging on a perfect, shared understanding of reality. It's like how every human, regardless of language, eventually agrees that fire is hot and water is wet.

But there's a catch. Knowing where a book is on a shelf (representation) is very different from actually being able to run to that shelf, grab the book, and read a specific sentence from it while the library is on fire and you only have a tiny flashlight (capability). In the world of AI, this "flashlight" is the inference budget—the strict limit on how much memory and computing power the model can use for every single question it answers. The big question isn't just "Can the model understand the world?" but "Can it actually find the right answer in a sea of information without running out of memory?" This is the puzzle this paper tackles: does making a model bigger guarantee it can solve harder problems, or is there a specific architectural trick required to actually get the job done?

The Apple in the Infinite Stream

This paper proposes a new rule for the AI world called the Capability Convergence Hypothesis (CCH). It argues that while AI models might all learn to "see" the world the same way as they get bigger, they won't necessarily get better at doing things unless they are built with a specific two-part structure.

To understand this, the authors use a thought experiment called "Newton's Apple in an Infinite Stream." Imagine a river flowing forever, carrying millions of pieces of paper. Somewhere near the start, someone writes "Newton, apple, 1666" on a piece of paper. The river then floods with billions of other papers that look and sound very similar (like "Newton, gravity, 1666" or "Apple, pie, 1666"). Your job is to wait until the very end of the river and pull out the exact original "Newton, apple, 1666" paper.

The paper argues that most current AI models are like two different types of swimmers trying to find that paper:

The Compressive Swimmer (SSMs): These models are like a swimmer who carries a tiny, heavy backpack. They can remember the general direction of the river (e.g., "it's a story about Newton"), but because their backpack is so small, they have to throw away the specific details to make room for new water. By the time they reach the end of the river, they've forgotten the exact words "apple" and "1666." They hit what the authors call the Shannon Wall: a hard limit where you simply cannot fit all the necessary details into a small memory.
The Verbatim Swimmer (Transformers): These models are like a swimmer who carries a giant, floating library of every single paper they've seen. They can remember the exact words perfectly. However, they have a Horizon Wall: they can only look back a short distance. If the river is too long, the "Newton, apple" paper has already floated out of their view. They also hit a Circuit Wall: if the task requires a long chain of complex logic (like solving a puzzle step-by-step), their fixed-size brain gets stuck and can't connect the dots.

The Winning Strategy: The Hybrid Bridge

The paper's main discovery is that to solve the "Newton's Apple" problem, you don't just need a bigger swimmer; you need a Hybrid. This is a model that combines both types of swimmers into one team:

The Idea (State Channel): One part keeps a tiny, efficient summary of the river's direction (the "Idea"). It remembers where to look without getting bogged down by details.
The Shadow (Index Channel): The other part keeps a massive, searchable list of every specific word it has seen (the "Shadow"). It remembers the exact details.

The authors call this the Access-Complete Hybrid. Their experiments show that when you combine these two, the model can solve problems that neither pure "Compressive" nor pure "Verbatim" models can solve, no matter how much you scale them up.

What the Experiments Showed

The researchers didn't just guess; they ran pre-registered tests (meaning they wrote down their rules before seeing the results) on small-scale models to see if this theory held up.

The Scissors Gap: They found a dramatic difference, which they call a "scissors gap." When they tested a pure "Compressive" model on a task with 128 hidden facts, it failed almost completely (99.4% error). But when they added just one layer of the "Verbatim" index to that same model, the error dropped to nearly zero (0.000%). This proved that the ability to solve the problem wasn't about the model being "smarter" or "bigger," but about having the right access structure.
The Training Reliability: They also found that while a pure model might theoretically solve a hard puzzle if it got incredibly lucky during training, the Hybrid model solved it reliably every single time. It's like the difference between a student who might pass a test if they guess right versus a student who actually knows the material.
The Industry Trend: The paper also looked at real-world AI models released between 2023 and 2026. They found that the industry is naturally drifting toward this Hybrid design. Most top models are now using a mix of the two approaches, keeping a small "state" for efficiency and a larger "index" for accuracy.

What This Means (and What It Doesn't)

The paper is very careful about what it claims. It does not say that Hybrid models are perfect or that they can solve every problem. It explicitly rules out the idea that "bigger is better" on its own; a pure model, no matter how huge, will still hit a wall if it lacks the right access structure.

It also admits that their "Newton's Apple" test is a worst-case scenario. In real life, natural language might not be as messy as the infinite river they simulated. However, the core finding is robust: Capability is not free. You cannot just buy better problem-solving skills by throwing more computing power at a single-type model. Instead, you have to "purchase" the ability to solve complex, long-range problems by building a system that has both a compressed memory for direction and a detailed index for retrieval.

In short, the paper suggests that the future of AI isn't just about making models bigger, but about building them with the right two-channel architecture. The "Idea" keeps the model moving forward, and the "Shadow" ensures it never loses the specific facts it needs to succeed.

Technical Summary: The Capability Convergence Hypothesis (CCH)

Problem Statement
The paper addresses a critical gap in the understanding of large language model scaling. While the Platonic Representation Hypothesis (PRH) posits that heterogeneous models converge toward a shared statistical representation of reality ( $Z$ ) as they scale, this paper argues that representational convergence does not automatically entail capability convergence. Specifically, under a fixed per-token inference budget, models may align in their internal representations yet diverge significantly in their ability to solve tasks requiring long-range, verbatim retrieval combined with serial state tracking. The authors identify that "knowing" a fact in representation space is distinct from "accessing" it under resource constraints.

Methodology
The paper employs a multi-layered evidential approach combining formal theory, thought experiments, and controlled empirical testing:

Theoretical Framework & Thought Experiment:
- The Newton's-Apple Problem: A thought experiment where a model must retrieve a specific fact (e.g., "Newton, apple, 1666") planted at the beginning of a stream after $10^7$ tokens of semantic-overlap noise.
- Three Resource Walls: The authors define three theoretical barriers that prevent pure architectures from solving this problem under fixed budgets:
  - Shannon Wall (Unconditional): Pure compressive models (e.g., SSMs) have $O(1)$ state capacity. Information theory (Data Processing Inequality) dictates they cannot store $O(Nb)$ bits of independent bindings, leading to a hard error floor.
  - Horizon Wall (Unconditional): Pure attention models with fixed windows lose access to tokens outside the window, failing exact retrieval for long streams.
  - Circuit Wall (Conditional on $TC_0 \neq NC_1$ ): Fixed-depth attention-only stacks cannot perform serial composition of state updates (e.g., $S_5$ group actions) required for complex reasoning chains.
- The Access-Complete Hybrid: The proposed solution is an architecture holding both a compressive state channel (for $O(1)$ serial composition) and a verbatim index channel (for $\Theta(L)$ exact retrieval).
Formal Proofs:
- The paper establishes information-theoretic lower bounds proving that any architecture with $o(Nb)$ query-time state fails exact retrieval on high-entropy independent bindings.
- It defines Capability Closure ( $C_\infty$ ) to formally demonstrate that the set of solvable tasks for a hybrid strictly exceeds the union of pure families ( $C_\infty(T \sqcup S) \supsetneq C_\infty(T) \cup C_\infty(S)$ ).
Empirical Evidence:
- Census Analysis: A survey of 140 released architectures (2023–2026) showing a convergence toward "access-complete" designs with a specific global-attention share ( $\rho$ ) in the range $[1/12, 1/4]$ .
- Pre-Registered Small-Scale Tests: Controlled experiments (0.2M–1.1M parameters) testing specific predictions (P1–P3) under a frozen protocol. These tests isolate architecture from scale and data.

Key Contributions

The Capability Convergence Hypothesis (CCH): A formal hypothesis stating that under fixed inference budgets, capability converges not to a single architecture, but to the access-complete class—any form combining a compressive state channel and a scalable verbatim index channel.
The "Scissors Gap" and Information Floor: The paper provides the first pre-registered measurement of a "scissors gap": pure-state models hit a high error floor determined by information-theoretic limits, while hybrids (with the same state size plus an index) achieve near-zero error.
Formal Separation of Representation and Capability: The authors demonstrate that representational alignment (PRH) does not predict capability. In matched-scale experiments, models with high representational alignment (CKA) showed divergent retrieval capabilities based solely on their access structure.
Existence Proof of Super-Additivity: The paper proves that the combination of a state channel and an index channel creates a capability strictly greater than the sum of its parts, specifically for tasks requiring both serial composition and exact retrieval.

Results

Theoretical: The three walls (Shannon, Horizon, Circuit) are established as unconditional or conditional barriers for pure families. The hybrid architecture is shown to cross these walls by "paying the price" of an index channel rather than evading the bounds.
Census (R1–R4): Industry data shows a consistent trend toward hybrid architectures with a global-attention share ( $\rho$ ) clustering around 0.167 (median), suggesting an economic and capability-driven convergence toward the access-complete class.
Experimental (P1–P3):
- P1 (Scissors Gap): Supported. Pure SSMs hit the information floor (error $\approx 0.994$ ), while hybrids with the same state plus one global-attention layer achieved error $\approx 0.000$ .
- P2 (Bifurcation): Partially supported. Hybrids with sufficient state expressivity (e.g., $\beta \in (0,2)$ ) solved $S_5$ tracking, while less expressive variants failed. However, reliable trainability at the boundary was observed to be seed-dependent.
- P3 (Channel Commensurability): Failed. The prediction that solving hybrids would show higher cross-channel alignment than failing ones was reversed; solving hybrids showed lower alignment, indicating channel differentiation rather than convergence.
- Conjunction Witness: The hybrid passed a composite task (address composition + retrieval) on all seeds, while no other pure architecture met the criterion. One pure-recurrence seed solved it, indicating the separation is currently one of training reliability rather than existence at this scale.

Significance and Claims

The paper claims to establish a new boundary for understanding model scaling: capability is purchased by access structure, not given freely by scale.

Modest Scope: The authors explicitly state they do not claim to have invented hybrid architectures or to have proven that natural corpora satisfy the necessary separability assumptions. The "witness" is a worst-case construction; the claim is that access structure can be decisive, not that it is decisive on all natural workloads.
Falsifiability: The paper positions CCH as a falsifiable hypothesis. The core claims (access-completeness principle) are supported by lower bounds and pre-registered experiments. The "convergence" aspect is framed as an economics-motivated conjecture supported by consistency (census data) rather than direct demonstration.
Implications: The authors argue that future benchmarking and model design must treat access structure as a first-class variable. They propose that the industry is converging toward a specific design pattern (the access-complete hybrid) because it is the only way to satisfy the dual requirements of long-horizon reasoning and verbatim retrieval under fixed inference budgets.

In summary, the paper argues that while representations converge toward a shared reality (PRH), capabilities converge toward a specific architectural attractor (CCH) defined by the ability to hold both a compressed state and a verbatim index. This convergence is driven by the economics of inference and the information-theoretic necessity of paying for access capacity.