🤖 machine learning

CART: Context-Anchored Recurrent Transformer -- A Parameter-Efficient Architecture with Learned Stability

The paper introduces CART, a parameter-efficient language model that reuses a single core block via learned stability mechanisms, but finds that while prelude depth dominates performance, the architecture ultimately underperforms parameter-matched dense baselines and fails to support effective test-time depth scaling.

Original authors: Chad A. Capps

Published 2026-06-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chad A. Capps

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: A "Looping" Language Model

Imagine you are trying to write a story. The standard way AI does this (called a Transformer) is like hiring a team of 12 different editors. Editor 1 reads the first sentence, Editor 2 reads what Editor 1 wrote, Editor 3 reads what Editor 2 wrote, and so on. Each editor has their own unique style and notes. This works well, but it's expensive because you have to pay for 12 different people.

CART (Context-Anchored Recurrent Transformer) tries a different approach. It hires just one editor and asks them to read the story six times (or more).

The Goal: To get the same quality of writing but with a much smaller team (fewer parameters), saving money and memory.
The Catch: If you ask the same person to read the same thing six times, they might just get bored and repeat the same mistakes. The challenge is making that single editor get smarter with every pass without needing new people.

How CART Works: The "Anchor" and the "Gate"

CART has three special tricks to make this looping work:

The "Anchor" (The Frozen Map):
- Standard Looping: In other looping models, every time the editor reads the story, they redraw the map of where the important words are. This is slow and redundant.
- CART's Way: Before the looping starts, a small team of "pre-editors" (called the Prelude) reads the story once and draws a perfect, detailed map of the important words. This map is frozen.
- The Loop: The single looping editor then looks at this same, frozen map every single time they read the story. They don't redraw the map; they just use it as a stable anchor to refine their understanding. This saves a lot of computing power.
The "Gate" (The Stability Switch):
- The Problem: If you ask someone to think about something over and over, they might get confused or start hallucinating (going crazy).
- CART's Solution: There is a "gate" that controls how much of the previous thought the editor keeps. It's like a volume knob. If the editor is getting too chaotic, the gate turns the volume down slightly.
- The Discovery: The paper found that this gate learns to settle on a very specific "sweet spot" (a number between 0.79 and 0.83). It's not forced by the engineers; the AI learned this specific setting on its own to stay stable.
The "Loop Index" (The Step Counter):
- The editor is given a counter (1, 2, 3...) so they know which pass they are on. This helps them know if they are in the early stages of thinking or the final polish.

The Experiments: What Happened?

The researchers tested this on a single consumer graphics card (like a high-end gaming PC) using two stages of training.

Stage 1: The Quick Test (The "Hunch")

They trained many small versions of CART quickly.
The Hunch: They thought that for bigger, more complex models, looping more times (e.g., 8 times instead of 6) would be much better. It looked like "more loops = smarter AI."

Stage 2: The Full Training (The "Reality Check")

They trained the models for much longer (using about 1 billion words of text).
The Surprise: The hunch was wrong. When they trained fully, looping more times actually made the model slightly worse or no better at all.
The Analogy: Imagine asking a student to read a textbook chapter 10 times. In a quick test, it looks like 10 times is great. But after studying it for a whole semester, you realize that reading it 6 times is enough. Reading it 8 or 10 times just leads to diminishing returns or even confusion. The model learned that "more loops" didn't equal "more intelligence" in this specific setup.

The Big Verdict: Did It Win?

The researchers compared their "One Editor, Six Loops" model against a standard "Six Different Editors" model (a Dense Transformer) that had the same amount of memory.

The Result: The standard model (6 different editors) beat the CART model.
- The standard model was about 10% better at understanding language.
- Even when they gave the standard model the same "effective" power (by making it 12 editors long), it still crushed CART.

Why did CART lose?
The researchers did a "autopsy" (diagnostic ablations) to find out why:

The Frozen Map wasn't the problem: Even if they let the editor redraw the map every time, it didn't help much.
The "Machinery" was useless: The special gadgets they added (the stability gate, the step counter, the blending of past thoughts) turned out to be mostly decorative. Removing them didn't hurt performance much.
The Real Issue: The main problem was sharing the same weights. The idea that "one editor doing six passes" is as good as "six different editors" simply didn't hold up. The shared weights were a bottleneck. The "heterogeneous" design (mixing pre-editors, a frozen anchor, and a looping editor) was just too complex and less efficient than a simple, uniform stack of editors.

Summary in One Sentence

CART is a clever, memory-efficient idea that tries to make a single AI block "think" repeatedly by using a frozen reference map, but in the end, it turns out that having a team of unique specialists (the standard model) still works better than one specialist looping around, and the extra gadgets added to make the looping stable were mostly unnecessary.

Key Takeaway for the Future:
The paper concludes that while the "frozen anchor" saves some computing power, the promise of getting massive intelligence gains from simply looping a shared block of code didn't pan out at this scale. The architecture is stable and interesting, but it didn't beat the standard approach in terms of raw performance per dollar.

Technical Summary: CART (Context-Anchored Recurrent Transformer)

Problem Statement

The dominant paradigm in language modeling, the stacked transformer, scales parameter count linearly with depth ( $L$ unique layers), creating memory bottlenecks for deployment on constrained hardware. Depth-recurrent (or looped) transformers attempt to address this by reusing a single set of parameters across multiple passes ( $R$ ) through a shared block. However, naive weight-sharing often underperforms depth-matched unique models because a single block applied identically cannot adapt its behavior across loop iterations. Prior solutions, such as hyper-connections, loop-index embeddings, and explicit stability constraints, have attempted to mitigate this, but a critical architectural gap remains: most concurrent looped architectures (e.g., Hyperloop, Parcae, OpenMythos) recompute key-value (K, V) representations from the evolving hidden state at every iteration, failing to separate context encoding from iterative refinement.

Methodology: CART Architecture

CART introduces a parameter-efficient architecture that reuses a single shared core block $R$ times across depth, distinguished by three specific architectural contributions:

Once-Computed KV Anchor: Unlike prior works that recompute K and V at every loop step, CART computes K and V once from a multi-layer "prelude" (a unique set of $P$ transformer layers) and holds them fixed. The recurrent core then performs cross-attention against these frozen tensors. This separates context encoding (prelude) from iterative refinement (core), reducing per-loop FLOPs by eliminating redundant KV projections.
Learned Stability via Sigmoid-Gated LTI: To ensure the recurrent computation converges, CART employs a learned Linear Time-Invariant (LTI) gate: $h_{\ell} = \sigma(a) \odot h_{in} + f_{transformer}(h)$ . Unlike architectural constraints (e.g., zero-order hold discretization) used in other works, this gate learns its convergence rate (spectral radius $\varrho$ ) directly from data, guaranteeing $\varrho < 1$ without structural rigidity.
Causal Cross-Attention Constraint: The paper identifies a critical constraint for autoregressive recurrent models: cross-attention from the core to the prelude must be causal. Non-causal attention leaks future tokens (from the prelude's context) into the recurrent state, causing loss collapse.

The architecture follows a "prelude $\to$ anchor $\to$ core $\to$ coda" topology. The prelude ( $P$ unique layers) generates the context anchor $e$ . The core (shared weights, $R$ iterations) refines the state using the frozen K, V from $e$ . A single unique "coda" layer produces the final logits.

Experimental Setup

The authors evaluated CART in two stages on consumer-grade GPUs (RTX 3050 and RTX 3090) using a single-pass corpus of $\sim$ 1B tokens (TinyStories, Wikipedia, FineWeb-Edu).

Stage 1: A hyperparameter screen of 64 configurations ( $d \in \{256, 512, 768, 1024\}$ , $R \in \{2, 4, 6, 8\}$ , $P \in \{2, 3, 4, 6\}$ ) trained for 3,000 steps ( $\sim$ 49M tokens).
Stage 2: Full training of 36 configurations ( $P=6$ , $R \in \{6, 8, 10\}$ , 3 seeds each) for 30,500 steps ( $\sim$ 1B tokens).
Baselines: At the binding scale ( $d=1024$ ), CART was compared against Dense transformers matched on both stored parameters (7-layer) and effective parameters (12-layer).

Key Results

1. Stability and Spectral Radius

Across all 36 Stage 2 configurations, the learned LTI gate's spectral radius $\varrho$ settled in a narrow band of $[0.79, 0.83]$ . This value is not a fixed design target but emerges from training. $\varrho$ rises monotonically with model width ( $d$ ) and weakly with loop count ( $R$ ). The "universal" $\varrho \approx 0.893$ observed in Stage 1 was identified as a mid-training transient that decays during full training.

2. The Reversal of Recurrence Benefit ( $R$ )

A critical finding is the reversal of trends between Stage 1 and Stage 2:

Stage 1 (3k steps): Suggested that recurrence depth ( $R$ ) becomes more valuable as model width ( $d$ ) increases, with $R=8$ outperforming $R=2$ significantly at $d=1024$ .
Stage 2 (30.5k steps): This trend reversed. At full training, $R=6$ was the best configuration for all scales $d \ge 512$ . Increasing $R$ to 8 or 10 resulted in performance degradation (up to 1% regression on perplexity).
Inference Scaling: Variable- $R$ inference on the best $d=1024$ model (trained at $R=6$ ) showed that running more iterations than trained consistently degraded performance on HellaSwag, LAMBADA, and PIQA. This indicates a clean negative result for test-time compute scaling under the current training recipe.

3. Parameter Efficiency Claims

At the binding test ( $d=1024$ , $\sim$ 1B tokens):

Stored-Parameter Parity: A Dense 7-layer model (matched to CART's stored parameters) outperformed CART ( $P=6, R=6$ ) by 1–2% on perplexity metrics.
Effective-Parameter Parity: A Dense 12-layer model (matched to CART's effective parameters) outperformed CART by $\sim$ 10% on natural-language sets.
Conclusion: The hypothesis that shared-weight recurrence provides $R$ -fold effective capacity did not hold at this scale and training budget.

4. Diagnostic Ablations

The authors decomposed the performance gap between CART and the Dense baseline:

Frozen K, V: Unfreezing K, V (recomputing them at each step) did not significantly improve performance, ruling out the frozen anchor as the primary bottleneck.
Recurrence Depth: Reducing $R$ to 1 (removing recurrence) caused only a $\sim$ 1% drop, suggesting the recurrent core contributes marginally beyond the first iteration.
Shared Weights: Unsharing the core weights (giving each iteration unique parameters) improved performance by $\sim$ 5%, confirming that weight sharing is a significant capacity bottleneck.
Machinery Components: Ablations of the HyperConnection, LTI gate, and Loop Index Embedding (LIE) showed they are vestigial in the shared-weight configuration; removing them individually had no measurable effect on perplexity.
Residual Gap: The remaining $\sim$ 5% gap to the Dense baseline is attributed to the heterogeneous "prelude $\to$ anchor $\to$ core $\to$ coda" framing itself, rather than any single mechanism.

Significance and Claims

The paper claims CART as a rigorous empirical investigation into depth-recurrent transformers, offering three main contributions:

Architectural Distinction: CART is the first in its family to compute K, V once from a prelude and reuse them as a frozen anchor, separating context encoding from refinement.
Learned Stability: It demonstrates that a sigmoid-gated LTI filter can learn a stable convergence rate ( $\varrho \in [0.79, 0.83]$ ) without structural constraints, adapting to model width and loop count.
Empirical Reality Check: The paper provides a "clean negative result" regarding the parameter-efficiency of shared-weight recurrence at the tested scale. It demonstrates that:
- The benefit of recurrence depth ( $R$ ) does not necessarily grow with model width at full training; in fact, it may reverse.
- Test-time compute scaling (increasing $R$ at inference) is detrimental if not matched to training.
- The "shared-weight leverage" thesis (that $R$ iterations of a shared block equal $R$ unique layers) fails at $d=1024$ with $\sim$ 1B tokens, with the architecture losing to a parameter-matched Dense baseline.

The authors conclude that while CART offers deterministic latency and fixed memory usage (beneficial for deployment), its current formulation does not deliver the parameter efficiency required to outperform standard Dense transformers at the tested scale. The findings suggest that the heterogeneous architectural framing and weight sharing are the primary bottlenecks, not the specific recurrence machinery (HyperConnection, LTI, LIE).