💬 NLP

What Iterated Self-Feeding Probes of Language Models Measure, and a test that separates the construction from the model

This paper introduces a rigorous test to distinguish between artifacts inherent to the probe construction (such as kinematic damage spreading) and genuine model properties (like Lyapunov exponents) in iterated self-feeding language model experiments, revealing that previous claims of phase transitions were often misattributed to the models rather than the methodology.

Original authors: Nicolás Vera Zúñiga

Published 2026-08-12

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Nicolás Vera Zúñiga

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Detective's Dilemma: When the Tool Becomes the Clue

Imagine you are trying to figure out how a specific type of car engine works. You can't take it apart, so instead, you build a giant, automated test track. You feed the car's own exhaust back into its intake, over and over again, watching how the engine reacts to its own fumes. This is a bit like what scientists do with modern "Large Language Models" (AI). These AI systems are like super-smart engines that predict the next word in a sentence. Researchers often feed the AI's own output back into itself to see if it can fix its own mistakes, think deeper, or solve hard problems. This is called "self-feeding" or "iterated probing."

But here is the tricky part: when you build a test track, the track itself has rules. Maybe the track is too bumpy, or maybe the wind blows in a specific way that makes the car wobble. If you see the car wobbling, is it because the engine is broken, or because the track is bumpy? In the world of AI, scientists have been measuring things like "how fast the AI's thoughts change" or "how stable its answers are," assuming these numbers tell them about the AI's brain. This paper asks a simple, scary question: What if those numbers are actually just telling us about the test track we built, and not the AI at all?

The Great Mix-Up: Is it the AI or the Experiment?

In this paper, the researcher, Nicolás Vera Zúñiga, sets up a very specific, slightly weird experiment to find out. Imagine a giant circular necklace made of 4096 beads, where each bead is a word (or "token") from the AI's vocabulary. The AI looks at a small window of beads around a specific spot, guesses what the bead should be, and then swaps the current bead for its new guess. It does this for every bead on the necklace, over and over again, in a random order. This creates a closed loop where the AI is constantly rewriting its own story based on its own previous guesses.

To test this, the researcher creates two identical necklaces. On the second necklace, he flips just one bead at the very start. Then, he runs both necklaces through the exact same simulation, using the exact same random numbers to decide the swaps. If the AI's "brain" is sensitive, that single flipped bead should cause a ripple effect, making the two necklaces look more and more different over time. This is called "damage spreading." The researcher measures how fast this damage spreads (a number called $\lambda_{ca}$ ) and how much of it survives.

The Big Surprise: The "Phase Transition" That Wasn't

The researcher found something shocking. When he ran this experiment at very low "temperatures" (which means the AI is forced to be very confident and pick only the most likely words), the system suddenly collapsed. The necklace stopped changing and froze into a single repeating word (like a broken record stuck on "newline"). This looked like a dramatic "phase transition," similar to water turning instantly into ice.

The researcher measured this transition with extreme precision, down to three decimal places. It looked like a real discovery about how AI models behave. However, the paper proves this transition belongs to the experiment, not the AI.

Here is how they knew:

The Control Test: They tried the exact same experiment on a different type of AI model (a "masked language model") that is built differently. This second model never froze, even at the same low temperatures.
The Mechanism: They realized the freezing happened because of a specific mathematical "trap" in the way the first AI model picks its words. It's like a ball rolling into a deep hole; once it falls in, it can't get out. This hole exists because of the rules of the experiment (the specific way the AI is forced to update its beads), not because the AI is special.
The Verdict: The "phase transition" was a "manufactured" event. It was a property of the test track, not the car. The researcher admits they spent four months thinking they had discovered a new law of AI physics, only to realize they had just discovered a quirk of their own measuring stick.

What Does the Experiment Measure?

So, if the "phase transition" was fake, does the experiment tell us anything real? Yes, but you have to be very careful about what you look at. The paper introduces a "Discriminator Test" to separate the noise from the signal:

The "Construction" Readings (The Noise): Some numbers, like the speed at which the damage spreads ( $\lambda_{ca}$ ), are fixed by the geometry of the test. If you change the size of the window or the order of the beads, these numbers change. If you change the AI model, these numbers stay mostly the same. These readings tell you about the experiment, not the AI.
The "Model" Readings (The Signal): Other numbers, like the "attractor share" (how much the AI settles on one specific word), do change when you change the AI model. If you train a model longer, this number moves. If you switch to a different AI architecture, this number moves. This is the only part that actually tells us about the AI's brain.

The "Gotchas" and Retracted Claims

The paper is also a cautionary tale about how easy it is to fool yourself with statistics. The researcher lists four times they almost published a "discovery" that turned out to be wrong because they didn't check their math properly:

They once measured a "critical point" using a setup that was too small to be valid.
They accidentally used the same random numbers for different parts of the test, making the results look more certain than they were.
They tried to find a pattern in data that was actually just a flat line (zero variance), which made a correlation look real when it was just a coincidence of how the data was sorted.

Every time, they caught the mistake not by peer review, but by running a "known answer" test (like a simulation where the result is already known) and seeing that their tool gave the wrong answer.

The Bottom Line

This paper doesn't give us a new super-power for AI. Instead, it gives us a new pair of glasses. It teaches us that when we feed an AI its own output, we are mixing two things: the AI's actual intelligence and the artificial rules of our experiment.

The main takeaway is that we cannot just look at a number and assume it tells us about the AI. We have to play a game of "control and variable": if we change the AI and the number stays the same, the number is about the experiment. If we change the experiment and the number stays the same, the number is about the AI. The paper proves that some of the most exciting "discoveries" in this field might just be reflections of the mirror we are holding up, not the face we are looking at. It's a call for scientists to be humble, to check their tools, and to realize that sometimes, the most interesting thing they find is the flaw in their own measuring tape.

Technical Summary: What Iterated Self-Feeding Probes of Language Models Measure

Problem Statement

A growing class of methods probes language models (LMs) by feeding their own output back into their input, creating dynamical systems such as self-consistency, iterated refinement, and agentic loops. While these methods are routine, a fundamental question remains unanswered: what do the dynamics of these self-feeding probes actually measure?

The paper argues that current readings from such probes conflate two distinct sources of information:

Construction-determined quantities: Properties arising solely from the mechanics of the probe (e.g., the loop structure, resampling rules, and coupling method).
Model-determined quantities: Properties intrinsic to the language model itself.

Without a method to distinguish between these, researchers risk misattributing artifacts of the measurement apparatus to the model. The author reports a specific instance where they measured a "phase transition" to three decimal places, only to discover it belonged entirely to the probe's construction, not the model.

Methodology

The Construction

The author employs a specific construction to isolate the confound between probe and model:

State: A ring of $N$ token cells with periodic boundaries.
Dynamics: In-place resampling. At each step, a site $x_i$ is resampled from the model's windowed conditional probability $p(x_i | x_{i \pm r})$ at temperature $T$ . The center token is masked from its own window.
Coupling: Two rings ( $x$ and $\tilde{x}$ ) are advanced using Common Random Numbers (CRN). They are identical except for one flipped token at $t=0$ . Both use the same stream of uniform variates and the same visit order.
Null Hypothesis: Under CRN, if two rings have identical windows, they draw the exact same token (inverse-CDF sampling against a shared uniform). Thus, undamaged twins diverge by exactly zero. Any divergence is strictly attributable to the injected flip.

Measured Quantities

$\lambda_{ca}$ (Token-space Lyapunov exponent): The exponential growth rate of the number of differing sites ( $d_t$ ) before saturation.
$D_{norm}$ : The saturating level of damage, normalized by an independent-noise floor.
Attractor Share: The occupancy of the dominant token in the settled ring.

Validation Strategy

The instrument is calibrated via reproduction on systems with known answers, specifically the Domany–Kinzel probabilistic cellular automaton. The author verifies that their implementation produces bit-exact matches to independent predictions on this automaton before applying it to language models.

Key Contributions

1. A Validated Instrument

The author provides a calibrated tool that distinguishes between the probe's geometry and the model's behavior. The instrument was validated by reproducing the Domany–Kinzel damage field exactly and recovering known transition matrices for elementary cellular automata.

2. Identification of a "Manufactured" Phase Transition

The author discovered a sharp absorbing-state transition in the probe's dynamics at a critical temperature $T_c \in [0.4343, 0.4391]$ .

Nature: This transition is a property of the probe, not the model.
Mechanism: It arises from an attracting fixed point of the argmax map in the autoregressive construction (specifically for models like Pythia-410m), causing the lattice to collapse onto a single token (e.g., newline) at low temperatures.
Evidence: The transition disappears when the construction is altered (e.g., using a masked-LM construction without such a fixed point) but persists across 19 different models and scale ladders (70x variation).

3. The Discriminator Test

The paper introduces a rigorous test to separate construction-determined from model-determined readings:

Test: Vary one factor (construction or model) while holding the other fixed.
Criterion: If a reading moves when the construction changes but stays fixed when the model changes, it is construction-determined. If it moves when the model changes but stays fixed when the construction changes, it is model-determined.
Application:
- Construction-determined: The damage light cone (kinematic) and the radius scaling of $\lambda_{ca}$ (model-invariant).
- Model-determined: The attractor share, which consistently ranks models across different constructions.
- Developmental: $\lambda_{ca}$ tracks the training trajectory of a single model (crossing zero at a reproducible point) but fails to distinguish between different model families.

4. Estimator Gating and Retracted Verdicts

The author reports four retracted verdicts where estimators produced confident but incorrect numbers because they were applied to quantities with no room to vary (e.g., saturated predictors, undefined values treated as zero, or correlated replicas). They release a package of "guards" (e.g., dynamic-range checks, noise gates, tie-aware ranking) to prevent these errors.

Results

Probe Artifacts: The "phase transition" observed in autoregressive models is an artifact of the probe's geometry (specifically the argmax map's fixed point). It is reproducible but carries no information about the model's capabilities.
Model Sensitivity:
- $\lambda_{ca}$ : While it shows a developmental transition (crossing zero during training), it is model-invariant across different architectures and scales. It does not serve as a reliable metric for comparing different models.
- Attractor Share: This metric successfully distinguishes between model families (e.g., RWKV vs. Attention models) and is robust to changes in the probe's construction (radius, temperature).
Ablation Response: Removing internal components (e.g., attention blocks) changes the dynamics, confirming the instrument reads the model. However, the response is non-monotonic and depends on the initial state of the reference, indicating a complex statistical relationship rather than a simple causal mechanism.
Predictive Power: The attractor share predicts instruction-following failures, whereas the developmental transition ( $\lambda_{ca}$ ) does not. Conversely, $\lambda_{ca}$ (via diversity) predicts greedy degeneration, while the attractor share does not.

Significance and Claims

The paper claims that iterated self-feeding probes measure a mixture of the construction and the model, and that these components are indistinguishable by inspection alone.

Modesty of Claims: The author explicitly states this is not an interpretability result. They do not claim $\lambda_{ca}$ localizes a specific internal mechanism. Instead, the contribution is measurement: defining what these probes read, what they do not, and how to distinguish the two.
Core Insight: An instrument can be valid for one type of comparison (e.g., tracking a single model's training) and empty for another (e.g., comparing different model families).
Methodological Shift: The author argues that quantities derived from self-feeding loops must be subjected to the "discriminator test" (varying construction vs. varying model) before being interpreted as properties of the language model.
Reproducibility: All code, data, and figure scripts are released with strict traceability (SHA-256 hashing) to ensure that figures cannot outlive the code that produced them.

The paper concludes that the "neat summary" that the instrument measures itself is false; the instrument does read the model, but it also reads the probe with equal precision. The cost of failing to distinguish them is the misattribution of probe artifacts (like the manufactured phase transition) to the model.