🤖 machine learning

Behaviour Is an Incomplete Measure of Reasoning Development: Cross-surface pre-arrival accessibility and the limits of developmental inference in a recurrent-depth reasoner

This paper demonstrates that in a recurrent-depth reasoner, internal hidden-state probes can detect reasoning capabilities long before they manifest in observable behavior, revealing that behavioral thresholds and decoder accessibility are distinct, incomplete measures of the actual developmental timeline of reasoning.

Original authors: Simon Lam-Muir

Published 2026-08-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Simon Lam-Muir

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving field of artificial intelligence, researchers often face a fundamental question: when does a machine actually learn something? For years, the standard answer has been behavioral. If a computer program can correctly answer a question it has never seen before, we assume it has acquired the reasoning skill required to do so. We look at the output, the final answer, and declare the capability present. However, this approach treats the mind of the machine as a black box, assuming that the moment the correct answer appears, the internal process of figuring it out must have just happened. This paper challenges that assumption by looking inside the machine while it is still learning, asking whether the ability to solve a problem exists internally long before the machine is capable of showing it on the outside. The researchers are not just watching what the machine does; they are measuring what the machine knows, even when it cannot yet prove it.

To understand the stakes, imagine trying to learn a new language. You might spend months memorizing vocabulary and grammar rules, yet still fail to hold a conversation. A teacher watching only your speech might conclude you have learned nothing. But if you could peek at your brain, you might find that the neural pathways for the language are already forming, waiting for a final spark to connect them. This is the gap the researchers investigated. They built a specialized computer model designed to solve logical puzzles involving chains of relationships, such as figuring out who is related to whom in a family tree. They trained this model using two different methods: one where the relationships were described using abstract, meaningless symbols, and another where the same relationships were described using English-like sentences. The goal was to see if the way the information was presented changed how quickly the model learned, and whether the model's internal state revealed knowledge that its behavior did not.

The results were striking and counterintuitive. When the model was trained on the abstract symbols, it mastered the three-step logical puzzles in just seventy training cycles. When the exact same model, with the exact same training plan, was trained on the English-like sentences, it took over thirteen thousand cycles to reach that same level of competence. That is a difference of more than one hundred and eighty times. Yet, once the sentence-trained model finally cracked the three-step puzzle, it solved the four-step version in just eight cycles. This suggests that the model spent thousands of cycles grinding through a difficult phase, seemingly making no progress, before suddenly unlocking the ability to solve harder problems. If a researcher had only watched the final answers, they would have seen a machine that failed for a long time and then suddenly succeeded, missing the entire hidden journey of internal development that happened during the struggle.

To see what was happening during those long, difficult cycles, the researchers looked inside the model's hidden states—the internal representations of the data that the machine uses to think. They used a simple tool, a linear probe, which acts like a decoder, to check if the correct answer was already present in the machine's mind before the machine actually outputted it. On the sentence-trained surface, this probe could detect the identity of the future answer with a small but reliable signal long before the model could answer correctly on its own. The probe found that information about the answer was accessible internally, even though the model could not yet express it. This internal accessibility was not a fluke; when the researchers switched back to the abstract symbols, they found the same pattern of early accessibility at specific points in the machine's processing. The machine was holding information about the answer internally, waiting for the right moment to release it.

However, the study also uncovered a significant limitation in how we try to measure this kind of learning. The researchers wanted to track exactly how this internal knowledge grew stronger over time, creating a map of the machine's learning curve. They quickly realized, however, that this specific measurement was impossible to perform accurately. The problem was that the group of questions the machine was being tested on kept changing. As the machine learned, it got better at answering certain questions, and those questions were removed from the test group. This meant that the very act of measuring the machine's progress was changing the population of questions being measured. It was like trying to measure how fast a runner is improving by only timing them on the tracks they haven't finished yet; as they get faster, the tracks change, making it impossible to compare their speed fairly over time. The researchers proved that this specific type of measurement is fundamentally flawed in this context, not because the machine didn't learn, but because the method of tracking it was broken.

The paper concludes that we cannot rely on a single way of knowing what a machine has learned. Behavioral success, the ability to give the right answer, is just one observable fact. Internal accessibility, the ability to detect the answer inside the machine, is another. And the actual process of development over time is a third, which often cannot be measured directly if the measurement method itself alters the conditions. The researchers found that a machine can possess a capability internally long before it displays it, and that the path to that capability can look completely different depending on how the problem is presented. Most importantly, they showed that simply observing the machine, no matter how closely, is not enough to understand the computation it has acquired. To truly know what the machine has learned, we must move beyond observation and begin to intervene, testing the machine's internal states directly to see if they are the actual cause of its behavior. Until then, the internal life of these systems remains partially hidden, even when they are finally getting the right answers.

Technical Summary: Behaviour Is an Incomplete Measure of Reasoning Development

Problem Statement

The paper addresses a fundamental "measurement problem" in developmental interpretability: the assumption that a model's acquisition of a capability can be uniquely identified by three common metrics—(1) behavioural thresholds (held-out accuracy crossing a specific point), (2) internal accessibility (a probe or decoder successfully recovering a property from hidden states), or (3) retrospective analysis of final checkpoints.

The author argues that these three quantities need not identify the same event. When they disagree, the disagreement does not reveal which metric corresponds to the actual computation the model acquired. The paper posits that neither behavioural performance nor decoder accessibility alone is sufficient to identify the specific computation training has acquired, necessitating a shift toward causal intervention.

Methodology

The study utilizes a controlled, closed-world setting to mechanically verify ground truth, avoiding the estimation errors inherent in natural language datasets.

Environment: An oracle-defined relational world with a fixed set of entities and relations. $k$ -hop queries are defined by chains of relations, with answers mechanically known. Evaluation items are constructed with verified zero overlap against training items.
Model: A 30M-parameter recurrent-depth transformer. It features a 768-dimensional state and a 4-layer block applied repeatedly within a single forward pass, decoupling computational depth from parameter count.
Curriculum: Training proceeds via a hop-depth curriculum. A stage $k$ trains on items of depth $\le k$ and clears when held-out accuracy strictly exceeds 0.95.
Experimental Design (Two Surfaces): The same relational task family is presented via two distinct surfaces while holding the world, architecture, curriculum, optimizer, and acquisition criterion constant:
1. Symbolic Surface: Entities and relations are opaque tokens.
2. Verbal Surface: Queries are expressed as English-like sentences with nested genitive structures.
  Crucially, these are separate trained realizations, not a single model evaluated twice.
Measurement Framework:
- Separation of Axes: Training time (parameter updates) and inference time (recurrent iterations) are kept distinct.
- Probes: Linear classifiers fitted on hidden states at preregistered, event-relative positions.
- Controls: Every probe result is benchmarked against: (a) uniform chance (1/40), (b) a population frequency baseline, and (c) an untrained "zero-step" control (a deterministic reconstruction at initialization).
- Evaluability Checks: A mechanical proof is required before inference to ensure the population supports the estimand. If the population changes as a function of the measurand, the measurement is deemed non-identifiable.

Key Contributions and Results

1. Behavioural Development Has Hidden Structure

Changing the training surface while holding all other variables fixed produced radically different developmental trajectories:

Acquisition Cost Disparity: Achieving 3-hop competence cost 70 logical epochs on the symbolic surface but 13,055 logical epochs on the verbal surface—a contrast of 186.5×.
Trajectory Shapes: The symbolic surface showed a "staircase" of small steps. The verbal surface exhibited a "long grind" (13,055 epochs) followed by a "rapid race" (8 epochs to clear 4-hop).
Later-Depth Behaviour: During the 13,055-epoch verbal grind for 3-hop tasks, held-out 4-hop and 5-hop accuracy remained at floor levels (peaking at 3/40 and 2/40 respectively, ending at 0/40). This demonstrates that a single behavioural threshold collapses complex developmental shapes into a single point, obscuring the route taken.

2. Pre-Arrival Internal Accessibility

Internal measurements revealed information accessible before behavioural expression:

Verbal Surface: A linear probe fitted before the model could answer correctly recovered future-answer identity with an accuracy of 0.056159 (vs. 0.025 chance, $p=0.012987$ ). This signal was weak and heterogeneous (16/40 classes contributed).
Symbolic Surface: Analogous pre-arrival accessibility survived the surface change. At a primary upstream position, probe accuracy reached 0.1020 (vs. 0.0460 control, $p=0.000999$ ), with 21/40 classes contributing.
Interpretation: "Ignition" (decisive behavioural expression) is not the first appearance of answer-relevant information; rather, it is the moment information that was already weakly accessible internally becomes decisively readable. However, accessibility does not imply causal use.

3. Limits of Longitudinal Measurement (Identifiability Failure)

The paper demonstrates a structural failure in attempting to track the development of this accessibility over time:

The Problem: Probe eligibility is defined by behavioural arrival (items are eligible only while the model has not yet answered them correctly). As training proceeds, items improve and leave the eligible population.
The Consequence: The population being measured changes as a function of the measurand. A probe-accuracy curve over checkpoints would conflate representational change with population migration.
Mechanical Proof: The author proved that no fixed panel of items (covering all 40 classes) exists across any contiguous window of checkpoints. The whole-stage intersection contains zero items.
Outcome: The developmental accessibility curve is not cleanly evaluable. The instrument's refusal to emit a number is itself informative, indicating that the design cannot answer the question, rather than the accessibility failing to develop.

Significance and Claims

The paper does not claim to identify the specific computation acquired by the model. Instead, its significance lies in establishing the insufficiency of observational metrics:

Decoupling Observables: It separates three distinct observables—behavioural competence, inference-time accessibility, and training-time development—showing they can diverge significantly.
Surface Dependence: It demonstrates that surface representation (symbolic vs. verbal) fundamentally alters the cost and shape of learning, even when the underlying task and architecture are identical.
Evaluability as a Prerequisite: It argues that evaluability must be proven before measurement. A measurement that cannot be validly estimated due to population selection bias is a failure of the design, not a null result.
Call for Causal Intervention: The paper concludes that observation, no matter how refined (probes, decoders), cannot identify the acquired computation. The necessary next step is causal intervention (manipulating intermediate states to test if they causally drive the computation), rather than developing better decoders.

The work serves as a methodological critique, showing that "richer" internal observation can encounter identification failures when the population eligible for measurement is defined by the behavioural event being studied.