🤖 machine learning

Final Checkpoints Are Not Enough: Analyzing Latent Reasoning Faithfulness Along Training Trajectories

This paper reveals that the faithfulness of latent reasoning methods is not static but dynamically evolves during training, with causal contributions to final answers decaying over time and varying significantly by answer format, highlighting that final checkpoint evaluations alone are insufficient to assess reasoning reliability.

Original authors: Hengyu Jin, Shu Yang, Di Wang

Published 2026-07-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hengyu Jin, Shu Yang, Di Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant student taking a very difficult math test. There are two ways this student can solve the problems:

The "Show Your Work" Method (Chain-of-Thought): The student writes down every single step on a piece of paper. You can read their notes to see exactly how they got the answer.
The "Silent Thinking" Method (Latent Reasoning): The student solves the problem entirely in their head. They don't write anything down; they just whisper the final answer. This is faster and more compact, but you can't see how they did it.

This paper is about the second method. The big worry is: Is the student actually thinking, or are they just guessing? Maybe they memorized the answer key, or maybe they are using a shortcut that skips the actual thinking steps. If you poke their brain (or the computer code behind it) to see if the "silent thinking" steps are actually necessary, do they still get the right answer?

The researchers call this "faithfulness." If the silent steps are faithful, they are the cause of the answer. If they aren't faithful, the student is just pretending to think while the real answer comes from somewhere else.

The Big Mistake Everyone Made Before

Previous studies looked at these "silent thinkers" only after they finished their training (the final exam). They found that many of these models were "unfaithful"—you could mess up their silent thinking steps, and they would still get the right answer.

The authors of this paper say: "Wait a minute! We only looked at the finish line. What happened during the race?"

They decided to watch the models train from Day 1 to the very end, like watching a sports game instead of just looking at the final scoreboard. They used two special tools to test the models:

The "What If" Trick: They changed a tiny detail in the question (like changing a road in a map) to see if the model changed its answer. If the model is faithful, it should change its answer.
The "Brain Fog" Test: They temporarily "fogged up" the model's silent thinking steps with random noise to see if the model could still solve the problem.

What They Found (The Three Big Surprises)

1. Two Different Roads to the Same Dead End

They studied two different types of "silent thinkers" (let's call them Method A and Method B).

The Result: At the very end, both methods looked terrible. They both failed the "What If" trick and the "Brain Fog" test. They seemed equally unfaithful.
The Twist: The journey to that failure was completely different.
- Method A started out being very faithful. It was good at changing its answer when the question changed. But then, halfway through training, it suddenly stopped caring. It got lazy and started ignoring the thinking steps, even though its final score kept going up.
- Method B never cared about the thinking steps in the first place. It was unfaithful from Day 1 and stayed that way.
The Lesson: If you only look at the final exam, you think they are the same. But if you watch them train, you see one was a "late bloomer" who gave up, and the other was a "cheater" from the start.

2. The "Thinking" Steps Are Fading Away

They checked the model's internal "brain activity" (the hidden states) during the silent thinking.

The Result: As the models trained, the actual importance of those silent thinking steps dropped.
The Analogy: Imagine a chef cooking a soup. At first, the chef tastes the soup and adds salt (the thinking step). But as the chef gets more experienced, they stop tasting and just throw in a random pinch of salt because they think it's right. The paper found that the "tasting" part of the process became useless over time. The model stopped needing the thinking steps to get the right answer; it just started guessing the answer directly.
The Connection: The specific questions where the model stopped changing its answer (the "What If" trick) were the exact same questions where the "thinking steps" stopped mattering.

3. The Format of the Answer Changes Everything

This is the most surprising part. They tested the models on two types of questions:

Type A: "Is the answer X or Y?" (Binary Choice)
Type B: "What is the answer?" (Open-Ended)

The Magic Switch:

When the model had to pick between two options (A or B), the "thinking steps" became useless as training went on (just like in the previous tests).
But when the model had to write out the answer (Open-Ended), the "thinking steps" actually became more important as training went on!

The Analogy: It's like a student who learns to guess the right multiple-choice answer by looking at the pattern of the bubbles on the sheet (ignoring the math). But if you ask them to write the answer on a blank page, they have to do the math. The way you ask the question changes whether the model is actually thinking or just guessing.

The Bottom Line

The paper concludes that you cannot judge if a "silent thinker" AI is honest or faithful just by looking at its final test score.

Timing matters: A model might look faithful early on and then lose that ability, or vice versa.
Format matters: A model might be faithful when writing an essay but unfaithful when picking a multiple-choice answer.

The "faithfulness" of these AI models isn't a fixed trait of the machine; it's a moving target that depends on how it was trained and how you ask it the question.

Technical Summary: Final Checkpoints Are Not Enough: Analyzing Latent Reasoning Faithfulness Along Training Trajectories

Problem Statement
Latent reasoning methods perform multi-step inference entirely within a model's continuous hidden states, offering a more compact and efficient alternative to explicit Chain-of-Thought (CoT) reasoning, which relies on discrete natural language tokens. However, the opacity of these hidden states raises a critical question of faithfulness: do the latent reasoning steps causally drive the final answer, or are they merely correlated with the output?

Prior work has investigated this question only at converged checkpoints, reporting that latent reasoning steps are often unfaithful—meaning they can be replaced, perturbed, or erased without altering the final prediction. A significant gap remains in understanding how these unfaithful behaviors form during training. Furthermore, treating a trained model as a fixed endpoint obscures whether different training paradigms reach similar unfaithful conclusions via qualitatively different trajectories.

Methodology
The authors track the evolution of faithfulness across saved checkpoints for two distinct latent reasoning paradigms, COCONUT (staged curriculum) and CODI (self-distillation), using a shared GPT-2 backbone. The analysis employs two primary interventions:

Verified Counterfactual Edit (Input Level): Using the ProsQA benchmark (a graph reachability task solvable by a deterministic BFS oracle), the authors construct 405 paired inputs. They minimally edit a single edge in the underlying graph to flip the oracle answer from a target $A$ to an alternative $B$ . Faithfulness is measured by whether the model's prediction follows this edit (Contrast Consistency and Conditional Flip Rate).
Noise-Ablation Activation Patch (Activation Level): Adapted from causal-mediation analyses, the authors patch all latent reasoning steps with content-erasing values (zero, dataset mean, or norm-matched Gaussian noise). They measure the Output Change Rate (OCR) and the Preserved-When-Correct (PWC) rate to determine if the latent steps causally contribute to the prediction.

The study addresses three Research Questions (RQs):

RQ1: Do different paradigms exhibit qualitatively different output-level faithfulness trajectories, even if their final checkpoints appear similar?
RQ2: Does the causal contribution of latent steps decay across training, and does this align with output-level transitions?
RQ3: Do these trajectories depend on the answer format (binary choice vs. open-ended decoding)?

Key Results

Divergent Trajectories to Similar Endpoints (RQ1): While both COCONUT and CODI converge to low-faithfulness endpoints (low Contrast Consistency and Conditional Flip Rate), their training trajectories differ significantly.
- COCONUT initially exhibits high counterfactual faithfulness (following edits) but undergoes an abrupt shift at the curriculum boundary (epoch 15–16), where faithfulness collapses and the model begins retaining the original answer despite edits.
- CODI enters a "retention" regime (low faithfulness) almost immediately from the earliest checkpoints and remains there, never visiting the counterfactual-following phase.
- Implication: A single snapshot at convergence masks these fundamentally different learning dynamics.
Decay of Causal Contribution (RQ2): At the activation level, the causal contribution of latent reasoning steps decays across training for both paradigms.
- The Output Change Rate (OCR) falls toward zero, and PWC rises toward one, indicating that noise ablation on latent steps no longer affects the output.
- Cohort analysis reveals that the specific examples which flip from counterfactual-following to retention in the output (the "drift" cohort) are precisely the examples where the causal contribution of the latent steps decays.
Format Dependence (RQ3): The activation-level trajectory is not intrinsic to the model architecture but depends heavily on the answer format.
- On binary-choice tasks (ProsQA and GSM-Choice), the causal contribution of latent steps decays across training (OCR decreases).
- On open-ended decoding tasks (GSM-open), the causal contribution rises across training (OCR increases).
- Additionally, the ranking of model performance (COCONUT vs. CoT) flips depending on whether the task is framed as open-ended or binary choice, suggesting answer format alters both apparent performance and measured faithfulness.

Significance and Claims
The paper argues that latent reasoning faithfulness is a property of the training stage and the answer format, not merely a static attribute of the final architecture or checkpoint.

Methodological Contribution: The work demonstrates that evaluating latent reasoning solely at convergence is insufficient. It reveals that qualitatively different training dynamics can lead to similar unfaithful endpoints, and that faithfulness metrics can reverse direction based on output format.
Interpretation of Unfaithfulness: The findings suggest that in binary-choice settings, latent reasoning steps may develop a "removable answer bias" rather than performing necessary computation, whereas in open-ended settings, they may retain or increase their causal role.
Limitations: The authors note that their verified counterfactual edits rely on deterministic oracles (like BFS solvers), limiting immediate application to tasks without such ground-truth verification. The study is also scoped to the GPT-2 small backbone to ensure tractable reproduction of full training trajectories.

In conclusion, the paper posits that understanding latent reasoning requires analyzing the trajectory of learning and the format of the output, rather than relying on final-checkpoint evaluations which may obscure critical shifts in model behavior.