Technical Summary: Decodable but Not Faithful: Coupling Natural-Language Rationales to Programmatic Verifiers

Problem Statement

While large language models (LLMs) can generate plausible natural-language rationales for their predictions, these explanations often fail to faithfully represent the model's internal decision-making process. Models may "rationalize" decisions post-hoc, omit the true source of an answer, or utilize shortcuts invisible in the generated text. Prior work indicates that while chain-of-thought prompting improves task accuracy, the generated explanations are not necessarily evidence of causal reasoning.

External programmatic verifiers (e.g., theorem provers, game engines, code executors) offer a mechanism to check claims, but current usage is typically limited to outcome-level supervision (checking the final answer). This leaves a gap: a model may produce a correct claim while the accompanying explanation is unsupported, or encode verifier information internally without verbalizing it coherently. The central question addressed is whether forcing a model's rationale representations to be decodable by a verifier guarantees that the generated explanations are faithful to the model's reasoning.

Methodology: Verifier-Coupled Reasoning

The paper proposes a framework called Verifier-Coupled Reasoning to narrow the gap between verified claims and natural-language rationales.

Sequence Format and Objective

The model generates a sequence containing a natural-language rationale ($r$) and an explicit inline claim ($c$) in the format: [CONTEXT] x [RAT] r [CLAIM] c [EOS]. A programmatic verifier $V$ provides a target label $y = V(x)$ (e.g., proof accept/reject, win-rate bucket, complexity class).

An auxiliary consistency head (a linear classifier) is trained to predict the verifier's target $y$ from the mean-pooled hidden states of the rationale span ($z_{rat}$). The training objective combines standard next-token prediction ($L_{LLM}$), optional claim supervision ($L_{claim}$), and the consistency loss ($L_{cons}$):
$$L = L_{LLM} + \lambda_{claim}L_{claim} + \lambda_{cons}L_{cons}$$

Information-Flow Constraints

To ensure the claim head cannot bypass the rationale, the authors employ a Board-Explanation-Claim (BEC) attention mask in specific experiments. This structurally prevents claim tokens from attending to input context tokens, forcing them to rely solely on the explanation span.

Diagnostic Ladder

The paper introduces a multi-level diagnostic framework to distinguish between decodability (verifier information is present in the representation) and faithfulness (the generation reflects that information):

1. Decodability: Consistency-head accuracy (can the verifier label be linearly decoded from rationale states?).
2. Counterfactual Following: Do swapped spans (e.g., swapping a rationale with a different label) change the head's prediction?
3. Causal Influence: Does activation patching (replacing rationale hidden states with those from a different latent state) shift the downstream claim logits?
4. Generation Quality: Does the model produce coherent, faithful explanations?

Experimental Settings

The framework is evaluated across five distinct settings:

1. Synthetic: Controlled validation with latent states to isolate mechanisms and test causal interventions via activation patching.
2. LeanCheck: Formal theorem proving using Lean 4. Tests span-specific coupling (rationale vs. proof) under counterfactual conflicts.
3. KataGo: A Go engine providing dense, non-binary labels (win-rate buckets) for complex board states.
4. Code Coupling: Algorithmic claims (time/space complexity, algorithm class) where the failure mode of unfaithful generation is most visible.
5. FEVER: Real-language evidence verification, testing the trade-off between raw accuracy and evidence sensitivity.

Key Results

1. The Decodability–Faithfulness Gap

The central finding is that decodability does not guarantee faithfulness.

• Synthetic & LeanCheck: Consistency training successfully makes verifier information decodable and causally influential. In LeanCheck, rationale-only and proof-only pooling achieve perfect directional separation under counterfactual conflict (100% following the swapped span's label).
• KataGo: Natural-language commentary spans encode 10-way win-rate buckets with 81.1% accuracy, demonstrating the method scales to dense, non-binary domains.
• Code (The Critical Failure Mode): In the code setting, the model achieves 98.6% representation coupling (the hidden states perfectly encode the verifier claims), yet the generated prose remains unfaithful. The model emits fluent, grammatical explanations describing an unrelated algorithm (e.g., describing a "sum of a list" while the code implements "selection sort") while correctly emitting structured claim tags (e.g., $O(n^2)$).
  • This gap is structural, not capacity-driven: identical results are observed in both pretrained GPT-2 and models trained from scratch.
  • The gap is not weight-driven: Increasing the consistency loss weight ($\lambda_{cons}$) up to $10^4$ does not improve generation quality, as the consistency loss acts on representations, not emitted tokens.

2. Diagnostic Ladder Insights

• Causal Influence: In synthetic settings, activation patching of rationale states at layer 0 shifts claim behavior in 73–89% of consistency-trained cases (vs. 31% baseline), confirming that the coupling is causal, not merely correlational.
• FEVER Trade-off: In real-language settings, pooling over the full sequence (including the claim text) yields near-perfect accuracy (99%) but fails to isolate evidence sensitivity. Evidence-only pooling yields lower accuracy (44%) but better isolates whether the model actually processes the evidence, revealing an accuracy-sensitivity trade-off hidden by standard evaluation.

3. Per-Claim Analysis

In the code setting, consistency loss disproportionately benefits fine-grained claims (e.g., algorithm class, key operation) over binary ones. Binary claims are partially learnable from code structure alone, whereas fine-grained claims rely heavily on the consistency objective for high accuracy.

Significance and Claims

The paper establishes that consistency losses are effective diagnostics and representation-shaping tools, but they are not sufficient conditions for faithful reasoning.

• Representation vs. Generation: The mechanism successfully shapes the internal representation to encode verifier-relevant information (decodability), but without a generation-level objective that rewards the text for matching the verifier, the model can emit fluent but unfaithful explanations.
• Limitations of Probing: High probe accuracy (decodability) should not be equated with faithful reasoning. A model can "know" the answer in its hidden states while "saying" something entirely different in its output.
• Path Forward: Closing the gap requires objectives where the gradient reaches the generated tokens and rewards them for verifier consistency. The authors suggest Reinforcement Learning with Verifiable Rewards (RLVR) as the natural instrument, where the model is rewarded based on the checkable inline claims it actually produces, rather than an auxiliary probe.

The work provides a rigorous diagnostic ladder to evaluate reasoning systems, demonstrating that while we can force models to "know" the truth in their representations, ensuring they "say" the truth requires more than just representation-side constraints.