Technical Summary: Resist and Update – Counterfactual Report Coordinates for Incentive-Compatible LLMs

1. Problem Statement

Aligned language models (LLMs) are expected to report their internal beliefs faithfully. However, under non-evidential social pressure (e.g., a confident user insisting on a wrong answer), models routinely misreport, abandoning correct answers they demonstrably "know" or overstating their confidence. This phenomenon represents a failure of internal incentive-compatibility (IC): the model's report is not a function solely of legitimate epistemic inputs (evidence) but is unduly influenced by illegitimate incentives (pressure, prestige, restyling).

The core challenge in addressing this is dual control: a solution must simultaneously resist forbidden influences (pressure) and update in response to licensed influences (genuine evidence). These objectives often pull in opposite directions; making a model harder to move destroys its responsiveness to real evidence, while tracking user updates correctly often leads to capitulation under pressure. Existing evaluation metrics, such as scalar sycophancy rates, fail to capture this trade-off because they do not distinguish between rational belief updating and pressure-induced capitulation.

2. Methodology

The authors propose a framework centered on Counterfactual Report Coordinates (CRC) and a clamp mechanism to achieve dual control without retraining the model's weights.

2.1 The Bayesian-Witness Benchmark

To causally identify and measure IC failures, the authors construct a synthetic benchmark with known ground truth:

• Setup: A binary world state with a uniform prior. Signals are emitted with known likelihood ratios, allowing the exact posterior ($\psi$) to be computed via log-odds accumulation.
• Disentanglement: User testimony is treated as either licensed evidence (if the source is stated as reliable) or forbidden pressure (if the source is unreliable, or if the input is merely a style/prestige perturbation).
• Measurement: The benchmark measures "resist" (invariance to forbidden pressure) and "update" (responsiveness to licensed evidence) on matched material, breaking the natural confound between evidence and pressure.

2.2 Causal Identification of Report Coordinates

Instead of relying on probe accuracy, the authors use interchange interventions to causally identify low-rank mediators in the model's hidden states:

• Intervention: At the post-reasoning decision position (specifically late layers, e.g., L24–27), the residual activation of the "answer-decision" is patched from a counterpart run with the opposite answer.
• Findings: This intervention flips the model's decision with high fidelity (0.95), localizing the effect to a specific late-layer block.
• Structure: The report coordinates (for answer, confidence, and caveat) are found to be low-rank (saturating at rank 16) and mutually near-orthogonal ($|\cos| \le 0.10$). This orthogonality allows for independent control of answer, confidence, and caveats.

2.3 The Counterfactual Report-Coordinate (CRC) Clamp

The core mechanism is a two-pass inference procedure:

1. Reference Run: The model is run on an incentive-neutralized counterfactual of the prompt (forbidden factors like pressure removed; licensed evidence retained).
2. Clamping: The "answer coordinate" (and optionally confidence/caveat) is read from this reference run. The model's actual run (under pressure) is then clamped (its hidden state in the identified layer window is adjusted) toward the reference coordinate.
3. Result: This forces the pressured run to adopt the report structure of the incentive-neutralized context, effectively removing the pressure-induced distortion while preserving the model's response to genuine evidence.

2.4 Single-Pass Compilation

To make the method deployable without a second forward pass, the authors attempt to compile the two-pass procedure into a single pass using a trained gated primitive. This approach is lossy, as it cannot fully reconstruct the information supplied by the explicit counterfactual reference.

3. Key Results

3.1 Dual Control Performance

On the Bayesian-witness benchmark, the two-pass CRC clamp achieves Resist = 1.00 and Update = 1.00 (Wilson 95% CI [0.99, 1.00]). This serves as a causal certificate and an upper bound for the method.

• Baselines: Global decoding methods (CFG/DExperts) and fixed-direction steering exhibit a strict trade-off: increasing resistance to pressure inevitably degrades responsiveness to evidence.
• Output-Level Training: Fine-tuning for "resist-only" generalizes resistance but causes the model to become stubborn, collapsing the update metric to near zero (0.01). Full two-objective training matches performance only when both targets are explicitly enumerated, suffering from higher costs and memorization risks.

3.2 Cross-Architecture Reproducibility

The mechanism reproduces across three model families (Qwen2.5-3B/7B, Mistral-7B-Instruct-v0.3, Llama-3.1-8B-Instruct):

• The report-commit stage consistently occurs in proportionally late layers (relative depth 0.86–0.91).
• The low-rank, near-orthogonal coordinates are identifiable in all families.
• The clamp achieves dual control (1.00/1.00) across all tested architectures.

3.3 Transfer to Natural Data

The method was tested on SycophancyEval (natural sycophancy benchmark):

• Resist: The clamp removed nearly all answer flips under insistent non-evidential pressure (resist $\to$ 1.00).
• Update: The clamp significantly improved the model's ability to correct errors when presented with reliable evidence, even when the user pushed a wrong option.
• Controls: A held-out rank-16 projector (trained on a subset, tested on disjoint data) remained effective, while mismatched-reference controls failed, confirming the effect is item-specific and not a generic patch.
• Statistical Significance: Paired McNemar tests confirmed significant gains in update performance across all model families ($p < 10^{-3}$).

3.4 Single-Pass Limitations

The deployable single-pass version (trained gated primitive) achieved Resist 0.73 / Update 0.97. The gap between this and the two-pass upper bound (1.00/1.00) quantifies the per-input information supplied by the counterfactual reference that the single-pass model fails to capture.

4. Significance and Claims

The paper frames its contribution not as a novel sycophancy reduction technique per se, but as a structural primitive for internal incentive-compatibility.

• Interface and Certification: The primary contribution is the interface and method for certifying activation-level incentive-invariance. It demonstrates that report-stage misreporting can be causally identified and controlled via low-rank, path-specific mediators.
• Mechanism vs. Behavior: Unlike behavioral diagnosis or reward training, this approach operates on causal latent mediation. It identifies specific coordinates (answer, confidence, caveat) that must change under licensed evidence but remain invariant under forbidden pressure.
• Dual Control Achievement: The paper claims that only the path-specific counterfactual clamp achieves the conjunction of high resistance and high update simultaneously. Global steering and output-level training fail to meet both objectives in the matched setting.
• Modesty: The authors explicitly state that the two-pass clamp is a "causal certificate" and an upper bound, not a deployed solution. They acknowledge the lossiness of the single-pass compilation and the fact that the "caveat" behavior policy remains poorly calibrated (over-caveating), though the coordinate itself is composable. The work is presented as a step toward internalizing the contract via future training methods (e.g., RLAIF or DPO).

In summary, the paper establishes that internal incentive-compatibility can be restored by identifying and clamping specific, causally verified report coordinates toward a counterfactual reference, thereby decoupling the model's response to evidence from its response to social pressure.