Confidence and Calibration of Activation Oracles for Reliable Interpretation of Language Model Internals

Technical Summary: Confidence and Calibration of Activation Oracles for Reliable Interpretation of Language Model Internals

Problem Statement
Activation oracles are language models fine-tuned to translate a target model's hidden states into natural language descriptions, enabling the interpretation of internal representations such as memorized facts, hidden objectives, or secret words. While these oracles have demonstrated promise in recovering information that surfaces only in activations, their outputs currently lack a notion of certainty. This absence of calibrated confidence is critical for downstream actionable decisions, such as alignment auditing, deception detection, and the elicitation of hidden objectives. Standard uncertainty quantification (UQ) techniques, developed for un-steered models, have not been rigorously evaluated on activation oracles, which operate as "steered" models where the residual stream is overwritten at inference time with vectors from a target model. The central problem addressed is identifying which confidence estimation methods remain valid and well-calibrated under this specific steering perturbation.

Methodology
The authors benchmarked six distinct UQ methods on two activation oracles: a released Qwen3-8B oracle and a newly trained Qwen3.6-27B oracle (adapted for a hybrid linear-plus-full attention architecture). The evaluation utilized the "secret-word taboo" task, where target models are fine-tuned to keep one of 20 specific words secret. For each of the 6,000 samples per oracle (comprising 20 target words, 100 context prompts, and 3 verbalizer prompts), the oracle attempts to decode the secret word from the injected hidden state.

The six evaluated methods were:

1. Log-probability (Baseline): The joint token log-probability of the predicted answer word.
2. Temperature Bootstrap: Sampling $k=20$ answers at varying temperatures ($T \in \{0.3, \dots, 1.5\}$) and using the empirical frequency of the modal answer as confidence.
3. Direct Numeric Self-Report: Prompting the oracle to output a numeric confidence score (0–100) after generating the answer.
4. MCMC Power-Sampling Acceptance: Using a single block Metropolis-Hastings (MH) chain with power sampling ($\alpha > 1$) and measuring the empirical acceptance rate.
5. MCMC Power-Sampling Agreement: Running $k=10$ independent MH chains and measuring cross-chain agreement (analogous to self-consistency).
6. Steering Sensitivity: Sweeping the steering coefficient ($c$) around the default value and measuring the stability of the decoded answer.

Performance was measured using Accuracy, Expected Calibration Error (ECE), Brier Score, Negative Log-Likelihood (NLL), and AUROC (ranking ability).

Key Results

• Best-Performing Method: Temperature bootstrap mode frequency emerged as the best-calibrated method on both models. On Qwen3-8B, the optimal temperature was $T=1.0$ (ECE 5.7%), and on Qwen3.6-27B, it was $T=1.5$ (ECE 10.3%). The optimal temperature correlates with task accuracy; as the model becomes less accurate (harder task), a higher temperature (flatter sampling) yields better calibration.
• Log-probability as Triage: While log-probability had higher ECE than the optimal bootstrap configuration, it achieved high AUROC (ranking capability) and serves as a fast, low-cost triage signal, especially when post-hoc calibration (e.g., isotonic rescaling) is applied.
• Failure of Self-Report: Free-form numeric self-report was the worst-calibrated method. On the 27B oracle, it was "anti-calibrated," with the model expressing higher confidence in wrong answers (mean 98.9%) than correct ones (97.6%). This replicates the "probe-vs-verbalized" confidence gap observed in standard LLMs, suggesting the natural language elicitation channel is dominated by training priors rather than the activation-grounded signal.
• Failure of MCMC Acceptance: The single-chain MCMC acceptance ratio failed to distinguish between correct and incorrect outputs (AUROC ~0.53–0.60). The authors attribute this to the steered conditional distribution being sharply peaked at the greedy answer; consequently, the MH chain accepts proposals regardless of correctness, saturating the acceptance rate.
• Steering Sensitivity: This method produced coarse, discrete confidence values (0.2, 0.4, etc.) and poor calibration (ECE > 40%), rendering it unsuitable as a probabilistic signal, though it may serve as a binary diagnostic.

Contributions

1. First UQ Benchmark for Activation Oracles: The paper establishes the first systematic evaluation of uncertainty quantification specifically for steered activation oracles, covering six methods across two models and four calibration metrics.
2. Evaluation of Novel Methods: It introduces and evaluates three methods designed specifically for the steered-oracle setting (MCMC acceptance, MCMC agreement, and steering sensitivity), providing mechanistic explanations for why MCMC acceptance and steering sensitivity yield negative results in this context.
3. Practical Recipe: The authors propose a practical tuning strategy: select the bootstrap temperature such that the mean mode frequency on a held-out word slice matches the empirical accuracy on that slice.
4. Replication of Confidence Gaps: The study confirms that the gap between internal activation signals and verbalized confidence exists in activation oracles, with free-form self-report failing to capture the uncertainty signal present in the hidden states.
5. New Oracle Release: The paper releases the first activation oracle for a hybrid linear-plus-full attention architecture (Qwen3.6-27B), along with the patched trainer and verbalizer weights.

Significance and Claims
The paper claims that for practitioners building systems on activation oracles, temperature bootstrap mode frequency is the production-ready confidence signal, with the temperature tuned to match empirical accuracy. While log-probability is less calibrated, it remains a viable, low-cost alternative for triage if combined with post-hoc rescaling.

The authors explicitly state that MCMC acceptance ratios are not suitable confidence signals for this task due to the mode-peaked nature of the steered posterior. Furthermore, they caution that free-form numeric self-report is unreliable and anti-calibrated in this setting. The study suggests that if introspective UQ is required, constrained label scoring (bypassing free-text generation) may recover the signal, as demonstrated in a small pilot study.

The work is modest in its scope, noting that results are based on two Qwen-family models and one training recipe. While the consistency across models suggests a paradigm-level claim, the authors do not assert generality across all architectures. The findings emphasize that reliable interpretation of language model internals requires not just decoding the hidden state, but rigorously quantifying the confidence of that decoding, for which bootstrap methods currently offer the most robust solution.