Technical Summary: Bridging the Knowledge-Prediction Gap in LLMs on Multiple-Choice Questions

1. Problem Definition

Large Language Models (LLMs) frequently exhibit a "knowledge-prediction gap," a phenomenon where a model possesses the correct answer within its internal hidden representations but fails to generate it as the final output. This discrepancy is particularly evident in Multiple-Choice Question (MCQ) settings, where models may select incorrect options despite linearly encoding the correct answer in their residual streams. Prior work has attributed such failures to superficial cues, option biases, or stylistic artifacts, but a unified explanation connecting these failures to the model's internal geometric structure has remained unexplored. Specifically, it is unclear whether this gap is pervasive across diverse domains, what geometric structure underlies the misalignment between internal knowledge and output behavior, and whether this gap can be mitigated at inference time without retraining.

2. Methodology

The authors propose a three-step analysis and intervention framework:

A. Quantifying the Gap via Linear Probing

The authors define the knowledge-prediction gap as the discrepancy between knowledge encoded in internal representations and the predictions generated. To measure this, they train two distinct linear probes on the residual stream activations of LLMs:

1. Knowledge Probe: A $k$-class linear classifier trained to predict the ground-truth answer ($y$) from hidden states.
2. Prediction Probe: A $k$-class linear classifier trained to predict the model's own generated choice ($\tilde{y}$).

They evaluate these probes across diverse models (e.g., Llama 3.1 8B, Qwen 2.5 7B) and benchmarks (Reasoning, Knowledge, Truthfulness, Bias). To quantify the gap, they introduce two metrics:

• Prediction Agreement (AGR): The frequency with which the knowledge probe and the model's output select the same option.
• Distributional Divergence (KLD): The Kullback-Leibler divergence between the probability distributions of the knowledge probe and the model's output.

B. Geometric Interpretation

The authors interpret the weight vectors of the trained probes as defining low-dimensional subspaces within the residual stream:

• Knowledge Subspace: Spanned by the weights of the knowledge probe.
• Prediction Subspace: Spanned by the weights of the prediction probe.

They hypothesize that a large knowledge-prediction gap corresponds to a systematic misalignment between these two subspaces. When the gap is large, the coordinates of a hidden state projected onto the prediction subspace diverge significantly from its coordinates in the knowledge subspace.

C. KAPPA: Inference-Time Intervention

To mitigate this gap, the authors introduce KAPPA (Knowledge-Aligned Prediction through Projection-based Adjustment). KAPPA is a lightweight, inference-time intervention that applies an affine transformation to the residual stream.

• Mechanism: It solves a constrained optimization problem to minimally perturb the hidden state $h$ such that its coordinates in the prediction subspace align with its coordinates in the knowledge subspace.
• Formulation: The transformation $T$ maps $h$ to $h'$ by solving:
  $$\min_{h'} \|h' - h\|^2_2 \quad \text{s.t.} \quad W_{pred}^\top h' = W_{know}^\top h$$
  This yields a closed-form solution that adjusts the hidden state only within the prediction subspace, leaving orthogonal components unchanged.
• Generalization: The method includes hyperparameters $\alpha$ and $\beta$ to control the strength of the alignment, allowing for flexible adjustment of how strongly internal knowledge influences the prediction.

3. Key Contributions

1. Empirical Evidence of a Widespread Gap: The study establishes that the knowledge-prediction gap is pervasive across diverse MCQ benchmarks and model families. The gap is most pronounced in reasoning, truthfulness, and bias tasks, while being less evident in knowledge-intensive benchmarks.
2. Geometric Explanation: The paper provides a geometric interpretation of the gap, identifying it as a misalignment between distinct knowledge and prediction subspaces in the residual stream.
3. KAPPA Intervention: The authors present KAPPA, an inference-time method that geometrically calibrates hidden states to align model predictions with internal knowledge. Unlike prior steering methods that apply fixed corrections, KAPPA adapts its adjustment based on the specific knowledge-prediction gap of each instance.

4. Results

• Gap Magnitude: Linear probes consistently outperform base model generation accuracy across all tested datasets, confirming that models encode linearly accessible knowledge they fail to utilize. The gap varies significantly by domain, with TruthfulQA and BBQ showing the largest discrepancies (e.g., $\Delta$ACC up to +21.3% for Qwen 2.5 7B).
• Performance Improvement: KAPPA consistently reduces the gap and improves accuracy over baseline models and existing inference-time methods (such as CAA and DoLA).
  • On TruthfulQA, KAPPA improved Llama 3.1 8B accuracy by 8.9–16.9 points.
  • On Reasoning benchmarks (GSM8k, BBH), KAPPA achieved consistent gains of 1.5–5.0 points.
  • KAPPA significantly increased the Agreement Rate (AGR) between the knowledge probe and model output, often approaching the accuracy of the knowledge probe itself.
• Generalization:
  • Across Models: KAPPA effectively mitigates the gap across models of varying sizes (4B to 14B) and architectures (Mistral, Llama, Qwen).
  • Cross-Dataset Transfer: While subspaces are generally dataset-specific, successful transfer was observed between tasks requiring similar underlying skills (e.g., from BBH-NLP to GSM8K).
  • Free-Form Generation: Applying KAPPA to free-form settings (without MCQ options) improved accuracy on TruthfulQA, demonstrating that the geometric alignment extends beyond multiple-choice formats.

5. Significance and Claims

The paper claims that many LLM failures, particularly hallucinations and reasoning errors, stem not from a lack of knowledge but from a misalignment between what the model internally encodes and what it ultimately predicts. By identifying distinct knowledge and prediction subspaces, the authors provide an interpretable geometric explanation for these failures.

The introduction of KAPPA demonstrates that this gap can be bridged through representation-level alignment without additional training. The authors conclude that representation-level alignment is a promising direction for eliciting more faithful predictions from LLMs. They note that while their work focuses on linearly accessible knowledge, future work is needed to understand how to align more complex, non-linearly encoded knowledge.