Technical Summary: Faithful-Patchscopes

1. Problem Statement

Large Language Models (LLMs) have demonstrated the ability to interpret their own internal hidden representations through the Patchscopes framework (Ghandeharioun et al., 2024). This framework decodes hidden states into human-readable explanations by "patching" a source representation into a target prompt. However, this paper identifies a critical limitation: systematic unfaithfulness in these explanations.

The authors observe that LLMs often rely on inherent linguistic patterns (empirical priors) rather than the contextual information encoded in the hidden representations during the decoding process. For instance, if a hidden representation encodes the attribute "purple" for "broccoli" (contrary to the common association of broccoli with "green"), the model frequently ignores the encoded "purple" and generates "green" in its explanation. This behavior is driven by model bias induced by imbalanced linguistic patterns in training data, where high-frequency associations override low-frequency contextual attributes.

The paper argues that existing debiasing methods are ill-suited for Patchscopes:

• Supervised Fine-Tuning (SFT) requires expensive annotated datasets and globally alters model parameters, thereby changing the original hidden representations that Patchscopes aims to explain.
• Prompt-based methods are unstable and highly sensitive to prompt design.
• Steering vector interventions assume bias exists in well-defined, linearly separable directions, which is difficult to estimate reliably.

2. Methodology

2.1. Dataset and Analysis

To systematically study this issue, the authors constructed a dataset covering four tasks: Color, Gender, Culture, and Age.

• Biased Subset: Contains noun-attribute pairs where the model exhibits a strong preference for a primary attribute ($a_{pri}$) based on linguistic frequency (e.g., broccoli $\rightarrow$ green), while a secondary attribute ($a_{sec}$) is contextually provided (e.g., broccoli $\rightarrow$ purple).
• Non-biased Subset: Contains pairs where no strong linguistic preference exists.

Findings:

• Causal Link: Logistic regression analysis confirms that imbalanced linguistic patterns (frequency gaps between attributes) are a meaningful determinant of model bias. A one standard deviation increase in frequency difference increases the odds of bias by a factor of ~9.3.
• Faithfulness Drop: On the biased subset, the faithfulness of vanilla Patchscopes (measured by Show Rate, the ratio of correct attribute generation) decreases by an average of 18.84% compared to the non-biased subset.

2.2. Locating Bias-Sensitive Layers

The authors propose a method to identify the specific layers where bias is encoded and begins to influence downstream computation. They introduce two metrics:

1. Logit Difference (LD): Measures the difference in logits between the context-aligned attribute and the biased attribute at each layer using a logit-lens approach.
2. Gradient Similarity Alignment (GSA): Measures the alignment between the output gradient and the attribute direction to determine if the model relies on biased or contextual information.

These metrics are combined into a layer-selection score ($l^*$) to identify the optimal layer for intervention. The authors found that bias is not uniformly distributed but emerges in intermediate layers.

2.3. Proposed Solution: BALOR

The paper introduces Bias Alignment through Logit Recalibration (BALOR), a decoding-time intervention that mitigates bias without modifying model parameters.

Core Mechanism:
BALOR utilizes a contrastive prompt pair:

• Target Prompt ($T$): Contains the patched hidden representation (encoding the contextual attribute).
• Contrastive Prompt ($T^*$): Replaces the placeholder with the original token, containing no contextual information, thus reflecting only the model's empirical bias/priors.

Logit Recalibration:
The method subtracts the scaled logits of the contrastive prompt from the target prompt's logits before applying the softmax function:
$$p_{balor}(y | T) = \sigma \left[ (1 + \alpha) l_\theta(y | T) - \alpha l_\theta(y | T^*) \right]$$
Where:

• $l_\theta(y | T)$ and $l_\theta(y | T^*)$ are the pre-softmax logits.
• $\alpha$ is a hyperparameter controlling the amplification level.

Theoretical Guarantee:
Theorem 4.1 proves that in log-odds space, if a token is more supported by the patched evidence than by the contrastive prior, BALOR monotonically increases its relative preference as $\alpha$ increases. This effectively isolates the bias component and amplifies the contextual information.

Modes:

• Shared (Mode S): The recalibrated distribution is used to sample tokens for both prompts, keeping sequences synchronized.
• Divided (Mode D): Recalibration is applied only to the target prompt; the contrastive prompt decodes with original logits.

3. Key Contributions

1. Dataset and Analysis: Designed a dataset to evaluate Patchscopes under biased conditions, demonstrating that model bias fundamentally limits explanation faithfulness (causing an 11.84% to 28.47% decrease in performance).
2. BALOR: Introduced a logit-recalibration decoding method that mitigates bias at the inference stage by contrasting patched representations against empirical priors.
3. Empirical Validation: Conducted extensive experiments across four modern LLMs (Llama2-7b, Llama3.2-1b, Llama3-8b, Qwen3-4b), showing BALOR consistently outperforms baselines.

4. Experimental Results

Experiments were conducted under zero-shot and few-shot settings across the four tasks.

• Performance: BALOR (specifically Mode S) consistently outperformed all baselines, including Vanilla Patchscopes, prompt-guidance methods (Closed-Book, Internal-Evidence, Data-Bound), SFT, and FairSteer.
• Improvement: BALOR achieved up to a 33% relative performance improvement in Show Rate (SR) compared to the vanilla baseline.
• Stability: Mode S demonstrated high stability across different hyperparameters ($\alpha$) and sampling temperatures. Mode D showed more variability and occasional instability due to sequence divergence between prompts.
• Comparison:
  • Prompt-based baselines were unstable and often underperformed the vanilla baseline.
  • SFT methods degraded significantly when the fine-tuning data conflicted with evaluation data.
  • FairSteer performed poorly when the debiasing direction was inaccurate.

5. Significance and Claims

The paper claims that imbalanced linguistic patterns induce model bias that distorts the faithfulness of hidden representation explanations. By proposing BALOR, the authors provide a mechanism to recover the true information encoded in hidden representations without altering the model's parameters or internal representations.

The authors emphasize that their work improves the reliability of LLM interpretability. They state that while their method enhances faithfulness, the resulting explanations should not be treated as definitive evidence of model reasoning but should be used alongside complementary evaluation techniques. The work aims to help researchers better understand how LLMs encode and use contextual information, which is crucial for the trustworthy deployment of these models. The authors do not foresee negative societal consequences directly arising from this work.