Technical Summary: Unlearners Can Lie: Evaluating and Improving Honesty in LLM Unlearning

1. Problem Statement

Large Language Model (LLM) unlearning aims to selectively remove specific training data (e.g., sensitive or harmful information) while preserving the model's general utility. However, existing unlearning methods often fail to ensure honesty in the unlearned model. While a model may successfully "forget" target knowledge in terms of accuracy, it may exhibit dishonest behaviors such as:

• Hallucination: Fabricating plausible but incorrect information about forgotten topics.
• Inconsistency: Providing different answers to semantically equivalent queries or failing to maintain a refusal stance across multi-turn dialogues.
• Spurious Refusal: Outputting "I don't know" (IDK) responses while retaining the underlying knowledge (masked knowledge) or selecting IDK options based on positional bias rather than genuine uncertainty.

The paper argues that current evaluation metrics (e.g., accuracy on forget sets) are insufficient because they do not capture self-knowledge (the model's ability to recognize its limitations) and self-expression (the consistent and truthful communication of those limitations).

2. Methodology

2.1 Formal Definition of Honest Unlearning

The authors define honest unlearning through two pillars:

1. Retain Set: The model must preserve utility and the ability to faithfully express retained knowledge.
2. Forget Set: The model must effectively remove targeted knowledge and, crucially, acknowledge its limitations truthfully and stably when queried about forgotten content.

2.2 Evaluation Framework

To systematically evaluate honesty, the paper introduces a suite of metrics across Question Answering (Q&A) and Multiple-Choice Questions (MCQ) settings:

• Utility Metrics: MMLU, Instruction Following (IF), and Number of Correct answers (NC) on retained data.
• Honesty on Retain Set: Agreement Rate (AR) and Misleading Robustness Score (MRS).
• Forget Set Metrics:
  • Rejection Rate (RR): Proportion of questions where the model explicitly refuses or expresses uncertainty.
  • QAMRC (Q&A Multi-turn Rejection Consistency): Measures if a refusal in the first turn remains stable after a follow-up query.
  • CIR (Choose-IDK Rate) & COR (Choose-Other Rate): In MCQs, CIR measures selection of an "I don't know" option. COR measures selection of a semantically irrelevant option in the same position to detect positional bias.
  • STD & MCQSC: Measure stability of responses under prompt variations and repeated questioning.

2.3 Proposed Method: ReVa (Refusal-Vector Alignment)

The authors propose ReVa, a representation-alignment procedure designed to improve honesty in models that have already undergone feature-randomization unlearning (e.g., RMU).

• Mechanism: Instead of supervised fine-tuning (SFT) to output specific tokens like "I don't know," ReVa aligns the model's internal residual stream representations with a distilled refusal vector.
• Process:
  1. Extract a refusal direction $r$ from a model's internal activations when it expresses epistemic uncertainty (e.g., on out-of-knowledge questions).
  2. Fine-tune the unlearned model to align the activations of forget-set inputs toward this distilled refusal direction $r$ at specific transformer layers (optimized at layer 18 for Zephyr-7b).
  3. The loss function minimizes the distance between the model's activation and the scaled refusal vector: $L_{ReVa} = \mathbb{E}_{x \sim D_F} [\frac{1}{L(x)} \sum ||M^{(l)}_\theta - c \cdot r||^2_2]$.
• Advantages: ReVa avoids the superficial lexical mapping of IDK-SFT, is computationally efficient compared to gradient-ascent methods, and preserves utility on retained knowledge.

3. Key Results

3.1 Evaluation of Existing Methods

The paper evaluates 9 methods across three families (Rejection-based, Feature-randomize, Gradient-ascent) on the WMDP-Bio benchmark:

• Rejection-based (e.g., IDK+AP): Achieve high rejection rates but often suffer from "shallow and deceptive" behavior. They may retain high accuracy on forget questions despite outputting "IDK," indicating masked knowledge.
• Gradient-ascent (e.g., NPO, GA): Severely degrade utility (NC and MMLU drop significantly). They exhibit spurious IDK behavior: high CIR but also high COR, and extremely low first-token entropy. This suggests they select "IDK" options due to positional bias or token suppression rather than genuine uncertainty.
• Feature-randomize (e.g., RMU): Best at preserving utility and removing knowledge (low ACC), but fail at honesty. They often hallucinate on the forget set or provide inconsistent answers across turns, lacking stable self-expression.

3.2 Performance of ReVa

ReVa, applied as a post-processing step to RMU-unlearned models, demonstrates superior performance:

• Honesty: Achieves a Rejection Rate (RR) of ~60.86% and a QAMRC (stability after 2 rounds) of 45.42%, significantly outperforming baselines.
• Utility: Maintains high utility on the retain set (MMLU ~57.56, NC ~1592), comparable to the original RMU model and better than RMU+IDK.
• Stability: Reduces output variance (STD) and improves consistency in MCQs.
• Efficiency: Requires only ~5.91 minutes of training on 2x A6000 GPUs, compared to ~210 minutes for IDK+AP.

4. Key Contributions

1. Identification of Dishonesty: The paper identifies and categorizes dishonest behaviors (hallucination, inconsistency, spurious refusal) in current unlearning methods, arguing that unlearning is not just about forgetting but about honest self-expression.
2. Honesty Benchmark: A comprehensive evaluation framework with dedicated metrics (RR, QAMRC, CIR/COR, MRS) to assess both forgetting effectiveness and the stability of uncertainty expression.
3. Theoretical and Empirical Analysis: Analysis reveals that gradient-ascent methods lead to unbounded optimization, causing utility collapse and spurious token preferences, while rejection-based methods often learn superficial patterns.
4. ReVa Method: Introduction of a lightweight, representation-alignment method that enables models to genuinely acknowledge limitations without sacrificing general capabilities.

5. Significance and Claims

The paper claims that honesty is a critical, yet overlooked, dimension of LLM unlearning. Current methods that fail to acknowledge limitations can create safety risks (e.g., hallucinating dangerous medical or biological information) and erode user trust.

The authors assert that ReVa represents a significant step toward "honest unlearning" by enabling models to:

• Effectively forget targeted knowledge.
• Consistently and truthfully acknowledge what they no longer know.
• Maintain high utility on retained knowledge.

The paper concludes that while ReVa improves honesty and stability, it is not a perfect solution (e.g., MCQ performance still has room for improvement) and that the study is currently limited to the WMDP benchmark. The work emphasizes that future unlearning research must move beyond simple accuracy metrics to ensure models are not only "unlearned" but also "honest."