Technical Summary: LLM Ghostbusters: Surgical Suppression of Hallucinations via Adaptive Forgetting

1. Problem Statement

Large Language Models (LLMs) employed in code generation frequently produce hallucinations: outputs that are linguistically fluent but factually incorrect. In the context of software development, this manifests as package hallucination, where a model invents non-existent names of software libraries (e.g., a fake Python package).

This failure mode creates a critical supply chain vulnerability known as Slopsquatting. An attacker can proactively register these hallucinated package names on public registries (such as PyPI) with malicious payloads. If developers or autonomous agents subsequently install these packages based on the LLM's recommendation, the malicious code is executed.

Existing mitigation strategies are insufficient for post-deployment fine-tuning:

• Full retraining is computationally prohibitive.
• Retrieval-Augmented Generation (RAG) and inference-time verification increase latency and complexity without altering the model's underlying learned representations.
• Standard Machine Unlearning (e.g., Gradient Ascent, NPO) typically requires a pre-specified, finite "forgetting set." However, the space of hallucination-triggering prompts is open, and the specific hallucinations a model produces are only known once they are provoked.
• Partial Model Collapse (PMC) offers a mechanism for targeted content removal but assumes a fixed, known dataset, which does not apply to the dynamic, unbounded nature of hallucinations.

2. Methodology: Adaptive Unlearning (AU)

The authors propose Adaptive Unlearning (AU), a closed-loop system for post-deployment fine-tuning designed to surgically suppress hallucinations while preserving general model utility. AU operates exclusively on synthetic data generated by the model itself and requires no human annotation.

Core Components

1. Adaptive Prompt Mutation Loop:

   • Instead of relying on a static forgetting dataset, AU continuously discovers new hallucination-triggering contexts.
   • The system samples from the current model. If a prompt does not trigger a hallucination or has been trained for a fixed number of epochs, it is considered "exhausted."
   • The prompt is then mutated (semantically varied) to provoke new hallucination patterns, ensuring the model learns to suppress patterns rather than memorizing specific prompt-output pairs.

2. Tri-Mask Token Routing:

   • A novel ternary gradient routing mechanism divides the token stream for each training sample into three disjoint populations:
     • Mask = 0 (Regularize): Contextual tokens (prompts, code structure, punctuation). These receive a scaled Cross-Entropy (CE) loss to anchor the model's distribution and prevent incoherent outputs.
     • Mask = 1 (Strengthen): Valid tokens (e.g., correct package names confirmed by a registry oracle). These receive a standard CE loss to reinforce correct generations.
     • Mask = 2 (Suppress): Hallucinated tokens (e.g., non-existent package names). These receive a Negative Preference Optimization (NPO) loss to push probability mass away from these tokens.
   • This tri-masking ensures that reinforcement and suppression gradients act on disjoint token sets, thereby preventing the catastrophic collapse that frequently occurs when opposing objectives are applied to the same tokens.

3. Hybrid Objective Function:

   • The total loss combines the three terms:
     $$L_{AU} = \lambda_{retain} L_{CE}^{T_{retain}} + \lambda_{forget} L_{NPO}^{T_{forget}} + \lambda_{reg} L_{CE}^{T_{reg}}$$
   • This enables the model to simultaneously learn within the same sequence what it should not generate (hallucinations) and what it should generate (valid alternatives).

4. Nested Training Loops:

   • To stabilize the unlearning process, AU employs a two-timescale training structure:
     • Outer Epochs: Generate new samples and detect hallucinations to update the tri-masks.
     • Inner Epochs: Train on the cached samples from the outer epoch over multiple steps before resampling. This allows the model to consolidate the suppression signal rather than constantly chasing "new" samples.

3. Key Contributions

1. Adaptive Unlearning Framework: The first unlearning method that couples an adaptive hallucination discovery loop with a hybrid token-level objective, enabling generalization to unseen prompts without a pre-specified forgetting set.
2. Tri-Mask Token Routing: A mechanism enabling surgical interventions within a single sequence by dividing tokens into valid, hallucinated, and context categories, allowing CE and NPO objectives to operate without conflict.
3. Adaptive Prompt Mutation: A strategy that extends unlearning from finite sets to open prompt spaces, ensuring the model suppresses hallucination patterns rather than overfitting to specific prompts.
4. Nested Training for Stability: A cache update scheme addressing the instability of earlier collapse-based unlearning methods, which is crucial for stable hallucination reduction.
5. Systematic Joint Evaluation: An empirical comparison evaluating hallucination rate, distributional drift, and coding utility simultaneously, demonstrating that effective mitigation requires balancing all three axes.

4. Experimental Results

The authors evaluated AU on DeepSeek-Coder models (7B and 16B parameters) using a dataset of code generation prompts designed to elicit package hallucinations.

• Reduction in Hallucinations:

  • For the DeepSeek-7B model, AU reduced the overall package hallucination rate from 21.2% (Baseline) to 2.56%, an 88% reduction.
  • For the DeepSeek-16B model, the rate dropped from 27.75% to 6.13%, a 78% reduction.
  • This significantly outperformed baselines: Gradient Ascent (GA), Negative Preference Optimization (NPO), and Partial Model Collapse (PMC).

• Preservation of Utility:

  • AU maintained or slightly improved performance on standard coding benchmarks (HumanEval, MBPP, and their EvalPlus variants).
  • For DeepSeek-7B, the average coding benchmark score improved to 68.57 (vs. 66.98 for the baseline model).
  • For DeepSeek-16B, the score remained stable at 73.17 (vs. 77.4 for the baseline, with a slight decline attributed to MoE training dynamics, yet still superior to other unlearning baselines).

• Distributional Drift (KL Divergence):

  • Analysis showed that the behavioral changes induced by AU were heavily concentrated in the package generation distribution (KL divergence ~1.4 Nats for package queries), while remaining minimal for general code generation (KL divergence ~0.08 Nats) and instruction following.
  • This confirms the "surgical" nature of the unlearning and avoids broad collateral damage to the model's capabilities.

• Ablation Studies:

  • Loss Composition: Neither CE-only (reinforcement) nor NPO-only (suppression) was sufficient; both resulted in hallucination rates >12%. The hybrid approach was essential.
  • Inner Epochs: Training without inner epochs (single-stage resampling) failed to achieve effective hallucination reduction. A minimum of approximately 10 inner epochs was required for stable consolidation.
  • Prompt Mutation: Adaptive mutation was critical for generalization. Without mutation, hallucination rates remained high (~10.7%); with 6 mutations, they dropped to ~3.9%.

5. Significance and Claims

The work claims that Adaptive Unlearning offers providers a principled, practical pathway to address hallucinations in deployed LLMs between release cycles.

• Security Impact: By reducing package hallucination rates by over 80%, AU directly shrinks the attack surface for Slopsquatting attacks, a critical supply chain vulnerability.
• Operational Feasibility: Unlike full retraining, AU operates on synthetic, self-generated data, requires no human annotation, and relies neither on external knowledge bases nor on inference-time verification overhead.
• Generalizability: Although the experiments focus on package hallucinations in Python, the authors argue that the framework is ecosystem-agnostic and applicable to any domain with a deterministic oracle (e.g., API endpoints, citations, historical claims).
• Limitations: The authors modestly note that the current oracle checks string-level validity (existence in the registry) rather than semantic correctness within the code. They also acknowledge that full-parameter fine-tuning at larger scales (beyond 16B) requires resources beyond their current study, and that generalization to general (non-code-related) LLMs remains an open extension.

In summary, the work presents AU as a method that successfully navigates the trade-off between suppressing specific undesirable behaviors and preserving general model utility, thereby filling a gap where previous unlearning methods failed due to the open nature of hallucinations.