DART: Mitigating Harm Drift in Difference-Aware LLMs via Distill-Audit-Repair Training

1. Problem Statement: The "Harm Drift" Phenomenon

Current Large Language Models (LLMs) tuned for safety often exhibit identity-blindness, defaulting to treating all demographic groups identically even when factual, legal, or ethical contexts require recognizing differences (e.g., acknowledging higher disease incidence in specific ancestry groups or religious hiring preferences). This leads to two main failure modes:

1. Over-refusal/Incorrectness: Models incorrectly refuse to answer or provide generic "equal treatment" responses when differential treatment is warranted.
2. Harm Drift: The paper identifies a novel phenomenon where fine-tuning models to improve decision accuracy (correctly identifying when differences matter) inadvertently increases the harmfulness of the generated explanations (rationales).
   • Mechanism: As models learn to correctly classify "YES" (differential treatment warranted), they often elaborate on sensitive premises, introduce harmful stereotypes, or normalize biases in their reasoning to justify the correct answer.
   • Consequence: The model becomes more accurate but less safe, producing "Correct but Harmful" outputs that reinforce harmful beliefs or mislead users.

2. Methodology: The DART Framework

The authors propose DART (Distill–Audit–Repair Training), a multi-stage pipeline designed to decouple accuracy optimization from harm mitigation. The process involves three distinct stages:

Stage I: Teacher Distillation (Accuracy Optimization)

• Goal: Improve the student model's decision accuracy without immediate safety constraints.
• Process:
  • A powerful Teacher model (e.g., DeepSeek-Chat) generates rationales for ground-truth labeled examples.
  • The Teacher is conditioned on the gold label (YES/NO) to ensure the rationale explains the correct classification.
  • A Student model (e.g., Llama-3-8B) is fine-tuned using Low-Rank Adaptation (LoRA) on these teacher-generated rationales.
• Outcome: The student model achieves high decision accuracy but exhibits harm drift, where its rationales become more toxic or stereotypical than the baseline.

Stage II: Harm Audit (Detection)

• Goal: Identify specific cases where the distilled model's rationale is more harmful than the baseline's, despite being more accurate.
• Process:
  • Paired Generation: The baseline ($M_0$) and distilled model ($M_{int}$) generate outputs for the same test prompts.
  • Two-Step Detection:
    1. Toxicity Screening: A classifier flags cases where the toxicity delta ($H(r_{int}) - H(r_0)$) exceeds a threshold.
    2. LLM-as-Judge Confirmation: An LLM judge reviews flagged cases to confirm if $M_{int}$ introduced new harms, normalized biases, or missed harms that $M_0$ identified.
  • Severity Stratification: Confirmed drift cases are categorized by severity (Mild, Moderate, Severe, Extreme) based on the nature of the harm (e.g., normalizing bias vs. explicit slurs).

Stage III: Targeted Repair (Harm Mitigation)

• Goal: Correct the identified harmful rationales while preserving the accuracy gains from Stage I.
• Process:
  • Safe Target Generation: The Teacher model is queried again for the flagged drift cases, but with strengthened safety instructions (e.g., "Do not repeat harmful content," "Use abstract language").
  • Severity-Weighted Oversampling: The repair dataset is constructed by oversampling severe cases (e.g., 4x for Extreme, 3x for Severe) to ensure the model prioritizes fixing the most harmful errors.
  • Continued Fine-tuning: The student model is further fine-tuned on this combined dataset (original distillation data + severity-weighted repair data).
• Inference-Time Policy: An optional layer of structured prompting is applied at deployment to constrain rationale length and content based on the predicted class (YES/NO), providing an orthogonal safety control.

3. Key Contributions

1. Identification of Harm Drift: The paper formally characterizes "harm drift," a failure mode where accuracy improvements via fine-tuning degrade rationale safety, distinct from general safety degradation or over-refusal.
2. The DART Pipeline: A novel staged approach (Distill-Audit-Repair) that resolves the accuracy-safety trade-off. Unlike joint optimization (which forces suboptimal compromises), DART maximizes accuracy first and then performs targeted, severity-weighted repairs.
3. Comprehensive Evaluation: The framework is validated across eight benchmarks, multiple model families (Llama, Qwen, Mistral, Gemma), and open-ended real-world queries.

4. Key Results

The experiments were conducted primarily on Llama-3-8B-Instruct:

• Accuracy Gains:
  • Overall accuracy on difference-awareness classification improved from 39.0% to 68.8%.
  • Crucially, accuracy on EQUAL cases (where groups should be treated identically) jumped from 11.3% to 72.6%, correcting the model's bias toward over-predicting differential treatment.
  • Accuracy on DIFF cases (where differences are warranted) remained stable, showing no under-prediction.
• Harm Reduction:
  • The number of harm drift cases was reduced by 72.6% (from 435 to 119 in the test set).
  • Toxicity scores decreased significantly, with the final model ($M_{DART}$) achieving lower toxicity than the baseline ($M_0$).
• Generalization:
  • Open-Ended Queries: On 280 real-world queries (medical, legal, policy), difference-appropriate responses increased from 39.8% to 77.5%, while refusal rates dropped from 34.3% to 3.0%.
  • Cross-Model: Consistent gains were observed across Llama-3-3B, Qwen2.5-14B, Mistral-7B, and Gemma-2-9B.
• Ablation Studies:
  • Joint Optimization Failure: Training with a toxicity penalty during the initial stage failed to achieve both high accuracy and low toxicity, confirming the necessity of the staged approach.
  • Inference-Time Policy: While helpful, the policy alone could not fix the underlying reasoning bias; it served as a complementary safety layer.

5. Significance and Implications

• Reconciling Safety and Accuracy: The paper demonstrates that safety and accuracy in demographic reasoning are not inherently conflicting. With explicit detection and repair mechanisms, models can learn to distinguish when group differences are relevant without generating harmful content.
• Beyond Over-Refusal: It addresses a subtle but critical failure mode where models answer correctly but justify their answers with harmful reasoning, a problem often missed by standard refusal-based safety metrics.
• Practical Framework: DART provides a practical, scalable pipeline for developers to fine-tune models on sensitive topics without compromising safety, using severity-weighted repair to focus computational resources on the most critical errors.
• Evaluator Limitations: The authors note that standard hate-speech classifiers can be brittle in difference-awareness contexts (flagging correct counter-speech as hate), highlighting the need for more nuanced evaluation metrics for demographic reasoning tasks.

In summary, DART offers a robust solution to the "harm drift" problem, enabling LLMs to be both factually precise regarding demographic differences and ethically safe in their explanations.