Technical Summary: Aligning Large Language Model Behavior with Human Citation Preferences

Problem Statement

While Large Language Models (LLMs) increasingly incorporate citations to enhance the verifiability of their outputs, the mechanisms governing what content warrants a citation remain underexplored. Current research focuses heavily on which documents to link (e.g., via RAG), but insufficient attention has been paid to the decision of when to cite based on text content. An imbalance in citation frequency—either too few or too many—can erode user trust, reduce satisfaction, and impair decision accuracy. This study addresses the critical gap in understanding whether LLMs inherently align with human preferences regarding citation necessity and whether these behaviors can be controlled.

Methodology

Dataset Construction

The authors constructed a specialized dataset to characterize the relationship between human citation preferences and LLM behavior:

• Source: 6,000 sentences extracted from the WikiSQE dataset, derived from Wikipedia.
• Categorization: Sentences were labeled with quality tags and regrouped into eight citation-motivation categories:
  1. Missing Information (Who, When, Which, Where)
  2. Sic (Textual errors)
  3. Doubt (Dubious claims)
  4. Vague (Ambiguous wording)
  5. POV (Non-neutral/biased statements)
  6. Medical Content (Unsourced health claims)
  7. Jargon (Overly technical terms)
  8. Unclear (Hard to understand)
• Annotation Strategy: The study utilized pairwise comparisons. For every combination of the eight categories (28 pairs), annotators were presented with two sentences and asked to select which one required a citation. This resulted in 2,596 high-quality comparison pairs after quality control.
• Participants: 402 human annotators provided ground-truth preferences.

Model Evaluation

The study evaluated 11 LLMs (including open-source models like Llama, Mistral, and DeepSeek, and closed models like GPT-5, Claude, and Gemini) on the constructed dataset. The evaluation measured the agreement rate between the model's citation choice and the human preference for each sentence pair.

Alignment via Training

To investigate if citation behavior could be corrected, the authors applied Direct Preference Optimization (DPO) using the human-annotated data. They fine-tuned five target models (Llama 1B, 3B, 70B, Mistral Small, and Mistral Large) using LoRA to align their citation decisions with human preferences.

Key Results

1. Limited Alignment in Current Models

Current LLMs exhibit only weak alignment with human citation preferences, with average agreement rates plateauing in the low 60s (e.g., DeepSeek at 62.7%, Llama 70B at 61.6%). A clear parameter-scale effect was observed, where larger models generally performed better, though even the largest models failed to reach high levels of agreement.

2. Systematic Divergences

The analysis revealed specific, systematic biases in model behavior compared to humans:

• Over-selection of "Citation Needed" Labels: Models are significantly more likely to cite sentences explicitly marked with "Citation needed" on Wikipedia than humans are. Closed models showed up to 27.4% higher selection rates for these sentences compared to humans, suggesting a strong imprint from training data rather than genuine understanding of cite-worthiness.
• Under-selection of Numeric Sentences: Models systematically underselect sentences containing numbers (quantitative expressions), with rates up to 22.6% lower than human preferences. Humans typically demand citations for statistics, but models often fail to recognize this need.
• Under-selection of Personal Names: Similarly, models underselect sentences containing personal names by up to 20.1% relative to humans, despite the high factual sensitivity required for biographical content.
• Medical Content Consensus: Both humans and models consistently prioritize citations for medical content, showing the highest agreement rates across all categories.

3. Effectiveness of DPO

Training models via DPO on the human-annotated data successfully improved alignment:

• Performance Gains: The approach yielded an average improvement of 5.76% in agreement with human preferences.
• Model-Specific Impact: Smaller models benefited most, with Llama 1B showing an 11.8% improvement. However, larger models like Llama 70B saw a slight decline (-1.6%), suggesting that alignment strategies may need to be tailored to model scale.
• Mitigation of Bias: DPO demonstrated the ability to attenuate the influence of training data biases (such as the over-reliance on "Citation needed" templates), bringing model behavior closer to human judgment.

Key Contributions

1. First Study on Citation Necessity: This work presents the first investigation focusing specifically on the need for citations within text, examining the preferences of both humans and LLMs.
2. Specialized Dataset: The construction of a 6,000-sentence dataset with human citation-preference labels across eight distinct content-based categories.
3. Analysis of Training Data Influence: The study demonstrates that LLM citation behavior is strongly influenced by training data patterns (e.g., Wikipedia templates), leading to divergences from human preferences.
4. Demonstration of DPO Efficacy: The paper shows that Direct Preference Optimization can effectively calibrate model behavior to better match human citation preferences, offering a mechanism to control citation frequency and type.

Significance and Implications

The paper argues that citation behavior should not be assumed to emerge naturally from pretraining but must be explicitly trained and evaluated. The findings suggest that:

• Explicit Training is Necessary: Relying on pretraining alone is insufficient for achieving cite-worthiness alignment.
• Controlled Deployment: Future AI systems should consider category-aware routing (e.g., boosting triggers for numeric and person-name sentences) and frequency control to balance verifiability with user experience.
• Agent Pipeline Optimization: Since many modern systems allow LLMs to implicitly govern search and citation decisions, improving cite-worthiness alignment at the model level can directly enhance the reliability of AI-agent pipelines.

The authors conclude that while current models show promise, significant work remains to align them with human expectations, particularly in handling specific content types like numbers and names, and that preference-based training offers a viable path forward.