C$^2$-Cite: Contextual-Aware Citation Generation for Attributed Large Language Models

1. Problem Statement

While Retrieval-Augmented Generation (RAG) has improved the factual accuracy of Large Language Models (LLMs), ensuring the credibility and traceability of generated content remains a challenge. Existing attributed LLMs (models that generate citations like [i]) suffer from a critical flaw: they treat citation markers as generic, passive placeholders rather than context-sensitive pointers.

• The Core Issue: Instruction-tuned models often fail to interpret the semantic relationship between a citation symbol (e.g., [1]) and the specific retrieved document it references.
• Consequences: This leads to:
  • Disjointed References: Citations that do not logically support the preceding text.
  • Hallucinated Citations: Generating non-existent symbols or linking to the wrong source.
  • Poor Integration: The model fails to weave retrieved knowledge smoothly into the narrative, often degrading response coherence when forced to insert citations.

2. Methodology: C2-Cite Framework

The authors propose C2-Cite, a novel fine-tuning framework designed to transform citation markers from passive tokens into active knowledge pointers. The framework is built upon a standard decoder-only LLM architecture and introduces three key mechanisms:

A. Contextual-Aware Embedding (CAE)

Instead of using standard token embeddings for citation markers, C2-Cite injects semantic information directly into the citation token.

• Mechanism: Multi-token citation markers (e.g., [i]) are converted into single special tokens (e.g., ⟨c_i⟩).
• Embedding Fusion: The embedding for ⟨c_i⟩ is replaced or augmented with the mean-pooled contextual embedding of the corresponding retrieved document $d[i]$.
• Effect: This allows the model to "read" the source document's semantics directly through the citation token during generation, grounding the symbol in specific content.

B. Contextual Citation Alignment

To ensure the model learns to associate the correct citation token with the correct document, a Citation Router and Alignment Loss are introduced.

• Citation Router: A binary classifier determines if a generated token is a default token or a citation token.
• Alignment Mechanism: For citation tokens, the model maps the hidden state to a "citation space" and calculates similarity against the contextual embeddings of all available documents.
• Loss Function: A cross-entropy loss ($L_{citation}$) forces the model to classify the citation token into the correct document index based on semantic similarity, rather than just pattern matching.

C. Contextual Attentive Augment

To prevent citations from disrupting the flow of the generated text, the framework enhances attention mechanisms.

• Mechanism: A loss function ($L_{attn}$) constrains the attention scores between a new token and a preceding citation token.
• Goal: It encourages the model to maintain strong attention from the text following a citation back to the content preceding it, ensuring inter-sentential coherence even when citations are inserted.

D. Overall Optimization

The total loss function is a weighted sum of:

1. Standard Language Modeling Loss ($L_{default}$)
2. Citation Alignment Loss ($L_{citation}$)
3. Router Loss ($L_{router}$)
4. Attention Augmentation Loss ($L_{attn}$)

3. Key Contributions

1. Novel Framework: C2-Cite is the first framework to explicitly model the semantic relationships between citation markers and their referenced content during the fine-tuning process.
2. Active Knowledge Pointers: By encoding document context into citation symbols, the framework transforms citations from static placeholders into dynamic, semantically rich pointers.
3. Robust Performance: The method achieves state-of-the-art results across diverse datasets without relying on complex, multi-stage data synthesis pipelines used by competitors.
4. Open Source: The implementation is publicly available, promoting reproducibility.

4. Experimental Results

The framework was evaluated on the ALCE benchmark using three datasets: ASQA (multi-document QA), ELI5 (long-form QA), and QAMPARI (multi-answer QA). The base model used was Llama3-8B.

• Comparison: C2-Cite++ (the full framework) was compared against Prompt-based (P-ICL), Iterative (I-ICL), and Fine-tuning (FT) baselines, including the SOTA baseline Front.
• Key Metrics:
  • Citation Quality (F1): C2-Cite++ outperformed the SOTA baseline (Front) by an average of 5.8%.
  • Response Correctness: C2-Cite++ outperformed Front by an average of 17.4%.
• Efficiency: The framework achieved the highest Throughput (samples/second) among top-performing models, demonstrating superior operational efficiency compared to iterative retrieval methods.
• Ablation Studies:
  • Removing Contextual-Aware Embedding caused a 25.6% drop in Citation-F1 and a 21.7% drop in Correctness, proving its critical role.
  • Removing Attentive Augment caused a 17% drop in Citation-F1, highlighting its importance for maintaining text coherence.

5. Significance and Impact

• Solving the "Placeholder" Problem: The paper addresses a fundamental limitation in current attributed LLMs where citation symbols are treated as syntactic noise rather than semantic carriers.
• Balancing Quality and Attribution: Unlike previous methods that often sacrificed response fluency for citation accuracy (or vice versa), C2-Cite successfully balances both, ensuring that the addition of citations does not degrade the quality of the generated answer.
• Scalability: By integrating citation learning directly into the fine-tuning process with a lightweight router and embedding mechanism, the approach is more efficient than complex iterative retrieval or heavy data synthesis pipelines.
• Trustworthiness: By enabling models to generate citations that are semantically grounded in the source text, C2-Cite significantly enhances the reliability and verifiability of LLM outputs, a crucial step for real-world deployment in domains requiring high factual accuracy.