💬 NLP

Citation Grounding: Detecting and Reducing LLM Citation Hallucinations via Legal Citation Graphs

This paper introduces "Citation Grounding," a novel metric and framework for detecting and reducing legal citation hallucinations in large language models by leveraging a massive Ukrainian court decision graph to evaluate citation accuracy and employing a preference learning method (CG-DPO) to train models to distinguish valid citations from corrupted ones.

Original authors: Volodymyr Ovcharov

Published 2026-06-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Volodymyr Ovcharov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a very smart, fast-talking legal assistant to write a report for you. This assistant has read millions of books and knows a lot of facts. However, there's a catch: when asked to cite specific laws to back up their arguments, this assistant sometimes makes them up. They might invent a law that doesn't exist, quote a law that was cancelled years ago, or quote a law from a different country entirely. In the legal world, this is called a "hallucination," and it can get people in serious trouble.

This paper introduces a new way to catch these lies and a method to teach the assistant not to tell them.

The Problem: The "Fake Law" Generator

The authors found that even the most advanced AI models, when asked to write about Ukrainian law, invent fake legal citations about 13% to 21% of the time. It's like a student writing an essay and making up page numbers for a textbook that doesn't exist.

The Solution: The "Citation Grounding" Test

To fix this, the researchers built a massive digital map called a Citation Graph. Think of this graph as a giant, super-accurate library catalog built from 100 million real court decisions. It contains every single time a real judge has ever cited a specific law in a real case.

They created a test called Citation Grounding (CG). Here is how it works:

The Map: They have the "truth map" (the graph of real court cases).
The Test: They ask an AI to write a legal answer.
The Check: They look at every law the AI mentions and check the map.
- Did the law exist? (Precision)
- Was it the right law for this specific situation? (Relevance)
- Was the law actually in effect on that specific date? (Temporality)

If the AI cites a law that isn't on the map, or cites it in the wrong context, the test flags it as a hallucination.

What They Found

They tested five different AI systems on 100 legal questions.

The Results: Most AIs got about 80% of their citations right. The rest were fake.
The "RAG" Winner: One system used a special trick called RAG (Retrieval-Augmented Generation). Imagine this AI doesn't just rely on its memory; it actually goes to the library, finds the real books, and reads them before answering. This system made the fewest mistakes and cited the fewest laws overall, proving that "less is more" when you are actually looking up the facts.
The Surprising Pattern: They expected that AI models that cited more laws would make more mistakes. But that wasn't true. Some models cited many laws and were still accurate; others cited fewer laws and were still wrong. It depends entirely on how the AI is built, not just how much it talks.

The "Self-Teaching" Method (CG-DPO)

The biggest breakthrough in the paper is how they plan to fix the AI without hiring a team of expensive human lawyers to check every answer.

Usually, to teach an AI to be honest, humans have to read its answers and say, "This is good," or "This is bad." This is slow and expensive.

The researchers realized their Citation Graph could do the human's job. They created a game:

Take a real court decision with correct citations.
Corrupt it: They used a computer program to swap laws, invent fake articles, or use outdated laws (like changing a 2020 law to a 2025 law that didn't exist yet).
The Lesson: They showed the AI two versions: the "Real" one and the "Corrupted" one. The AI learned to prefer the Real one.

Because the "truth" was already in the graph, they didn't need humans to label the data. They just let the graph do the grading. They trained a model using this method, and it became 98.5% accurate at spotting the difference between a real citation and a fake one.

The Bottom Line

This paper shows that:

AI lies about laws often, but we can catch it by comparing its answers to a massive map of real court history.
RAG systems (which look up facts before answering) are currently the best at avoiding these lies.
We can teach AI to stop lying by using the map itself to generate "correct vs. incorrect" examples, removing the need for human lawyers to do the tedious work of checking every single citation.

The authors have made their "map," their testing tools, and their training data available for anyone to use, hoping to make legal AI more reliable for everyone.

Technical Summary: Citation Grounding for Detecting and Reducing LLM Citation Hallucinations

1. Problem Statement

Large Language Models (LLMs) exhibit a systematic reliability issue in legal domains: citation hallucination. This involves generating plausible but fabricated references to statutes, citing repealed provisions, confusing jurisdictions, or constructing invalid legal reasoning chains. While LLMs perform well on legal reasoning benchmarks (e.g., bar exams), existing evaluation frameworks (LEXTREME, LexGLUE, LegalBench) focus on classification tasks and lack metrics to measure or detect citation accuracy in generative outputs. Manual verification by legal experts is the current standard but does not scale to continuous monitoring or low-resource languages.

2. Methodology

2.1 The Ground-Truth Citation Graph

The foundation of the proposed approach is a massive, bipartite citation graph $G = (V, E)$ extracted from the Unified State Register of Court Decisions (EDRSR) of Ukraine.

Scale: 100.8 million decision nodes ( $V_d$ ), 21,736 unique statute nodes ( $V_n$ ), and 502.2 million citation edges ( $E$ ).
Structure: Edges connect a court decision to the specific statute articles it cites.
Temporal Scope: The graph spans 2005–2026, covering three distinct legal epochs (peacetime, hybrid war, and full-scale invasion/martial law), enabling temporal validity checks.

2.2 Citation Grounding (CG) Metric

The authors propose Citation Grounding (CG), a metric that verifies LLM-generated citations against the ground-truth graph $G$ . It is decomposed into three components to enable differential diagnosis of hallucination types:

Citation Precision (CP): Verifies existence. Does the cited statute node exist in $V_n(G)$ ?
Citation Relevance (CR): Verifies contextual appropriateness. Is the cited article used in decisions semantically similar to the query? (Computed via XLM-RoBERTa embeddings).
Citation Temporality (CT): Verifies temporal validity. Was the statute in force at the date of the relevant legal event?

The composite Citation Quality ($CQ$) is calculated as the mean of the product of CP and the average of CR and CT. A non-existent statute (CP=0) results in zero quality, reflecting the legal standard that a single fabricated citation invalidates a document.

2.3 Citation Grounding DPO (CG-DPO)

To reduce hallucinations without human annotation, the paper introduces CG-DPO, a method for constructing Direct Preference Optimization (DPO) training pairs algorithmically.

Mechanism: Starting from verified court decisions in $G$ , the system creates "corrupted" versions of the text by modifying citations.
Corruption Strategies:
1. Article Swap: Replaces an article with a different one from the same codex (targets CP).
2. Law Swap: Replaces the codex while keeping the article number (targets CR).
3. Hallucination Injection: Replaces a real article with a non-existent one (targets CP).
4. Anachronism: Replaces a statute with one enacted after the decision date (targets CT).
Training: The original text with correct citations serves as the preferred response ( $y_w$ ), and the corrupted text serves as the rejected response ( $y_l$ ). This eliminates the need for human annotators.

3. Experimental Setup and Results

3.1 Empirical Evaluation

The authors evaluated five systems on 100 Ukrainian legal queries across seven domains (Civil, Criminal, Administrative, Labor, Family, Constitutional, Military):

Systems: Four commercial LLMs via AWS Bedrock (Claude Haiku 4.5, Mistral Pixtral Large, Amazon Nova Pro, Amazon Nova Lite) and one RAG-augmented production system (LEX Chat, based on Claude Sonnet 4).
Findings:
- Hallucination Rates: CG scores ranged from 0.791 to 0.873, indicating 13–21% of citations were hallucinated.
- RAG Superiority: The RAG-augmented system achieved the highest CG (0.873) with the lowest citation density (2.9 citations/response), suggesting retrieval constrains models to evidence-based citations.
- Density-Accuracy Correlation: Among raw models, the correlation between citation density and accuracy was weak ( $\rho = -0.12$ ), contradicting the hypothesis of a universal trade-off.
- Domain Variance: Constitutional law achieved perfect grounding (CG = 1.0) across all models. Family and Labor law showed the highest variance (CG 0.46–0.90).

3.2 CG-DPO Training Results

A Qwen2.5-7B-Instruct model was fine-tuned using LoRA on a dataset of 2,244 court decisions with algorithmically constructed preference pairs.

Performance: The model achieved 98.5% mean validation accuracy in distinguishing correct from corrupted citations across three random seeds.
Stability: The reward margin was +14.9 with a standard deviation of < 0.3 percentage points.
Convergence: Training converged rapidly, reaching near-perfect accuracy within the first epoch.

4. Key Contributions

Citation Grounding Metric: A novel, decomposable metric (CP, CR, CT) for measuring and diagnosing legal citation hallucinations at scale, utilizing a 502-million-edge citation graph.
Large-Scale Evaluation: An empirical study of five LLMs on Ukrainian law, revealing that RAG systems outperform raw models in citation accuracy and that hallucination rates vary significantly by legal domain.
CG-DPO: A method for generating DPO training pairs algorithmically from a citation graph, removing the dependency on expensive human legal annotation for citation alignment.

5. Significance and Claims

The paper claims that a large-scale citation graph serves a dual purpose: acting as an evaluation oracle to detect hallucinations and as a training signal to align models without human intervention.

Closed Loop: The approach closes the loop from detection to mitigation without manual annotation at any stage.
Scalability: CG-DPO demonstrates that graph-derived signals are sufficient for citation alignment, offering a scalable solution for legal AI in resource-constrained environments.
Metric Validity: The authors validate CG as a conservative metric; while it may produce false positives (flagging real but rarely cited statutes due to graph coverage gaps), it produces no false negatives regarding fabricated provisions.
Open Resources: The citation graph, evaluation framework, and CG-DPO dataset are released as open resources to facilitate further research.

The authors note limitations, including the graph's reliance on existing court decisions (coverage gaps for new laws), the current implementation of temporality (checking existence rather than specific version validity), and the focus on civil law systems, suggesting future work for common law adaptation and end-to-end generation evaluation.