Technical Summary: Parsing and Evaluating Source Attribution in LLM Deep Research Agents

Problem Statement

Large Language Models (LLMs) power "deep research agents" that synthesize information from hundreds of web sources into cited reports. However, the citations produced by these systems cannot be reliably verified. Current approaches suffer from two primary limitations:

1. Unverified Self-Citation: Trusting models to self-cite accurately risks hallucinated or misattributed references.
2. Insufficient RAG Validation: Retrieval-Augmented Generation (RAG) systems often fail to validate whether cited sources are accessible, topically relevant, or factually consistent with the claims they support.

Existing benchmarks (e.g., AttributionBench, CiteME) focus on binary attribution classification or citation matching but lack an end-to-end framework that combines citation extraction with multi-dimensional quality assessment (accessibility, relevance, and factual accuracy) in live web contexts. This gap raises a critical question regarding the reliability of citations encountered daily by millions of users.

Methodology

The authors introduce a reproducible, three-stage source attribution evaluation framework designed to assess inline citations from LLM-generated Markdown reports at scale.

1. Deterministic Citation Extraction

Unlike methods relying on LLM inference for extraction, the framework employs a Markdown Abstract Syntax Tree (AST) parser.

• Process: The parser canonicalizes the Markdown, removes code blocks, constructs an AST, and traverses it to extract citation-claim pairs.
• Logic: It handles various citation formats (numbered references, footnotes, inline links) and implements "backward attribution," where a citation at the end of a passage applies to all preceding uncited sentences.
• Output: A structured AttributionDocument containing deduplicated URLs and text spans associated with specific citation identifiers.

2. Source Retrieval

For every unique citation, the framework retrieves the actual web content. A web content extractor capable of handling JavaScript-rendered pages fetches the source material, truncating it to 5,000 characters for evaluation.

3. Multi-Dimensional Evaluation

The framework evaluates each citation-claim pair across three complementary dimensions using LLM-as-a-judge evaluators (calibrated through human review):

• Link Works: A binary check (1/0) verifying URL accessibility. It identifies broken links, paywalls, and HTTP errors (404, 403) without LLM inference.
• Relevant Content: An LLM judge determines if the retrieved source addresses the same topic as the claim, producing a binary score with a natural language explanation.
• Fact Check: The most stringent dimension. An LLM judge verifies if specific factual claims (numbers, dates, assertions) in the text are supported by the retrieved source content. Scores are binary (1 for supported/consistent, 0 for contradicted/absent/uncertain).

Experimental Setup

• Models: 14 LLMs (OpenAI, Anthropic, Google, and Open-Source) were benchmarked on 130 research queries from DeepResearch Bench and BrowseComp.
• Ablation Study: Two frontier models (GPT-5.4 and Claude Opus 4.6) were tested across seven search depth intervals (2 to 150 tool calls) to isolate the effect of search depth on citation quality.

Key Results

1. Disconnect Between Surface Quality and Factual Accuracy

Even the strongest frontier models exhibit a critical gap between surface-level metrics and factual reliability:

• Link Validity: >94% across all models.
• Topical Relevance: >80% for frontier models.
• Factual Accuracy: Significantly lower, ranging from 39% to 77%.
• Implication: Users are likely to find working, relevant links, but the specific facts attributed to those sources are unsupported nearly half the time.

2. Provider and Model Differences

• Open-Source Models: Struggle significantly with the generation task itself. Success rates for generating valid cited reports ranged from 17% to 40%, compared to 83–100% for frontier models. They also made fewer tool calls (7–60 vs. 73–211).
• Quantity vs. Quality Trade-off: OpenAI models generated the most citations (e.g., GPT-5 Mini with 1,272 attributions) but had lower Fact Check scores (39–59%). Anthropic models (e.g., Claude Opus 4.5) achieved the highest Fact Check accuracy (76.8%) despite lower task success rates, suggesting selective citation may be more effective than exhaustive citation.

3. The Information Overload Effect

The ablation study revealed a counter-intuitive relationship between search depth and accuracy:

• Factual Degradation: As search depth increased from 2 to 150 tool calls, Fact Check accuracy dropped by approximately 42% on average.
  • GPT-5.4 saw a steep decline from 79% to 17%.
  • Claude Opus 4.6 declined from 80% to 58%.
• Stable Surface Metrics: Link Works and Relevant Content scores remained stable (>92%) regardless of search depth.
• Conclusion: Increased retrieval volume does not improve factual synthesis; rather, it impairs it, likely due to attention dilution and the difficulty of synthesizing facts across a growing number of sources.

Contributions and Significance

The paper claims the following contributions:

1. First End-to-End Framework: It introduces the first system combining deterministic citation extraction with multi-dimensional quality assessment (accessibility, relevance, factual accuracy) for live web citations.
2. Comprehensive Benchmarking: It provides the first systematic comparison of 14 LLMs across major providers and open-source alternatives, identifying factual accuracy as the primary differentiator and point of failure.
3. Evidence of Information Overload: Through ablation studies, the authors provide empirical evidence that increasing search depth degrades factual accuracy while surface metrics remain stable, challenging the assumption that "more retrieval equals better answers."

Significance:
The authors argue that the high validity of links and relevance of topics creates a "false sense of trust" in LLM-generated research. Their framework serves as a necessary infrastructure to assess and expose the disconnect between surface-level citation quality and factual reliability. They suggest that for system designers, retrieval strategies should prioritize the depth of source understanding over the breadth of coverage, and for users, these findings highlight the risks of relying on unverified citations in high-stakes domains like healthcare, law, and finance.