Technical Summary: Wiki Live Challenge

Problem Statement

Deep Research Agents (DRAs) have demonstrated significant potential in autonomous information retrieval and report generation. However, existing evaluation frameworks suffer from two critical limitations:

1. Unreliable References: Current benchmarks often rely on reports generated by other Large Language Models (LLMs) or static web data as references. These sources lack the rigorous quality assurance, neutrality, and verifiability of human-expert content.
2. Subjective Evaluation: Evaluation metrics frequently rely on LLM-derived dimensions or coarse-grained rubrics that may deviate from human expert expectations, failing to provide objective, fine-grained assessments of writing quality and factual accuracy.

Consequently, there is a lack of a reliable benchmark to measure the gap between current DRAs and the high standards of expert-level research, specifically regarding the ability to produce neutral, comprehensive, and verifiable long-form articles.

Methodology

To address these gaps, the authors introduce Wiki Live Challenge (WLC), a live benchmark designed to evaluate DRAs against the latest Wikipedia Good Articles (GAs).

1. Data Construction (WLC)

• Source: The benchmark utilizes 100 recent Wikipedia Good Articles (GAs) created between March 1, 2025, and December 1, 2025.
• Selection Criteria: Articles were filtered to ensure they passed Wikipedia's strict human expert review process, adhering to core policies of neutrality, comprehensiveness, and verifiability.
• Diversity: The dataset spans 15 major categories (e.g., History, Natural Sciences, Media) and excludes simple list-based articles to ensure task complexity.
• Live Nature: The dataset is curated from recent content to mitigate data contamination from current model training cutoffs.

2. Evaluation Framework (Wiki Eval)

The evaluation framework, Wiki Eval, is grounded strictly in the official Wikipedia Good Article criteria and comprises two primary dimensions:

• Wiki Writing (Quality Assessment):

  • Based on three core GA dimensions: Well-written, Neutral, and Broad in its coverage.
  • Consists of 39 fine-grained criteria derived from Wikipedia's Manual of Style (e.g., lead section structure, avoidance of "peacock terms," proper attribution).
  • Mechanism: Uses an "LLM-as-a-Judge" approach where a Judge LLM (Gemini-2.5-pro) compares the DRA-generated article against the human-written Wikipedia reference for each criterion to determine a winner.

• Wiki Fact (Factual Assessment):

  • Coverage (Cov. Wiki): Measures the proportion of facts extracted from the Wikipedia reference that are covered by the generated article.
  • Reference Accuracy (Ref. Acc.): Verifies if statements in the generated article are supported by their specific cited sources using a fact-checking model and web retrieval (Jina Reader).
  • Conflict Analysis: Tracks rates of statements conflicting with Wikipedia facts (Wiki Conf.) or the model's own citations (Ref. Conf.).

Key Contributions

1. Wiki Live Challenge (WLC): A live benchmark providing high-quality, human-verified references (Wikipedia GAs) to challenge DRAs in writing expert-level articles.
2. Wiki Eval Framework: A comprehensive evaluation methodology grounded in Wikipedia's editorial standards, featuring 39 writing criteria and rigorous factual metrics (coverage and citation verification).
3. Empirical Analysis: Extensive experiments across diverse DRA systems (both proprietary and open-source) revealing significant performance gaps and analyzing error patterns (hallucinations, citation failures, and category-specific difficulties).

Experimental Results

The authors evaluated various DRAs, including proprietary systems (OpenAI o3, Gemini-2.5/3-pro, Qwen-3-max) and open-source frameworks (Deep Researcher, Tongyi DeepResearch, LangChain).

• Performance Gap: A substantial gap exists between current DRAs and human-written Wikipedia articles. Even the best-performing models (e.g., Gemini-3-pro Deep Research) achieved a Wiki Writing score of 58.33 (out of a potential 100 based on criteria wins), while open-source models like Deep Researcher scored as low as 2.28.
• Factual Coverage: All systems struggled with information coverage. The best agent (Gemini-2.5-pro) covered only ~30.76% of Wikipedia facts, indicating a lack of expert-level information gathering capabilities.
• Citation and Hallucination:
  • Open-source models frequently failed to generate properly formatted citations, making Reference Accuracy unmeasurable.
  • Proprietary models showed varying degrees of hallucination. For instance, LangChain (GPT-4.1) had a high conflict rate with Wikipedia facts (24.69%) despite low citation conflicts, suggesting reliance on unreliable sources.
  • Leakage: Some models (e.g., Perplexity) exhibited high rates of "leakage" (directly citing the target Wikipedia article), yet this did not guarantee high-quality or unbiased output.
• Category Difficulty: Performance varied significantly by domain. History and Mathematics were the most difficult categories (average win rates <20%), while Natural Sciences and Philosophy were easier (>40%). Difficulty correlated moderately with page views but not with article length or citation count.

Significance and Claims

The paper claims that WLC serves as a rigorous, reliable, and reproducible benchmark for advancing Deep Research Agent research. By utilizing Wikipedia Good Articles as a "gold standard," the authors demonstrate that current DRAs fall considerably short of human expert standards in writing quality, neutrality, and factual verifiability.

The authors position WLC not merely as a scoring tool but as a diagnostic framework that:

• Highlights specific failure modes (e.g., inability to handle specialized terminology, bias toward trending topics, and citation hallucinations).
• Validates the necessity of human-expert references over LLM-generated ones for objective evaluation.
• Provides a foundation for future improvements in agent planning, retrieval, and synthesis capabilities.

The work concludes that while DRAs show promise, achieving the comprehensive, neutral, and verifiable standards of human-authored encyclopedic content remains a significant challenge.