Technical Summary: Unraveling the Ai2 Asta Scholarly Research Assistant Citation System

Problem Statement

Despite the rapid integration of "Deep Research" (DR) tools into academic workflows, there is a scarcity of empirical evidence regarding the operation, stability, and potential biases of their citation systems. While Large Language Models (LLMs) have transformed research activities, their application in literature synthesis is often hindered by issues such as hallucinations, ghost references, and a lack of transparency in source attribution. Although tools utilizing Retrieval Augmented Generation (RAG) have improved transparency, advanced autonomous DR systems (like Ai2 Asta) introduce complex, iterative search and reasoning processes that may obscure how specific publications are selected for citation. There is a critical need to understand whether these tools introduce instability or bias into the scholarly record, potentially affecting the citation-based impact of publications and the reproducibility of literature reviews.

Methodology

This study employs an exploratory, quantitative approach to evaluate the citation system of Ai2 Asta, a scholarly research assistant launched by the Allen Institute for Artificial Intelligence (Ai2). The tool combines Deep Search (iterative, multi-step searching) with RAG, utilizing the Semantic Scholar API to access over 108 million abstracts and 12 million full-text snippets.

Experimental Design:

• Queries: Ten domain-specific questions were formulated, focusing on topics within quantitative science studies (e.g., Webometrics, h-index, Altmetrics, Open Access impact). These topics were chosen for their potential to generate comparative data and diverse viewpoints.
• Data Collection: Two independent rounds of data collection were conducted (October 29 and November 2) using identical configurations, machines, and locations to minimize technical variables.
• Metrics Extracted: For each generated report, the researchers extracted:
  • Search parameters (duration, number of retrieved passages, abstracts, and relevant papers).
  • Report structure (number of sections, comparative tables, word count).
  • Citation data: In-text citations, the final list of cited references (in BIB format), and bibliographic metadata (venue, year, citation counts).
• Analysis: Descriptive statistics were used to analyze citation intensity and diversity. Venn diagrams visualized the overlap of cited references between the two data collection rounds to assess stability. Additionally, the study analyzed the "Asta Summary Citation Counts" dataset (derived from 113,000 queries) to contextualize specific findings against broader usage patterns.

Key Results

1. Citation Intensity (RQ1)
The reports generated by Ai2 Asta exhibit high citation intensity.

• Volume: The mean number of cited references per report ranged from 22.2 to 27.3 across the two rounds.
• Density: There is a high ratio of in-text citations to unique references (mean ~76 in-text citations per report), indicating that specific sources are cited repeatedly throughout the text.
• Correlation: Longer reports generally contained more references, though this was not a universal rule. Notably, the number of final citations often exceeded the number of "relevant papers" retrieved during the deep search phase, suggesting an opaque selection mechanism occurs during report generation.

2. Citation Stability (RQ2)
The study identified notable instability in the citation system.

• Variability: Identical queries submitted at different times yielded significantly different sets of cited references.
• Overlap: The overlap between the two data collection rounds varied drastically by query, ranging from a low of 34.2% (Query #6) to a high of 95.5% (Query #10).
• Implication: The system does not consistently retrieve or select the same evidence base for the same question, raising concerns about reproducibility.

3. Citation Diversity and Bias (RQ3)

• Venue Concentration: While the set of sources was diverse, citations were heavily concentrated in a few venues. Scientometrics, PLoS ONE, and arXiv.org collectively accounted for approximately 30% of all cited references.
• Field Bias: The tool showed a strong bias toward fields with high publication volumes on arXiv and in multidisciplinary journals (e.g., Medicine, Computer Science).
• Recency: A significant proportion of cited works were recent (published 2023 or later), though this varied by query.
• Metadata Issues: The study noted inaccuracies in venue normalization (e.g., inconsistent journal name abbreviations) and missing source information for some references, which complicates bibliometric analysis.

Significance and Claims

The authors position this work as the first study to analyze the citation system of a Deep Research tool from an informetric perspective.

• Reproducibility Concerns: The primary finding is that while Ai2 Asta produces well-structured, high-quality reports, its citation system lacks stability and transparency. The disconnect between retrieved documents and final citations, combined with the variability of results for identical queries, poses challenges for quantitative science studies that rely on reproducible data.
• Impact on Scholarly Communication: The authors argue that the increasing use of DR tools could inadvertently shape researchers' citation behaviors. If a tool consistently selects a specific, potentially biased subset of literature (e.g., favoring arXiv or specific multidisciplinary journals), it may alter the citation-based impact of publications and authors in ways that are not transparent to the user.
• Call for Transparency: The study concludes that for AI-based scholarly assistants to be used responsibly in research evaluation and synthesis, there must be greater transparency regarding the selection mechanisms that determine which retrieved documents are ultimately cited.
• Future Directions: The paper modestly suggests that future research should expand the scope of queries across different disciplines, evaluate the structural coherence and argumentative strength of the reports, and conduct longitudinal studies to track how system updates affect citation patterns over time.

The authors emphasize that these findings are intended to guide researchers, bibliometricians, and developers toward a more critical and informed use of AI research assistants, rather than to dismiss their utility.