⚛️ high-energy experiments

Towards LLM-Powered Automation of a Dark Matter Constraint Repository

This paper presents an LLM-powered pipeline that automates the extraction and integration of dark matter constraints from arXiv papers into community repositories, achieving high accuracy in coupling classification and curve reconstruction while highlighting the unresolved challenges of governing AI-generated scientific data.

Original authors: Lanqing Yuan, Karthik Ramanathan

Published 2026-06-23

📖 4 min read🧠 Deep dive

CC BY 4.0

Original authors: Lanqing Yuan, Karthik Ramanathan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a massive, ever-growing library of rules that tell us what Dark Matter (the invisible stuff holding galaxies together) cannot be. Scientists call this a "constraint repository." It's like a giant "Wanted" poster board where researchers list the specific weights and behaviors that Dark Matter is ruled out for having.

Currently, keeping this library up to date is a lonely, exhausting job. One volunteer scientist has to manually read hundreds of new research papers, find the specific graphs, copy the numbers, and redraw the charts. As new discoveries pour in, this human-only system is at risk of collapsing under the weight of the workload.

This paper introduces a robot assistant (powered by Large Language Models, or LLMs) designed to help with this job. Here is how it works, using simple analogies:

1. The Detective (Discovery)

Think of the system as a tireless detective scanning a daily newspaper (arXiv, where scientists publish new papers). It doesn't read every word; it uses keywords and a smart "gut feeling" (AI classification) to spot only the articles that actually contain new rules about Dark Matter.

2. The Translator and the Jury (Extraction)

Once a relevant paper is found, the robot needs to extract the data. This is the hardest part because scientific papers are messy.

The Two-Stage Strategy: The robot first tries to read the text and tables (like reading a recipe). If the data is only hidden inside a complex graph (like a picture of a mountain range), it switches to "vision mode" to look at the image.
The Jury System: AI can sometimes "hallucinate" or misread a number. To fix this, the robot reads the same paper three times independently. Imagine three different people looking at a blurry sign. They vote on what the sign says. If two agree and one disagrees, the majority wins. If they all agree, the robot picks the "median" answer (the one closest to the middle) to avoid wild guesses. This ensures the data is solid.

3. The Dictionary (Convention Canonicalization)

This is the paper's secret sauce. Different scientists use different units to measure the same thing. It's like one person saying "12 inches" and another saying "1 foot." If you don't convert them, you think they are different.

The robot has a special "dictionary" built by a physics expert AI (called GPD). This dictionary knows exactly how to translate every scientist's weird units into the library's standard format.
If the robot encounters a unit it doesn't recognize, it doesn't guess. It flags it for a human to check, rather than making a silent mistake.

4. The Architect (Integration)

Once the data is clean and translated, the robot doesn't just write a report; it writes code. It automatically inserts the new data into the library's software, redraws the charts, and creates a "Pull Request."

Think of this as a construction worker who not only brings the new bricks but also builds the wall and leaves a note for the foreman saying, "I added this new section, please check it."
To make it easy for humans to check, the robot highlights the new data in bright red against a grey background of old data, so the expert can instantly see what changed.

How Well Did It Work?

The team tested this robot on 346 real scientific papers.

Accuracy: It correctly identified the type of Dark Matter rule 90% of the time.
Precision: When it did extract numbers, the results were usually within a factor of two of the "true" answer (which is very good for this complex field).
The Bottleneck: The robot is great at reading standard graphs. Its main struggle is with very rare, weird types of Dark Matter rules where scientists use unique, non-standard units. In these cases, the robot often gets confused because the "dictionary" isn't perfect yet.

The Big Catch

Even though the robot works well, no one has accepted its work yet.
The paper highlights a major problem: We don't have a rulebook for how to trust AI-generated scientific data. The robot can build the wall, but the "foreman" (the human community) doesn't know if they are allowed to let the robot's bricks stay. Until scientists create a formal process to review and approve AI work, the robot sits ready but unused.

In short: The authors built a highly skilled robot that can read, translate, and code new scientific limits for Dark Matter almost as well as a human. The technology works, but the "human rulebook" for accepting AI help hasn't been written yet.

Technical Summary: Towards LLM-Powered Automation of a Dark Matter Constraint Repository

Problem Statement
The experimental search for ultralight dark matter has accelerated, generating a dense landscape of constraints across at least 15 distinct coupling types (e.g., axion–photon, dark photon kinetic mixing). While repositories like AxionLimits compile these bounds into publication-quality exclusion plots, they rely on manual maintenance by individual volunteers. This creates a sustainability risk as the volume of new results outpaces human capacity. The manual workflow involves finding new results, digitizing figures or tables, converting parameterizations (which vary by paper and detector), writing plotting code, and regenerating figures. The core challenge is automating this loop without introducing errors in physics conventions or misinterpreting complex log–log exclusion plots.

Methodology
The authors present an end-to-end pipeline that automates the discovery, extraction, integration, and review of dark matter constraints. The system architecture consists of four primary stages:

Discovery: The system performs daily monitoring of arXiv using keyword filtering and Large Language Model (LLM) relevance classification to identify pertinent papers.
Two-Stage Extraction with Consensus Voting:
- Stage 1 (Text/Tables): The PDF is parsed, and text/tables are processed by an LLM to extract coupling types, mass–coupling data points, and declared units/conventions in structured JSON.
- Stage 2 (Vision Fallback): If Stage 1 yields insufficient data (e.g., fewer than 5 points or no valid mass window), the system renders figure pages to images. An LLM reads axis labels, tick marks, and curve positions from log–log exclusion plots.
- Consensus Mechanism: To address LLM non-determinism, each paper is read independently three times ( $N=3$ ). The coupling type is determined by majority vote. The final curve is selected as the medoid (the candidate with the smallest median pairwise distance to others in log–log space) rather than an average, ensuring the output is a real physical sample and not a fabricated hybrid.
- Quality Tiering: When candidates disagree, they are ranked by a lexicographic tuple prioritizing physical validity, non-degeneracy (spanning $\ge$ 1 dex), source tier (tables > text > vision), and corroboration.
Convention Canonicalization: A critical component addresses the fact that the same physics is often reported in incompatible variables (e.g., $g_{aNN}$ vs. dimensionless rescaled versions) across papers. The system uses a per-file registry, derived and verified using the agentic physics assistant Get Physics Done (GPD), to canonicalize extracted and reference curves before scoring. This layer flags unmappable cases rather than emitting incorrect numbers.
Integration and Review: The pipeline generates code (using Python's AST module for safe insertion into plotting classes), updates Jupyter notebooks, and regenerates plots. Updates are submitted as Pull Requests (PRs) containing a "highlighted" plot where existing limits are greyed out and the new proposal is shown in red, facilitating rapid visual verification by domain experts.

Key Contributions

End-to-End Automation: The first system to automate the full loop from arXiv discovery to code integration and human review for a live scientific repository.
Robust Extraction Stack: A novel approach treating LLM reads as noisy samples reconciled via consensus voting and source-quality tiering, paired with a physics convention-canonicalization layer built via code-verified derivation (GPD) rather than prompting.
Benchmark and Scoring: A 346-paper benchmark where ground truth is the upstream-curated repository itself. The scoring methodology separates genuine extraction error from non-comparability (e.g., wrong coupling or incompatible conventions).
Shift in Error Sources: Empirical evidence that log–log plot reading is no longer the dominant error source; the frontier has shifted to deterministic, convention-bound rare coupling types.

Results
Evaluated on a 346-paper benchmark (271 comparable papers):

Classification: The pipeline correctly classifies the coupling type for 90.5% of papers.
Extraction Accuracy: The median coupling residual is 0.33 dex (approximately a factor of two). 48.4% of curves fall within a factor of two of the ground truth, with a mean mass-range coverage of 76.2%.
Source Performance: Figure-vision extraction (median residual 0.338 dex) has achieved parity with text/table extraction (0.326 dex), indicating that consensus voting and quality-tier selection have closed the historical gap in reading dense plots.
Subsets: Performance varies by coupling type. Common types like Axion–Photon achieve high accuracy (0.207 dex), while rare types with idiosyncratic conventions (e.g., AxionEDM) show higher residuals (4.649 dex) and lower classification accuracy.
Deployment: The pipeline is deployed and has generated limit proposals. However, none have merged yet, highlighting that governance for AI-generated scientific data remains an unsolved problem.

Significance and Claims
The paper claims that the primary bottleneck in automating scientific data extraction has shifted from stochastic perception (reading plots) to deterministic domain knowledge (handling physics conventions). The authors argue that successful automation requires coupling stochastic perception tools (LLMs with voting) with verified, auditable domain-knowledge layers (convention registries).

The work underscores that while the technical pipeline is functional, the scientific community lacks a governance framework for AI-generated data. The authors recommend establishing a community editorial board (similar to PDG reviewers) and suggest that publishing machine-readable limit data alongside papers would bypass the extraction bottleneck entirely. The paper concludes modestly, noting that human review remains essential, particularly for the 10% of proposals targeting the wrong plot due to classification errors or rare coupling complexities.