DOME Copilot: Making transparency and reproducibility for artificial intelligence methods simple

DOME Copilot is a scalable, large language model-based solution designed to enhance the transparency and reproducibility of artificial intelligence in life sciences by automatically extracting structured method reports from manuscripts to interpret and annotate global AI literature.

The Gist

Imagine you've just baked the world's most delicious, revolutionary cake. You've shared the recipe with the world, but there's a catch: you wrote the instructions on a napkin in crayon, using vague phrases like "add some flour" and "bake until it smells good."

Now, imagine a thousand other bakers trying to recreate your cake. They can't. They don't know how much flour, what temperature the oven was, or what kind of eggs you used. The result? A crisis of confusion where no one can replicate your success.

This is exactly what is happening in the world of Artificial Intelligence (AI) in science. Scientists are making amazing discoveries using AI, but they often publish their "recipes" (methods) in a messy, incomplete way. This makes it impossible for others to check, reuse, or trust the results.

Enter DOME Copilot, the solution described in this paper. Think of it as a super-smart, automated sous-chef designed to fix this mess.

The Problem: The "Black Box" Recipe

Currently, when scientists publish AI research, they often leave out crucial details. It's like a magician refusing to explain how the trick was done. Because of this, the scientific community is facing a "reproducibility crisis"—nobody can repeat the experiments to see if they work.

To fix this, there is a set of rules called the DOME Recommendations. These are like a strict, standardized recipe card that scientists should fill out. But here's the problem: filling out these cards is boring, difficult, and takes hours. Most scientists are too busy doing the actual science to spend hours writing a perfect report. So, they skip it, and the "black box" problem continues.

The Solution: The DOME Copilot

The authors built a tool called DOME Copilot. Think of it as a robotic librarian who is incredibly fast at reading books and filling out forms.

Here is how it works, step-by-step:

1. The Input (The Napkin): A researcher uploads their messy scientific paper (PDF) to the DOME Copilot.
2. The Brain (The AI): The Copilot uses a powerful "Large Language Model" (a type of AI that reads and understands human language) to scan the paper. It acts like a detective, hunting for specific clues: What data did they use? What computer code did they run? How long did it take?
3. The Output (The Perfect Recipe): Instead of the researcher spending hours writing a report, the Copilot instantly generates a clean, structured, and standardized "DOME Report" based on what it found in the paper.
4. The Human Check: A human can quickly review the robot's work, fix any small mistakes, and then submit it to a public database (the DOME Registry).

Why is this a Big Deal?

The paper tested this tool on over 2,000 scientific papers, and it worked remarkably well.

• Speed: It does in two minutes what used to take a human hours.
• Accuracy: It understands the context of the paper almost as well as a human expert. If the paper doesn't have the answer, the Copilot honestly says, "I couldn't find this information," rather than making things up.
• Scalability: Because it's so fast, we can now go back and "clean up" thousands of old, messy AI papers from the past, turning them into clear, usable recipes.

The Three Superpowers of DOME Copilot

The paper highlights three main ways this tool helps:

1. The Self-Check: Before a scientist even submits their paper, they can use the Copilot to see, "Oh, I forgot to mention what software I used!" It helps them fix their own work early.
2. The Publishing Assistant: When a paper is sent to a journal, the Copilot can automatically draft the required report. This makes the job of journal editors and reviewers much easier, as they get a clear, standardized summary immediately.
3. The Time Machine: We can use the Copilot to scan the entire history of AI literature. It can go through thousands of old papers and extract the missing details, creating a massive, searchable library of clear AI methods that anyone can use.

The Bottom Line

DOME Copilot is a bridge between the messy reality of scientific writing and the clean, organized world of reproducible science.

By automating the boring, difficult part of reporting, it removes the barrier that stops scientists from being transparent. It turns the "black box" of AI into a glass box, where everyone can see exactly how the magic works. This doesn't just make science more honest; it makes it faster, safer, and more useful for everyone.

In short: It's the tool that ensures the next great AI discovery isn't just a lucky guess, but a recipe anyone can follow.

Technical Summary

1. Problem Statement

Despite unprecedented breakthroughs in life science research driven by Artificial Intelligence (AI), a significant reusability and reproducibility crisis persists. Many published AI methods act as "black boxes" where crucial disclosures (e.g., model architecture, hyperparameters, data sources) are either absent, obfuscated, or presented in unstructured formats.

• The Bottleneck: While reporting guidelines like the DOME Recommendations exist to standardize metadata, adoption is low. Manually compiling structured AI method metadata is time-consuming (taking hours per paper) and often unrewarded for researchers.
• The Consequence: Publishers struggle to enforce these guidelines, and researchers lack motivation to comply, leading to a vast backlog of unstructured AI literature that cannot be easily reused or audited.

2. Methodology: DOME Copilot

The authors developed DOME Copilot, a Large Language Model (LLM)-based tool designed to automate the extraction of structured AI method metadata from manuscript PDFs.

• Architecture & Workflow:

  • Input: Users upload a manuscript (PDF) and optional supplementary materials via a Gradio interface.
  • Processing Pipeline:
    1. Parsing: The PDF is parsed and text is extracted using an inference workflow built with LlamaIndex.
    2. Embedding: Text chunks are converted into vector embeddings using the Qwen3-Embeddings-4B model and stored in a vector index.
    3. Generation: An intermediate-sized LLM, Mistral Small 3.1 24B Instruct, utilizes Retrieval-Augmented Generation (RAG) and structured prompt guidance to generate the DOME-compliant annotations.
  • Output: The system produces a structured JSON file suitable for ingestion into the public DOME Registry.
  • Human-in-the-Loop: For individual submissions, the output can be reviewed and edited by humans before final submission. For bulk archival processing, outputs are transparently labeled as "AI-generated without human refinement."

• Development Iteration (v0 to v2):

  • The system underwent iterative refinement based on human expert curation of a benchmark dataset.
  • Key Improvements: The transition from v0 to v2 focused heavily on prompt engineering to reduce verbosity, enforce strict formatting, and eliminate redundancy. This resulted in significantly shorter, more targeted responses (Figure 1.F).

3. Key Contributions

• Automated Extraction Tool: The first scalable solution to automatically convert unstructured AI manuscripts into structured DOME-compliant reports.
• Scalable Infrastructure: A system capable of processing vast backlogs of literature (thousands of papers) that was previously impossible to curate manually.
• Integration Strategy: Designed to be embedded directly into journal submission workflows and discovery platforms (Europe PMC, EBI Search) to lower adoption barriers.
• Open Source: The code, data, and a public demo (Gradio interface) are made available to the community.

4. Results & Evaluation

The system was evaluated using three distinct datasets:

• Semantic Similarity (Dataset A):
  • Metric: BERTScore comparing 192 AI-generated annotations against human-curated ground truth.
  • Performance: The model achieved upper and lower quartiles between 0.35 and 0.50. While not a perfect match (BERTScore 1.0), this indicates stable semantic similarity. The authors note that human annotations vary in quality and style (sometimes copying raw text), whereas the AI generates succinct, structured summaries.
• Positive Generation (Dataset B - 1,012 AI papers):
  • Success Rate: High success in extracting available information.
  • Failure Modes: "Full generation" failures correlated with known underreported fields (e.g., model execution duration). "Partial generation" occurred in multi-field questions where only partial data (e.g., URL but no license) was present in the text.
• Negative/Rejection Performance (Dataset C - 1,012 non-AI papers):
  • Hallucination Avoidance: The model demonstrated robustness against non-relevant content.
  • Rejection Rates: It successfully rejected 546 (54%) of "strong negatives" (e.g., economics papers) during preprocessing. For "soft negatives" (e.g., AI literature reviews), it correctly returned "Not enough information" rather than hallucinating data.
• Efficiency: The average generation time is approximately two minutes per document, regardless of size.

5. Significance and Future Impact

• Solving the Scalability Crisis: DOME Copilot addresses the "intractable" problem of manual curation, enabling the systematic annotation of the global AI literature corpus.
• Enhanced Transparency: By standardizing method reporting, it facilitates better peer review, identifies methodological gaps early, and allows researchers to discover and reuse AI methods more effectively.
• Three Core Use Cases:
  1. Self-Check: Helps authors identify missing metadata before submission.
  2. Workflow Aid: Automates the drafting of structured reports for publishers/reviewers.
  3. Bulk Archival: Enables the retroactive structuring of historical AI literature.
• Broader Applicability: The framework serves as a proof-of-concept for applying LLMs to enforce reporting guidelines in other scientific domains where reproducibility is critical.

In conclusion, DOME Copilot represents a critical infrastructure advancement, shifting the burden of metadata compliance from a manual, researcher-centric task to an automated, scalable process, thereby unlocking the full potential of AI in life sciences.