🤖 AI

Building Agent Harnesses for Scientific Curation from Multimodal Sources

This paper introduces Beaver, an agent harness designed to improve scientific curation from multimodal sources by integrating frontier agents with task scaffolding, multimodal evidence tooling, and provenance tracking to achieve an 81.0 GRAS score, significantly outperforming existing agents by enabling auditable, iterative refinement of structured data extraction.

Original authors: Sheng Zhang, Qin Liu, Renqian Luo, Shufang Xie, Reuben Tan, Sean Hayes, Gregory Bryman, Wendong Ge, Roxy Zhang, Oluwaseun Egbelowo, Kelly Yee, Hoifung Poon

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sheng Zhang, Qin Liu, Renqian Luo, Shufang Xie, Reuben Tan, Sean Hayes, Gregory Bryman, Wendong Ge, Roxy Zhang, Oluwaseun Egbelowo, Kelly Yee, Hoifung Poon

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a librarian trying to turn a massive, messy library of scientific books into a neat, searchable spreadsheet. But these aren't normal books. They are filled with long paragraphs of text, dense tables of numbers, and complex charts.

The problem is that the information you need isn't in one place. To fill out a single row in your spreadsheet, you might need to read a sentence in the introduction, look at a number in a table on page 10, and check a trend line in a graph on page 15. Then, you have to do some math to make sure the units match (like converting "milligrams" to "grams") before writing it down.

The Problem: The "Smart" Robot Gets Lost
Scientists have built very smart AI robots (called "frontier agents") that can read and write. However, when you give them this messy library task, they often fail. They might get the general idea of the story but miss the specific numbers in the charts. They tend to copy the wrong text or get confused by the layout. It's like giving a brilliant student a test where the answers are hidden in different chapters, and they just guess the answer from the first page they see.

The Solution: Beaver, the "Agent Harness"
The researchers built a system called Beaver. Instead of just relying on the smart robot's brain, Beaver acts like a specialized workbench or a construction harness for the robot.

Think of it this way:

The Robot is the worker.
Beaver is the scaffolding, the tools, and the foreman that guides the worker.

Beaver doesn't just say, "Go read this paper and fill out the form." It breaks the job down into a strict, step-by-step assembly line:

Prep: It converts the messy PDF into a clean, easy-to-read digital outline.
Search: It tells the robot exactly where to look for specific clues.
Read: It gives the robot special "glasses" (tools) that can zoom in on charts and tables to read the numbers accurately, rather than just guessing what the squiggly lines mean.
Check & Trace: Every time the robot writes a number, Beaver forces it to point to exactly where it found that number (like a citation).
Normalize: It makes sure all the units are consistent (e.g., making sure everything is in "years" and not a mix of "months" and "years").

The "Self-Improving" Loop
Here is the clever part: Beaver doesn't just run once and stop. It has a self-improvement loop.

Beaver runs the task.
It looks at its own mistakes (the "artifacts" or logs of what went wrong).
It says, "Hey, the robot kept messing up the charts. Let's change the tool it uses to read charts."
It updates its own instructions and tries again.
It keeps doing this, fixing its own workflow step-by-step, until it gets really good at the job.

The Results
The paper tested Beaver against other top-tier AI systems.

Other Systems: They scored around 58% (like getting a D+ on a test). They struggled with the charts and the complex reasoning.
Beaver: It scored 81% (an A-).

The researchers found that the "brain" (the AI model) wasn't the most important part. The most important part was the harness—the way the task was organized, the tools provided, and the ability to learn from mistakes. Even when Beaver used the same "brain" as the other systems, it performed much better because it had a better system for doing the work.

Why This Matters
In the world of drug discovery and science, getting these numbers right is crucial. If a scientist misses a number in a chart, it could lead to bad decisions later. Beaver proves that if you build a smart, structured, and self-correcting system around an AI, you can turn a chaotic pile of scientific papers into reliable, organized data much better than just letting the AI "wing it."

In a Nutshell:
Beaver isn't just a smarter robot; it's a better manager for the robot. It provides the right tools, breaks the big job into small steps, and teaches the robot how to fix its own mistakes, turning a chaotic research task into a clean, accurate process.

Technical Summary: Building Agent Harnesses for Scientific Curation from Multimodal Sources

Problem Definition

Scientific discovery workflows increasingly rely on converting unstructured literature into normalized, structured records for downstream analysis, particularly in evidence-intensive domains like drug discovery. However, current automated approaches struggle with scientific curation from multimodal sources. The core challenge is that key evidence is scattered across long text, dense tables, and figures. Unlike standard document question-answering tasks, constructing a single curated record often requires:

Cross-modal reasoning: Linking interventions described in text methods to outcomes reported in tables or plots.
Normalization: Standardizing endpoints, units, values, and sample counts into a target schema.
Provenance tracking: Auditing each extracted value back to specific source evidence.

Existing benchmarks often focus on lightweight extraction (e.g., table detection or claim verification) or single-modality tasks, failing to capture the complexity of full-paper curation into rich, production-style schemas. Furthermore, frontier agents alone exhibit brittleness in these settings, often recovering high-level metadata but failing on high-value quantitative attributes derived from complex visual evidence.

Methodology: The Beaver Harness

The authors introduce Beaver, an agent harness designed to extract structured information from scientific papers while preserving provenance. Rather than relying solely on base model improvements, Beaver treats the harness itself as the object of iterative optimization. The system integrates three interacting components:

1. Multimodal Evidence Tooling

Beaver equips the base agent with an auxiliary tool set ( $U$ ) to improve access to scientific evidence:

Paper-to-Markdown Transformation: Converts articles into outlines and section-level markdown files to facilitate navigation of long documents.
Parsed Table Viewer: Enables the agent to read structured table data.
Calibrated Gridline Overlay: A tool for reading quantitative values from plots by overlaying gridlines, allowing the agent to recover precise numerical data from figures.

2. Task Scaffolding

Instead of a single-pass extraction, Beaver decomposes the curation task into sequential, controllable stages ( $k \in \{1, \dots, K\}$ ). Each stage predicts a cumulative table $\hat{Y}^{(k)}$ extending the attributes of previous stages. The workflow for each stage includes:

Preprocessing relevant documents, tables, and figures.
Navigating to candidate evidence.
Performing multimodal reasoning and annotation.
Extracting values with provenance traces.
Normalizing entities and units.
Reviewing and refining the output.

This decomposition reduces ambiguity, narrows the search space for each step, and allows for stage-localized failure diagnosis.

3. Artifact-Grounded Autoresearch

Beaver employs a two-level optimization loop (Algorithm 1) to iteratively improve the harness ( $H_t$ ) using persistent filesystem artifacts:

Outer Loop: Evaluates the current harness on a development set, decomposes performance by stage, and identifies the lowest-performing stage ( $k^*$ ).
Inner Loop: Initializes a stage-specific workspace containing logs, trajectories, and provenance traces. It proposes targeted patches (changes to scaffolds, tools, or prompts), applies them, and re-runs the specific stage.
Selection: Based on the new artifacts and performance metrics, the revision is either accepted as the new best harness or discarded. This process ensures that improvements are grounded in evidence and that unsuccessful patches do not accumulate.

Key Contributions

A New Benchmark: The authors constructed a benchmark for full-paper multimodal scientific curation using a professionally curated corpus of 425 PubMed articles on Alzheimer's disease. The benchmark features a rich schema (400+ attributes), expert-curated reference data, and requires long-horizon extraction across text, tables, and figures.
The Beaver Harness: A novel architecture combining multimodal tooling, task scaffolding, and artifact-grounded autoresearch.
Empirical Evidence on Harness Design: The study demonstrates that for complex scientific curation, the organization of the agent harness is a central determinant of performance, often outweighing the choice of the base frontier model.

Results

Experiments were conducted on a set of 23 full papers using the Gold-Referenced Attribute Score (GRAS), an attribute-level measure of agreement with gold curated records.

Performance: Beaver achieved a GRAS of 81.0%, outperforming the strongest frontier agent baselines (Codex, Claude Code, Copilot CLI, Meta-Harness) by over 23 absolute points (e.g., Codex scored 58.6%).
Ablation Studies: Removing key components significantly degraded performance:
- No Task Scaffolding: 60.5% (largest drop, indicating the necessity of staged workflows).
- No Multimodal Evidence Tooling: 66.1% (highlighting the difficulty of raw visual reasoning).
- No Provenance Traces: 70.7% (showing that traces aid the autoresearch loop in debugging and refining).
Compute-Performance Trade-off: Beaver sits on the empirical Pareto frontier, achieving the highest accuracy with a token budget comparable to other autoresearch variants. Variants without task scaffolding consumed significant compute but failed to reach comparable performance, suggesting that unstructured agent effort is inefficient.
Attribute-Level Analysis: The largest gains were observed on high-value attributes requiring cross-modal reasoning (e.g., endpoint.unit, value, percentchange), confirming that the harness effectively addresses the specific bottlenecks of multimodal extraction.

Significance and Claims

The paper claims that harness design is a central determinant of agent performance for scientific curation from multimodal sources. The authors argue that:

Scientific curation is a "long-horizon" agent task requiring document navigation, evidence alignment, and auditability, which standard frontier agents cannot reliably execute in a single pass.
The surrounding harness—specifically the provision of multimodal tools, the decomposition of tasks into stages, and the use of artifact-grounded feedback loops—can transform a strong base agent into a highly effective curation system.
Auditability is critical: The inclusion of provenance traces and the ability to diagnose stage-localized failures are not just optional features but essential for responsible agentic curation, as errors in normalized records can propagate into downstream scientific decisions.

The work suggests that future improvements in scientific AI will depend less on simply scaling base models and more on developing specialized, iterative harnesses that can manage complex, multimodal evidence and self-correct through grounded diagnosis.