💻 computer science

OpenBioRQ: Unsolved Biomedical Research Questions for Agents

The paper introduces OpenBioRQ, a novel retrieval-grounded agentic benchmark comprising 12,553 unsolved biomedical research questions that exposes critical failure modes in current models—such as citation hallucination and tool collapse—while demonstrating that even frontier agents struggle to solve a significant portion of these open-ended, non-saturating tasks.

Original authors: Minbyul Jeong

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Minbyul Jeong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a student taking a very difficult exam. In most medical exams today, the teacher gives you a question and a hidden answer key. If you get the answer right, you pass. If you get it wrong, you fail. But what happens when the teacher asks a question that no one knows the answer to yet? How do you grade a student when there is no answer key?

This paper introduces OpenBioRQ, a new way to test AI "agents" (smart computer programs that can search the internet and read papers) on these unsolved medical mysteries.

Here is the breakdown of what the researchers found, using simple analogies:

1. The Problem: The "Fake Link" Trap

Imagine you are writing a report. You write a claim, like "This new drug cures headaches," and you attach a link to a famous medical journal article to prove it.

The old worry: People thought AI would make up fake links (like linking to a website that doesn't exist).
The new discovery: The researchers found that the AI is actually very good at finding real links. Almost 100% of the links it gives actually work.
The real trap: The AI often links to a real article, but it's the wrong article. It's like linking to a recipe for "Chocolate Cake" when you are trying to prove that "Spicy Tacos cure headaches." The link works, the paper exists, but it has nothing to do with your claim.

The paper calls this the "Wrong-Paper" failure. About 16% of the time, the AI gives you a real, working link that completely fails to support what it just said. This is dangerous because it looks trustworthy at first glance.

2. The New Test: The "Unsolved Mystery" Box

Most medical tests for AI are like a crossword puzzle where the answers are already known. The AI just has to remember the right word.

OpenBioRQ is different. It's like a box of 12,500 unsolved medical mysteries.
The AI has to act like a detective: it must use tools to search databases, read thousands of papers, and try to figure out the answer.
Because there is no answer key, the researchers don't grade the AI on getting the right answer. Instead, they grade it on how honest and careful it is.
- Did it admit, "We don't know yet"? (This is good).
- Did it make up a confident answer with a fake link? (This is bad).
- Did it find a real link that didn't actually prove its point? (This is the "Wrong-Paper" trap).

3. The "Tool Collapse" Phenomenon

The researchers gave the AI a set of "tools" (like a magnifying glass, a library card, and a search engine) to help solve these mysteries.

They expected the AI to use these tools to find the best evidence.
The surprise: On the hardest questions, the AI sometimes just gave up using its tools. It stopped searching and just guessed based on what it remembered from its training.
It's like a detective who, when faced with a really tough case, decides to stop looking for clues and just guess the culprit's name.
The researchers found that for some models, giving them the tools didn't actually help them get a better score. They were "collapsing" and ignoring the very things they were supposed to use.

4. The "Checklist" Solution

How do you grade a test where there is no right answer?

The researchers created a frozen checklist for every single question.
Instead of asking an AI, "Is this a good answer?" (which is vague), they gave it a specific list of things to check, like:
- "Did you mention the specific protein?"
- "Did you admit that we don't have a cure yet?"
- "Did you avoid making up a fake clinical trial?"
This turned a vague judgment into a simple "Yes/No" checklist, making the grading much fairer and more consistent.

5. The Results: Who Passed?

The researchers tested several top-tier AI models (the "frontier agents").

The Difficulty: Even the smartest AI models only solved about 30% to 60% of these unsolved questions. The rest remained mysteries, which is exactly what you want in a hard test (it means the test isn't too easy).
The Citation Issue: Even the best models still fell into the "Wrong-Paper" trap about 10–16% of the time. They found real papers, but they didn't actually support their claims.
The Takeaway: Just because an AI gives you a link that works doesn't mean the information is correct. In the world of medical research, existence is not correctness.

Summary

OpenBioRQ is a new, tough test for medical AI. It stops asking "Do you know the answer?" and starts asking "Can you handle the unknown without lying?"

The main lesson is that AI is getting very good at finding real sources, but it is still very bad at making sure those sources actually prove what it claims. It's like a student who can find the library book but keeps citing the wrong page. Until AI learns to stop doing that, we can't fully trust it to summarize medical research, especially for questions we don't have answers to yet.

Important Note: The authors explicitly state this is a research tool to help scientists understand how AI works. It is not a tool for doctors to make decisions about patient care. You should not use these AI answers to treat a sick person.

Technical Summary: OpenBioRQ

Problem Statement

Current agentic benchmarks in biomedicine largely assume a known, verifiable answer key. This assumption masks critical failure modes: when a model can echo a pre-specified source, it avoids the need to independently verify whether a retrieved paper actually supports its claim. Consequently, existing evaluations fail to detect "wrong-paper" citations (where a real, resolvable identifier points to a paper that does not support the claim) and "agentic collapse" (where agents abandon tool use on difficult questions). Furthermore, standard citation audits often focus solely on the existence of a reference (does the PMID resolve?), missing the more hazardous failure of linking a valid identifier to an unsupported claim.

The paper identifies a gap in the evaluation landscape: the intersection of agentic settings (requiring tool use) and unsolved questions (no ground-truth answer key). This quadrant is underexplored because constructing benchmarks without answer keys is structurally difficult; without a key, it is challenging to verify if a question is genuinely open or if a model's answer is correct.

Methodology

The OpenBioRQ Benchmark

OpenBioRQ is a retrieval-grounded benchmark comprising 12,553 unsolved biomedical research questions across 12 domains. The corpus is constructed via a five-stage pipeline: crawl, extract, refine, deduplicate, and export.

Sources: Questions are extracted from authoritative sources framing open problems, including PubMed, clinical trial registries, arXiv, James Lind Alliance Priority Setting Partnerships, NICE research recommendations, and Cochrane research-gap literature.
Openness Verification: Unlike prior work that relies on source framing, OpenBioRQ employs a retrieval-grounded status verifier. This system gathers real follow-up evidence (e.g., citing papers, trial results) to re-judge whether a question remains open. A re-audit of the core set found 0/657 questions resolved.
Empirical Difficulty: Difficulty is not hand-labeled but derived empirically. A "roster" of three open-weight models (GLM-5.1, Qwen3.6, DeepSeek-V4) attempts every question. A question is labeled core only if all three fail it (score < 0.5). A frozen core of 423 questions is defined as those that all three fail even under deterministic decoding ( $T=0$ ).
Evaluation Protocol:
- Agentic Harness: Models interact with 10 biomedical REST APIs (e.g., PubMed, ClinicalTrials.gov, OpenFDA) for up to 10 rounds of reasoning and tool calls.
- Frozen Checklist Scoring: Instead of free-form LLM judging, each question is graded against a frozen, per-question checklist generated once and fixed. Criteria include "must mention," "must acknowledge" (uncertainty), "must ground" (evidence), and "must avoid" (fabrication). This raises inter-judge agreement (Spearman) from 0.35 to 0.82.
- Two-Level Citation Audit:
  - L1 (Existence): Does the identifier resolve to a real paper?
  - L2 (Content Support): Does the resolved paper actually support the claim?

Key Contributions

First Agentic × Unsolved Benchmark: OpenBioRQ is the first biomedical benchmark to combine an agentic setting with unsolved questions lacking an answer key, treating the task as a probe for faithfulness and abstention.
Two-Level Citation Audit: The paper introduces a rigorous audit distinguishing between citation existence and citation support. It reveals that while biomedical agents rarely fabricate identifiers (L1 failure rate $\approx 0.7\%$ ), they frequently link real papers to unsupported claims (L2 failure rate $\approx 15.9\%$ ).
Agentic Collapse Observation: The benchmark surfaces a failure mode where agents, particularly on the hardest questions, abandon tool use entirely (zero-tool collapse), reverting to parametric recitation.
Empirical Difficulty Construction: The paper demonstrates a method for constructing hard benchmarks without human difficulty labels by using model failure rates to define the "core" set, ensuring the benchmark remains non-saturating as models improve.

Results

Citation Faithfulness

Existence is not Correctness: On a set of 4,863 citations, 99.3% resolved to real papers. However, 15.9% of these resolving citations were "wrong-paper" (did not support the claim). This rate was confirmed by an independent judge family (Opus-4.7) at 10.6%.
Severity Gap: The paper hypothesizes that wrong-paper citations are more hazardous than fabricated ones because the resolvable identifier lends false authority to the claim.

Agent Performance

Non-Saturating Difficulty: Even the strongest frontier agents (Gemini-3-Pro, Opus-4.7, GPT-5.5) solve only 29–60% of the frozen core. The best agent leaves $\sim 33–40\%$ of questions unsolved, indicating the benchmark is discriminating and not saturated.
Agentic Collapse: On the hardest questions, models like GLM-5.1 and DeepSeek-V4 exhibit high zero-tool rates (65% and 62%, respectively). A no-tool ablation for GLM-5.1 showed no performance degradation, suggesting tools stop paying off exactly where they are needed most.
Divergence from Closed-Form QA: Models that cluster tightly on closed-form benchmarks (e.g., MedQA, where performance varies by only 3.9 points) show wide dispersion on OpenBioRQ (spanning 7.6x in solve rates), revealing a resolution gap in current evaluation metrics.

Robustness

Judge Independence: Results are robust to judge family swaps (GLM vs. Opus), with solve rate shifts $\le 4.1$ points and preserved model ordering.
Domain Coverage: The benchmark spans 12 domains, with the frozen core distributed across all of them, ensuring findings are not domain-specific artifacts.

Significance and Claims

The paper claims that OpenBioRQ targets research assistance—specifically evidence retrieval and faithful citation—rather than clinical decision support. Its primary significance lies in exposing a critical reliability gap in agentic biomedical systems: the tendency to attach real, resolvable citations to unsupported claims.

The authors argue that existence checks are "false comfort." In biomedicine, the dominant failure mode is not fabricating references (as seen in math domains) but misattributing real ones. Consequently, L2 content support, not L1 existence, must be the reliability guardrail.

Furthermore, the paper highlights agentic collapse as a structural failure where agents stop using tools on the most difficult questions, rendering tool-augmented benchmarks ineffective at measuring true capability in those regimes. By removing the answer key, OpenBioRQ forces agents to demonstrate grounding and calibrated abstention, behaviors that are invisible in standard closed-form QA.

The benchmark is released as a frozen artifact (fixed list of 423 core questions with a tool-response replay cache) to ensure reproducibility despite the moving target of live biomedical literature and API responses. The authors explicitly state the resource is for evaluating literature grounding and must not be used to guide patient care.