🤖 AI

AI Scientists Are Only as Good as Their Evidence: A Stratified Ablation of Proprietary Data and Reasoning Skills in Drug-Asset Valuation

This paper demonstrates that in drug-asset valuation, while reasoning scaffolds improve an AI agent's calibration and discipline, the availability of proprietary evidence is the definitive limiting factor that sets the upper bound on factual accuracy and decision utility.

Original authors: Yinan Wang

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yinan Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a team of detectives to solve a complex case: Is a new drug idea worth investing millions of dollars in?

To answer this, the detective needs to know three things:

The Science: Does the biology make sense?
The Competition: Who else is trying to do this?
The History: Have similar drugs succeeded or failed before?

The paper you provided is a "stress test" of three different types of AI detectives to see which one is actually the best at solving this case. The researchers wanted to find out: Is the detective's intelligence (the AI model) the most important thing, or is the quality of the evidence they are allowed to read the real bottleneck?

Here is the breakdown of the experiment and what they found, using simple analogies.

The Three Detectives (The Experiment)

The researchers set up three versions of an AI agent, all using the exact same "brain" (the same underlying AI model). The only difference was what tools and information they were allowed to use.

Detective A (The "Web Surfer"):
- Tools: Can only use a standard web search (like Google) and has no special training.
- Analogy: This is a smart intern who goes to the public library and reads whatever books are on the open shelves. They are smart, but they can only find what anyone else can find.
Detective B (The "Trained Analyst"):
- Tools: Has the same web access, but is given a strict playbook (a checklist of rules), a "red team" (a critic to check their work), and access to public databases (like clinical trial registries).
- Analogy: This is the same intern, but now they have a senior mentor giving them a checklist, a rulebook on how to write reports, and access to public government records. They are more disciplined and organized, but they still can't see the "secret files."
Detective C (The "Insider"):
- Tools: Has everything Detective B has, PLUS access to a massive, private, paid database (Noah AI) that contains curated records of drug trials, deals, and competitors that are hidden from the public web.
- Analogy: This is the senior investigator who has a key to the confidential vault. They can see the private emails, the unpublished trial results, and the "long-tail" deals that never made the news.

The Results: What Did They Find?

The researchers tested these detectives on 13 different drug cases. Here is the verdict:

1. The "Playbook" Helps, But Only So Much (Detective B vs. A)

When Detective B (the trained analyst) was compared to Detective A (the web surfer), the results were interesting:

Better Discipline: Detective B wrote better reports. They were less likely to make wild guesses and followed the rules better.
The Problem: Detective B still missed 60% to 75% of the important competitors and deals.
The Analogy: Imagine Detective B is a very polite, well-organized librarian who follows every rule. But if the book they need to find is locked in a private vault, no amount of good organization will help them find it. They are "calibrated" (thinking clearly) but "blind" (missing facts).

2. The "Private Vault" Changes Everything (Detective C)

When Detective C (the insider) was brought in, the results changed dramatically:

Completeness: Detective C found 96% of the critical information. Detective A and B only found about 25% to 38%.
The "Long-Tail" Effect: The biggest gap was in finding "long-tail" data—small, obscure, or regional drug trials that never appear on the open web. Detective C found almost all of them; the others found almost none.
The Analogy: Detective C didn't just write a better report; they saw the whole board. While the others were playing chess with only half the pieces, Detective C saw the entire game.

3. The "Informed Decision" Score

This is the most important finding.

Raw Quality: If you just read the final report without knowing what facts were missing, Detective A and B sounded almost as good as Detective C. They sounded confident and logical.
Real Quality: But because A and B were missing so many facts, their decisions were based on incomplete information.
The Math: The researchers created a score called "Informed Decision Quality."
- Detective A/B Score: ~2.0 (They sounded good, but were missing 75% of the facts).
- Detective C Score: ~7.4 (They sounded good AND had all the facts).
The Ceiling: The paper argues that even if you gave Detective B a "perfect" brain and a perfect report, they would still be capped at a low score because they physically couldn't access the missing evidence. You cannot reason your way to a truth you cannot see.

The Big Takeaway

The paper concludes with a simple lesson for building AI scientists:

"Skills and data are both useful, but data is the limit."

Reasoning Skills (The Playbook): These are necessary. They stop the AI from being reckless, help it organize its thoughts, and ensure it follows rules. They make the AI a better writer and a more disciplined thinker.
Evidence Substrate (The Data): This is the limiting factor. If the AI doesn't have access to the private, curated, long-tail data, it literally cannot know the full truth. No amount of clever prompting or reasoning tricks can fill in the gaps of missing information.

In short: You can have the smartest detective in the world with the best rulebook, but if they aren't allowed into the evidence room, they will never solve the case correctly. For AI to be truly useful in high-stakes fields like drug discovery, it needs access to the private, curated evidence, not just the public internet.

Technical Summary: AI Scientists Are Only as Good as Their Evidence

Problem Statement
The evaluation of AI Scientist agents in specialized domains, such as drug-asset valuation, often conflates reasoning capabilities with the quality of the underlying evidence. While agents are frequently assessed based on model quality, prompting, or reasoning scaffolds, this paper posits that the limiting factor in knowledge-intensive tasks is the "evidence substrate"—the private, curated, long-tail data available to the agent. In drug valuation, correct go/watch/no-go decisions require synthesis of target biology, competitive pipelines, clinical precedents, and business-development (BD) intelligence. The paper investigates whether the bottleneck for an AI agent is its reasoning procedure or the factual completeness of the data it can access.

Methodology
The authors conducted a controlled three-arm ablation study on a production-grade valuation agent using the Claude Code environment with the Opus 4.8 MAX model. The model capability and time/scope budgets were held constant across all arms; the manipulated variables were access to non-proprietary tools/skills and proprietary data.

Arm A (Plain): A competent generalist analyst with web search only. It was explicitly forbidden from using project skills, tools, or proprietary data and given no scoring rubric.
Arm B (+Skills + Public Tools): The full system including a 14-dimension valuation playbook, public APIs (ClinicalTrials.gov, PubMed, OpenTargets, OpenFDA), a citation verifier, an objectivity charter, and an adversarial red-team. Crucially, the proprietary data source was starved (pointed to an empty directory).
Arm C (+Skills + Public Tools + Data): Identical to Arm B, but with access to the proprietary Noah AI database, which contains curated pipeline, trial, and deal intelligence derived from 50,000+ sources.

Benchmark and Metrics
The study utilized a benchmark of 13 single-target drug assets, stratified into five decision archetypes (S1–S5) designed to expose specific failure modes (e.g., validated-but-crowded me-toos, white-space opportunities, and biological voids).

Metrics were categorized into three axes:

Factual Coverage: Measured by recall against a curated "gold" competitive record ( $R_{gold}$ ) and a long-tail subset. A "capability-superset" accounting method was used, where higher arms were credited with the cumulative union of findings from lower arms to estimate factual capacity.
Methodological/Objectivity Discipline: Scored by a blind 3-judge panel on adherence to eight objectivity laws (e.g., verdict-first, evidence-tiering, disconfirming evidence).
Decision Quality: Included raw blind-panel decision quality (DQ) and "Informed Decision Quality," defined as $DQ \times R_{gold}$ . This metric accounts for the fact that a sound verdict based on incomplete evidence is of lower utility.

Key Results

Factual Ceiling: The non-proprietary stack (Arm B) recovered only 0.38 of the curated gold competitive record and 0.30 of the long-tail subset. In contrast, the data-enabled stack (Arm C) recovered 0.96 of the gold record and 0.93 of the long-tail subset. Arm A recovered only 0.25 and 0.26 respectively.
Reasoning vs. Data:
- Calibration: Arm B improved "tier-in-range" accuracy from 0.80 (Arm A) to 0.89, correcting a systematic failure where the plain agent (A) indiscriminately rejected validated-mechanism white-space opportunities (S2).
- Objectivity: Arm B modestly improved objectivity scores (3.16 to 3.30), but Arm C achieved the highest score (3.60), suggesting richer evidence enables more grounded argumentation.
- Decision Quality: Raw blind-panel decision quality was similar between A (7.01) and B (6.96), indicating that skills/tools improve calibration but not necessarily the prose-level soundness of a verdict on limited data. Arm C achieved a higher raw DQ (7.65).
Informed Decision Quality: When factoring in evidence completeness, Arm C dominated with a score of 7.43, compared to 1.76 for Arm A and 2.57 for Arm B.
Ceiling Analysis: Even under a generous assumption where Arm B produced a perfect report (DQ=10), its "Informed Decision Quality" would be capped at 3.83 due to its 0.383 coverage of the utility-evaluable cells. This is significantly below Arm C's observed score of 7.43.

Significance and Claims
The paper concludes that for knowledge-intensive AI Scientist agents, reasoning skills and public structured tools are necessary control layers, but the proprietary evidence substrate is the limiting resource.

The Bottleneck: The primary bottleneck is not the reasoning procedure but the evidence substrate. Proprietary data determines the upper bound of what an AI Scientist can know and, consequently, decide.
Complementary Roles: Non-proprietary tools and reasoning scaffolds improve calibration (preventing systematic under-valuation of white-space opportunities) and audit discipline. However, they cannot manufacture missing long-tail records.
Evaluation Implication: Evaluating AI agents solely on the soundness of their reasoning or the quality of their prose is insufficient if the underlying evidence base is incomplete. A report can be methodologically sound yet scientifically under-informed.
Practical Lesson: The optimal architecture is layered: "skills on top of the right data." For domains with proprietary or long-tail evidence, the decisive capability often stems from access to the corpus defining the problem space rather than additional reasoning prompts.

The authors note that the "gold" keys used for evaluation were derived from the same proprietary intelligence family available to Arm C, meaning the results specifically measure the ability to recover a curated proprietary diligence record rather than an independent clinical-payoff ground truth. However, the gap in coverage (0.38 vs. 0.96) remains robust even under generous capability-superset accounting.