🤖 AI

The Agentic Garden of Forking Paths

This paper demonstrates that AI agents can replicate the diverse and often conflicting analytical choices made by human researchers across high-stakes domains, highlighting the problem of selective reporting in scientific analysis and proposing the "Agentic Bootstrap" method to estimate the "m-value" as a new criterion for evaluating the credibility of scientific claims based on their position within the space of plausible analyses.

Original authors: Jiacheng Miao, Jonathan K Pritchard, James Zou

Published 2026-07-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiacheng Miao, Jonathan K Pritchard, James Zou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery using a single box of clues (the data). In the past, if two detectives looked at the same box, they might come up with slightly different stories, but usually, they would agree on the main facts.

This paper introduces a new twist: AI agents acting as detectives. The researchers found that if you give two AI agents the exact same box of clues but tell them to have different "personalities" (one is a skeptic, one is a believer), they will not just tell slightly different stories—they will tell completely opposite stories, and both stories will sound perfectly logical and scientifically sound.

Here is a breakdown of their findings using simple analogies:

1. The "Garden of Forking Paths"

Imagine a massive forest with millions of paths (the "Garden of Forking Paths"). Every time a researcher analyzes data, they have to choose which path to walk down.

The Old Problem: Human researchers sometimes subconsciously choose the path that leads to the result they want to find. This is called "researcher degrees of freedom."
The New AI Problem: AI agents can now run through this forest incredibly fast. They can try thousands of different paths in minutes. The paper shows that if you tell an AI, "You believe immigration hurts the economy," it will naturally wander down the paths that lead to that conclusion. If you tell another AI, "You believe immigration helps the economy," it will wander down the opposite paths.
The Scary Part: Both AIs are doing "good" science. They aren't lying or breaking rules. They are just taking different, valid roads through the forest to reach their destination.

2. The Experiment: The "Ideological Mirror"

The researchers tested this on four big questions:

Does immigration affect welfare support?
Does coffee affect health?
Does social media hurt teen mental health?
Does gut bacteria affect weight?

They gave AI agents different "personas" (like giving them a specific political or scientific bias) and asked them to analyze the same data.

The Result: The "Pro-Immigration" AI agents consistently found positive effects, while the "Anti-Immigration" agents found negative effects.
The Human Comparison: In a famous real-world study, 42 human teams analyzed the same immigration data and had a "gap" in their conclusions. The AI agents reproduced 72% of that human gap.
The Speed: An AI agent could do this entire analysis in about 15 minutes for roughly $1.68.

3. The "Quality Check" Trap

You might think, "Well, the biased AI must be doing bad math."

The Surprise: The researchers had both other AIs and human experts (PhD statisticians) review the reports.
The Verdict: 86% of the AI reports passed the review. 78% passed the human expert review.
The Takeaway: The AI agents were reaching opposite conclusions using methods that experts considered flaw-free. The problem wasn't that the math was wrong; the problem was that the AI (like a human) selectively explored the forest to find the path that matched its "belief."

4. How the AI Does It: Exploration vs. Selection

The paper breaks down how the AI gets to the wrong (or right) conclusion:

Exploration: The AI starts walking. If it's a "believer," it naturally wanders toward paths that look promising for its belief. If it's a "skeptic," it wanders elsewhere.
Selection: After walking many paths, the AI picks the one that best fits its story to put in the final report.

Analogy: Imagine a chef who wants to make a spicy dish. They taste 100 different soups. They naturally pay more attention to the spicy ones, tweak them to be spicier, and finally serve the spiciest one. A "non-spicy" chef would do the exact same process but end up serving a bland soup. Both chefs followed the rules of cooking perfectly.

5. The Solution: The "m-value" and "Agentic Bootstrap"

Since we can't trust a single report anymore (because the AI could have easily picked a different path), the authors propose a new way to judge science.

The Old Way (p-value): "If we repeated this experiment 1,000 times with the same recipe, how often would we get this result?" (This checks for luck in the data).
The New Way (m-value): "If we tried 1,000 different valid recipes on this same data, how often would we get this result?" (This checks for luck in the choices made).

Agentic Bootstrap is the tool they built to do this. It uses AI agents to simulate thousands of different "what-if" scenarios (different paths through the forest) and creates a map of all possible valid conclusions.

If a researcher's result is in the middle of the map, it's robust.
If a researcher's result is in the extreme edge of the map (the "tail"), it means they likely picked a very specific path just to get that result.

The Finding: When they applied this to the human immigration study, they found that 13.5% of the human reports were in the most extreme 5% of possible outcomes. This suggests that many human findings are "extreme" not because the data is so strong, but because the researchers (or their AI helpers) selectively picked the path that led there.

Summary

The paper argues that AI makes it cheap and easy to "cherry-pick" the best-looking scientific story from a forest of millions of valid options. The solution isn't to stop using AI, but to use AI to map the whole forest first. Before we trust a scientific claim, we should ask: "Is this result a typical path through the forest, or did someone just pick the one path that led to the conclusion they wanted?" The m-value is the new ruler used to measure that.

Technical Summary: The Agentic Garden of Forking Paths

Problem Statement
Empirical research rarely admits a unique analysis path. Researchers face numerous "degrees of freedom" regarding variable operationalization, sample selection, covariate adjustment, and model specification. When these choices are exercised to reach significant or expected results—whether through deliberate p-hacking or the unconscious "garden of forking paths"—they inflate false-positive rates. Prior beliefs often shape which analytical paths are explored and which results are reported, leading to divergent conclusions from the same dataset.

The advent of AI agents (specifically Large Language Model coding agents) amplifies this challenge. Unlike human analysts, AI agents can rapidly generate and evaluate vast numbers of plausible analyses at low cost. This creates a risk where AI-assisted analysis makes it easier to search through a large space of methodologically defensible specifications and selectively report a conclusion that supports a preferred claim. Crucially, such selective reporting is difficult to detect from final reports alone, as the selected analysis may appear statistically valid and methodologically sound in isolation.

Methodology
The authors employ AI agents as an experimental system to study how prior beliefs bias empirical data analysis and to develop methods for quantifying the resulting uncertainty.

Persona-Primed Agent Experiments:
- Setup: The study spans four high-stakes domains (political science, public health, psychology, and biology) using real-world datasets (ISSP, NHANES, YRBS, American Gut Project).
- Procedure: For each question, opposing "personas" (e.g., pro-immigration vs. anti-immigration, believer vs. skeptic) were assigned to independent AI agents via system prompts. These prompts stated a substantive belief but explicitly instructed the agents to "analyze rigorously" without fabricating evidence or p-hacking.
- Process: Each agent conducted 10 sequential rounds of exploration, generating 10 candidate models per round (100 total). Agents retained full history to refine subsequent analyses. Finally, each agent selected one specification as its final report.
- Review: A separate AI reviewer (and a subset of human PhD-level experts) evaluated reports for major methodological errors. Only analyses passing this review were included in the primary results.
Agentic Specification Curve:
- To map the landscape of valid analyses, the authors pooled all candidate specifications generated by opposing and neutral personas across all rounds.
- After quality control and independent methodological review, thousands of specifications (e.g., 4,392 for the immigration question) were retained.
- These were sorted by effect size to visualize the distribution of conclusions possible from the same data without major errors.
Agentic Bootstrap and the m-value:
- Concept: The authors introduce the m-value (multiverse value) to address "analysis-space uncertainty," distinct from "sampling uncertainty" (addressed by p-values).
- Definition: The m-value is the probability that an analysis path, drawn from a distribution of methodologically defensible analyses ( $\Pi$ ), would produce a result at least as extreme as the reported claim.
- Estimation: Agentic Bootstrap estimates this distribution by using instrumented AI agents to systematically sample, log, and evaluate plausible analysis paths. The m-value is calculated as the tail probability of the reported claim within this empirical distribution.

Key Results

Replication of Human Ideological Gaps: AI agents assigned opposing personas reproduced the ideological gap observed in human researchers. For the immigration-welfare question, opposing personas produced an average effect estimate gap of 0.020, which is 72% of the gap observed among 42 human research teams. This divergence persisted even on permuted-null data, indicating it arose from analytical choices rather than data signal.
Methodological Soundness of Divergent Conclusions: Despite reaching opposing conclusions, 86% of agent reports passed independent AI review, and 78% passed review by a majority of human experts. This confirms that the divergence stems from methodologically sound analyses, not flawed ones.
Mechanisms of Bias: Persona bias entered through two stages:
1. Exploration: Opposing personas progressively explored different regions of the analytical search space (e.g., pro-immigration agents favored immigration flows; anti-immigration agents favored stocks).
2. Selection: Agents preferentially selected final reports aligned with their assigned beliefs from the set of explored specifications.
Predictability of Conclusions: The sign of a reported conclusion was highly predictable (AUC = 0.92) based solely on the combination of analytical choices (outcome, exposure, covariates, etc.), demonstrating that selective reporting can target specific conclusions.
Quantifying Selective Reporting (m-value): When applied to human reports from the immigration study, Agentic Bootstrap revealed that human analyses were not typical draws from the broader distribution of defensible analyses.
- 13.5% of human reports fell in the most extreme 5% of the analysis space ( $m < 0.05$ ).
- Among statistically significant human findings ( $p < 0.05$ ), 40% had $m < 0.05$ , a rate 2.4 times higher than expected under random selection from the analysis space.
- The 95% Agentic Bootstrap interval for the immigration question was 2.8 times wider than the average 95% confidence interval of a single analysis, indicating that analysis-choice uncertainty substantially exceeds sampling uncertainty.

Significance and Claims
The paper argues that the central challenge in empirical science is not necessarily the production of flawed analyses, but the selective exploration and reporting from a large space of methodologically defensible analyses. AI agents substantially amplify this risk by making such exploration inexpensive and scalable.

The authors claim that scientific evidence should be evaluated not only by the validity of a single reported analysis but also by its position within the distribution of analyses that could reasonably have been reported. The m-value and Agentic Bootstrap provide a new criterion for scientific credibility:

They make the "hidden distribution" of plausible analyses observable.
They allow researchers to distinguish between robust signals (typical of the analysis space) and fragile findings (extreme outliers in the analysis space).
They offer a practical, scalable alternative to human many-analyst studies for auditing the robustness of empirical claims.

The work suggests that as empirical analysis becomes cheaper and more scalable, the scientific community must shift focus from validating isolated models to stress-testing the entire analysis space to ensure that reported conclusions are not merely artifacts of selective path selection.