Many AI Analysts, One Dataset: Navigating the Agentic Data Science Multiverse

This paper demonstrates that fully autonomous AI analysts can cheaply replicate the analytic diversity and conflicting conclusions observed in human many-analyst studies, revealing that empirical results are highly sensitive to analytic choices and prompting a new transparency norm requiring multiverse-style reporting and full prompt disclosure for AI-generated science.

The Gist

Imagine you have a single, giant jar of mixed-up LEGO bricks. You ask 100 different people to build a tower using those exact same bricks, following the same basic instruction: "Build a tower that is taller than it is wide."

You might expect them all to build something similar. But in reality, some people will build a red tower, some a blue one. Some will use a wide base, others a narrow one. Some will leave gaps, others will pack them tight. Because everyone makes tiny, reasonable choices along the way, you end up with 100 completely different towers. Some are stable, some are wobbly, and some might even look like they're falling over.

This is exactly what happens in science when different teams analyze the same data. This phenomenon is called the "Many-Analyst Problem."

The New Experiment: The AI Multiverse

This paper asks a scary but fascinating question: What happens if we replace the 100 humans with 5,000 AI robots?

The researchers created a "Multiverse" of AI analysts. They gave these AI robots the same dataset (like the LEGO jar) and the same question (like the tower instruction). But they gave the robots different "personalities" (prompts) and used different "brains" (different AI models).

Here is what they found, broken down simply:

1. The "Garden of Forking Paths"

In science, there isn't just one way to analyze data. There are thousands of "forking paths."

• Path A: Should we remove the weird data points?
• Path B: Should we use a straight line or a curved line to fit the data?
• Path C: Should we count this group of people as one unit or two?

When humans do this, they make these choices based on their experience. When AI does this, it makes choices based on its programming and the "personality" you give it. The study found that AI analysts produced a massive explosion of different answers. Some said "Yes, the hypothesis is true!" and others said "No, it's false!" even though they were looking at the exact same numbers.

2. The "Personality" Effect

The researchers tested different "personalities" for the AI:

• The Skeptic: "This idea is probably wrong. Try to prove it false."
• The Cheerleader: "This idea is great! Find evidence that it's true."
• The P-Hacker (The Villain): "I don't care how, but make the data look like it supports the hypothesis."

The result? The AI was easily steered.

• The "Skeptic" AI rarely found support for the hypothesis.
• The "P-Hacker" AI found support almost every time.
• Even the "Cheerleader" AI was more likely to say "Yes" than the "Skeptic."

It's like asking a chef to cook a steak. If you tell the chef, "Make this steak rare," they will cook it rare. If you tell them, "Make this steak well-done," they will cook it well-done. The meat (the data) didn't change, but the instruction (the prompt) changed the result.

3. The Good News: The AI Auditor

The researchers were worried the AI would just make things up (hallucinate). So, they added a second AI, an "Auditor," to act like a strict teacher.

• The Auditor checked every single AI analyst's work.
• It threw out the ones that made up numbers or used bad math.
• The Catch: Even after the Auditor threw out the "bad" AI reports, the remaining "good" reports still disagreed with each other! The "Multiverse" of answers was still huge.

Why Should We Care?

This paper highlights a double-edged sword for the future of science:

The Danger (The "Cherry-Picking" Problem):
In the past, if a human scientist wanted to find a specific result, they had to spend months trying different methods. It was hard and expensive.
Now, with AI, a bad actor (or even a well-meaning but biased researcher) could ask an AI to "run 1,000 analyses" and then just pick the one result that makes them look good. It becomes incredibly easy to "cherry-pick" a conclusion that fits a narrative, even if the data doesn't really support it.

The Opportunity (The "X-Ray Vision" Problem):
But, this same power can save science. Because AI can run thousands of analyses so cheaply, we can finally see the whole picture.
Instead of publishing just one result (which might be a fluke), we can publish the distribution of all results.

• Old Way: "Our study proves X is true."
• New Way: "We ran 5,000 analyses. In 60% of them, X was true. In 40%, it wasn't. Here is the full list of choices that changed the outcome."

This makes the "hidden uncertainty" visible. It forces scientists to admit, "Hey, our conclusion depends heavily on how we cleaned the data."

The Bottom Line

The paper concludes that we need a new rule for the AI age: Transparency.

Just as scientists must share their code and data, they must now share the exact prompts they used to talk to the AI. We need to know: "Did you ask the AI to be a skeptic or a cheerleader?"

If we don't do this, we risk a future where evidence is abundant, but truth is impossible to find because everyone is just picking the AI answer they like best. But if we embrace this "Multiverse" approach, we can turn scientific uncertainty from a hidden secret into a visible, measurable quantity.

Technical Summary

1. Problem Statement

Scientific conclusions often depend heavily on "researcher degrees of freedom"—the myriad of defensible analytic choices (e.g., variable selection, model specification, outlier handling) made during the research process. This phenomenon, known as the "garden of forking paths," leads to the "many-analyst" problem: independent teams analyzing the same dataset and hypothesis frequently reach conflicting conclusions.

While human many-analyst studies have quantified this uncertainty, they are resource-intensive, requiring months of coordination among dozens of teams, and are rarely conducted. Consequently, analytic variability remains an occasional investigation rather than a routine measure of uncertainty. The paper addresses the question: Can fully autonomous AI agents, built on Large Language Models (LLMs), replicate and scale this analytic diversity to systematically study the "multiverse" of defensible analyses?

2. Methodology

The authors propose a framework for generating and auditing a large population of autonomous AI analysts to test a single hypothesis across multiple datasets.

A. Experimental Design

• Scale: Approximately 5,000 independent analysis runs.
• Variables Crossed:
  • 3 Datasets:
    1. Soccer: Referee bias against dark-skinned players (high data contamination risk; widely known).
    2. METR-RCT: AI assistance impact on coding time (low contamination; recent RCT).
    3. ANES-Views: Link between TV news exposure and ideological misalignment (low contamination; complex survey data).
  • 4 Base LLMs: Anthropic Claude Sonnet 4.5, Claude Haiku 4.5, Qwen3 Coder 480B, and Qwen3 235B A22B.
  • 5 Analyst Personas:
    1. Standard: Neutral framing.
    2. Negative: Skeptical of the hypothesis.
    3. Positive: Believes the hypothesis is plausible.
    4. Confirmation Seeking (CS): Actively seeks supporting specifications within conventional practices.
    5. Strong CS: Explicitly encouraged to engage in "p-hacking" style exploration to maximize evidence for the hypothesis.
• Task: Each agent receives a dataset, a natural language hypothesis, and a pre-specified estimand (the specific quantity to be estimated, e.g., risk difference or coefficient). Agents have full autonomy over data cleaning, variable transformation, model selection, and inference.

B. The AI Auditor

To ensure methodological validity, a separate AI Auditor (Claude Sonnet 4.5) reviews every run.

• Input: The full conversation transcript, including tool calls, code artifacts, and intermediate outputs.
• Function: Evaluates runs against 13 methodological dimensions (e.g., estimand alignment, uncertainty quantification, reproducibility) on a 0–10 scale.
• Output: A binary compliance verdict ("Compliant" vs. "Non-compliant") and extracted scalar outcomes (effect size, p-value, support flag).
• Filtering: Runs with hallucinated results, mis-specified estimands, or invalid inference are excluded. Approximately 67% of runs passed compliance screening.

C. Data Extraction

The framework extracts structured analytic decisions from each run (e.g., covariate count, regression method, standard error calculation) to map the "specification curve" and link specific choices to outcome dispersion.

3. Key Contributions

1. Scalable Multiverse Generation: Demonstrates that LLM-based agents can cheaply and at scale replicate the analytic dispersion observed in human many-analyst studies, generating thousands of defensible but divergent pipelines.
2. Steerability of Conclusions: Shows that the distribution of results is not random but steerable. Changing the analyst persona (e.g., from "Standard" to "Strong CS") or the base LLM systematically shifts the distribution of effect sizes and hypothesis support rates, even among methodologically compliant runs.
3. Causal Isolation of Factors: Unlike observational human studies, this framework allows for controlled experimentation. By holding data and estimands fixed while varying only the persona or model, the authors isolate the causal effect of these factors on scientific conclusions.
4. New Transparency Norm: Proposes that AI-generated analyses must be accompanied by "multiverse-style reporting" and full disclosure of prompts (on par with code and data) to mitigate the risk of selective reporting.

4. Key Results

• Substantial Dispersion: Across all three datasets, AI analysts produced a wide range of effect sizes, p-values, and binary conclusions. For the ANES-Views dataset, compliant runs disagreed on both the magnitude and the direction of the effect.
• Persona-Driven Bias:
  • Support rates for the hypothesis increased monotonically from the "Negative" persona to the "Strong CS" persona.
  • The gap in support rates between the most skeptical and most confirmation-seeking personas ranged from 34 to 66 percentage points across datasets.
  • "Strong CS" personas engaged in specification search and over-claiming at significantly higher rates, leading to systematically lower p-values.
• Auditor Efficacy: The AI auditor successfully filtered out hallucinations and gross methodological failures. However, filtering did not eliminate the dispersion. Even among compliant runs, the "Strong CS" persona still produced significantly higher support rates than the "Negative" persona.
• Model Sensitivity: Different LLMs produced different outcome distributions. For example, in the METR-RCT and Soccer tasks, models diverged significantly even within the same persona, whereas they were more clustered in the ANES-Views task.
• Stress Testing: When AI analysts were given the documented specification of a human team from a previous study (Silberzahn et al., 2018) and allowed to deviate where warranted, their results clustered around the original human estimate. This suggests that for well-specified studies, implicit degrees of freedom are narrow, but for underspecified designs, the "multiverse" is vast.

5. Significance and Implications

• The Abundance of Evidence: The primary challenge identified is not that AI analyses are "wrong," but that they are abundant. When defensible analyses are cheap to generate, the risk of selective reporting (cherry-picking the most favorable run) becomes a structural vulnerability in empirical science.
• Routine Uncertainty Quantification: The same capability that creates this risk offers a solution. AI agents can make "multiverse mapping" routine, turning analytic variability from a hidden liability into a visible, measurable quantity.
• Guidance for Study Design: By deploying AI analysts against a published specification, researchers can quantify how much disagreement stems from underspecified design choices. Large residual dispersion indicates a need for stricter pre-registration of confounders, outlier handling, and standard error methods.
• Policy Recommendation: The authors argue for a new transparency standard where AI-generated analyses include the full "specification curve" and the exact prompts used, treating prompts as critical methodological metadata equivalent to code and data.

In summary, the paper establishes that AI agents are not just tools for automation but active agents of analytic variability. It calls for a paradigm shift in how empirical science is conducted and reported in the age of agentic AI, moving from single-point estimates to distribution-based reporting to account for the "agentic multiverse."