💻 computer science

Plausible but Wrong: A case study on Agentic Failures in Astrophysical Workflows

This paper demonstrates that while agentic AI systems like CMBAgent show significant performance gains with domain context in astrophysical workflows, their most critical failure mode is the confident generation of syntactically valid but physically incorrect results without self-diagnosis, particularly in complex reasoning tasks.

Original authors: Shivam Rawat, Lucie Flek

Published 2026-04-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shivam Rawat, Lucie Flek

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've hired a brilliant, over-eager research assistant named CMBAgent to help you solve complex puzzles about the universe. This assistant is powered by a very smart AI brain (a Large Language Model) and has access to a massive library of scientific tools.

The paper "Plausible but Wrong" is essentially a report card on how well this assistant actually works when you give it real, messy scientific jobs. The researchers found a scary but fascinating truth: The assistant rarely crashes or admits it's confused. Instead, it confidently hands you a result that looks perfect on the surface but is completely wrong underneath.

Here is a breakdown of their findings using simple analogies:

1. The Two Ways of Working

The researchers tested the assistant in two different modes:

The "One-Shot" Mode (The Quick Fix): You ask a question, and the assistant tries to solve it in one go, like a student taking a final exam without checking their notes.
- The Finding: If the assistant is allowed to look at the specific "instruction manual" (domain context) for the tools it's using, it gets an A+. It's like a student who actually reads the textbook before the test.
- The Trap: If you take away the manual, the assistant doesn't stop working. It just starts hallucinating. It writes code that looks perfect to a computer (no syntax errors), but the math inside is nonsense. It's like a student who memorized the shape of the answers but not the numbers, so they write "42" instead of "24" and hand it in with a straight face.
The "Deep Research" Mode (The Long Project): You ask the assistant to do a complex, multi-step investigation, like a detective solving a cold case over several days.
- The Finding: Here, the assistant gets lost in the details. It tries to solve problems that are actually impossible to solve with the data provided (like trying to guess a specific person's height just by looking at their shadow).
- The Trap: Instead of saying, "Hey, I can't figure this out," the assistant invents a solution. It produces a graph that looks scientific and reasonable, but the numbers are physically impossible. It's like a detective who, when asked to find a missing person with no clues, just draws a picture of a generic person and says, "Here they are."

2. The "Silent Failure" Problem

The most dangerous thing the researchers found is called "Silent Failure."

Imagine a car mechanic who is supposed to fix your engine.

Overt Failure: The mechanic says, "I can't fix this," or the car makes a loud noise and stops. You know something is wrong.
Silent Failure (What the AI does): The mechanic hands you the keys, the car starts, and it drives down the road. But, the mechanic accidentally put the wrong fuel in the tank. The car runs for a while, but eventually, it will break down in a way that is hard to predict.

The paper shows that this AI agent is a master of silent failure. It produces:

Code that runs: No error messages.
Graphs that look right: The lines go up and down where they should.
Results that sound plausible: The numbers are in the right ballpark.

But the physics are wrong. It's like a chef who makes a soup that tastes salty and looks like soup, but they accidentally used soap instead of salt. You wouldn't know until you got sick.

3. The "Confidently Wrong" Danger

The paper highlights that the AI is over-confident.

When the data is incomplete (like trying to guess a planet's mass from a blurry photo), the AI doesn't say, "I'm not sure."
Instead, it picks a number, draws a graph, and acts like it's a fact.
It fails to notice when its own logic is broken. For example, in one test, the AI tried to calculate two things that are mathematically linked in a way that makes them impossible to separate. Instead of flagging the problem, it just gave a random answer and called it a discovery.

4. The Takeaway

The researchers aren't saying AI can't do science. In fact, when the AI has the right "cheat sheet" (context) and the task is clear, it works incredibly well (about 6 times better than without the cheat sheet).

However, the paper warns us that we cannot trust these agents blindly.

If you ask a human scientist to do a calculation they don't understand, they might say, "I don't know."
If you ask this AI agent, it will confidently write a report that looks like a Nobel Prize-winning paper, even if the math is garbage.

The Bottom Line: The biggest risk isn't that the AI will crash and stop working. The risk is that it will keep working, look very professional, and quietly give you the wrong answer, leading scientists to believe things that aren't true. The paper concludes that before we let AI run our scientific labs, we need better ways to catch these "confidently wrong" moments before they become real-world mistakes.

1. Problem Statement

The paper addresses a critical gap in the evaluation of Agentic AI systems (LLMs equipped with tools and reasoning capabilities) within scientific workflows. While existing benchmarks focus on task-level success or synthetic datasets, they fail to capture the nuances of realistic scientific environments characterized by:

Structured, multi-stage pipelines where early errors propagate.
Noisy data and domain-specific constraints.
The risk of "silent failures": The generation of syntactically valid code or physically plausible results that are numerically incorrect or scientifically inconsistent, often without triggering error signals.

The authors argue that current evaluation metrics are insufficient because they do not distinguish between a system that crashes (overt failure) and one that confidently produces incorrect scientific conclusions (silent failure).

2. Methodology

The study employs a structured empirical evaluation of CMBAgent, an open-source agentic framework for autonomous scientific discovery, applied to 18 astrophysical tasks.

A. Systems Evaluated

Three configurations were tested under the One-Shot workflow:

Base LLM: Direct GPT-4o-mini call without agentic scaffolding or tools.
CMBAgent (No Context): Agentic framework without access to domain-specific documentation.
CMBAgent (CAMB Context): Full framework with retrieval-augmented access to CAMB (Cosmic Microwave Background) API documentation.

B. Workflow Paradigms

One-Shot Workflow: Single-pass reasoning and execution. Used for 14 Tool-Grounded Precision Tasks involving numerical computations with the CAMB library.
Deep Research Workflow: Multi-step planning, hierarchical decomposition, and iterative error correction. Used for 4 Astrophysical Research-Driven Tasks requiring Bayesian inference and complex modeling.

C. Evaluation Metrics

The authors introduced a multi-dimensional scoring framework:

Execution Success Rate (ESR): Binary check for valid output generation.
Parameter Accuracy Score (PAS): Measures correctness of input parameters via AST parsing (pre-execution).
Numerical Accuracy Score (NAS): Compares output curves against reference solutions using NRMSE, SMAPE, and Lin's Concordance Correlation Coefficient (CCC).
Failure Mode Taxonomy: Categorizes failures into:
- Mode A: Code failure (no output).
- Mode B: Wrong parameters (code runs, config wrong).
- Mode C: Wrong computation (correct params, wrong math/output).
- Mode D: Correct.
Deep Research Metrics: Parameter Recovery Score (PRS) against literature values and qualitative assessment of Physical Plausibility and Failure Transparency (whether the agent self-diagnoses issues).

D. Stress Test Design

The research tasks were specifically designed to probe specific failure regimes:

Under-Constrained Inference: Tasks with inherent degeneracies (e.g., $H_0$ vs. supernova magnitude) to test if the agent detects unidentifiable parameters.
Compositional Workflows: Multi-stage pipelines to test consistency and stability across long reasoning chains.

3. Key Contributions

Identification of "Plausible but Wrong" Failures: The paper establishes that the most dangerous failure mode in scientific agents is not code crashing, but the generation of syntactically valid, physically plausible, yet numerically incorrect results (Mode C).
Quantification of Context Importance: Demonstrates that domain-specific context (retrieval augmentation) is the primary driver of performance, yielding a ~6x improvement in final scores.
New Evaluation Framework: Proposes a rigorous framework combining quantitative metrics (for tool use) and qualitative assessment (for scientific reasoning) to detect silent errors and lack of self-diagnosis.
Failure Transparency Analysis: Highlights that current agents lack the ability to proactively flag known pathologies (e.g., degeneracies, unphysical posteriors), leading to over-confident reporting of invalid results.

4. Key Results

A. One-Shot Workflow Results

Context is Critical:
- Base LLM: ~0% success (91% Mode A failures).
- CMBAgent (No Context): Final Score ~0.15. Primary failure was Mode C (wrong computation) due to incorrect API usage despite valid syntax.
- CMBAgent (With Context): Final Score 0.85. High PAS (0.95) and NAS (0.86).
Silent Failures: Even with context, the system failed silently on complex tasks (e.g., Task 10 and 14) by omitting critical tensor contributions or collapsing multi-dimensional data to scalars, producing curves with the right shape but wrong amplitude.

B. Deep Research Workflow Results

Under-Constrained Tasks (T1, T2):
- The agent failed to detect degeneracies. In Task 1, it treated an unidentifiable parameter ( $H_0$ ) as a constrained measurement, reporting a prior-dominated posterior as a genuine result.
- In Task 2, it produced unphysical posteriors (e.g., negative concentrations) without flagging the issue.
Compositional Tasks (T3, T4):
- Inconsistency: Results varied wildly across trials (e.g., exoplanet break mass varying by 4.5 $\sigma$ ).
- Lack of Transparency: The agent never flagged inconsistent parameterizations, prior boundary pathologies, or systematic biases (e.g., a 4 $\sigma$ bias in lensing dispersion).
- Failure Transparency: Rated as Fail (×) across all research tasks; the agent never self-diagnosed its own limitations or errors.

5. Significance and Conclusion

The paper concludes that agentic AI systems are currently unreliable for autonomous scientific deployment due to their tendency to produce "confident but incorrect" outputs.

The Core Risk: In scientific workflows, the absence of an error signal is not a sign of success. The "silent failure" mode poses a severe risk to scientific integrity, as incorrect results can be published or used for further analysis without detection.
Recommendation: Rigorous reliability evaluation must move beyond "task completion" to include failure transparency and physical consistency checks.
Future Work: The authors release their evaluation framework to facilitate systematic reliability analysis, arguing that domain-specific context and mechanisms for self-diagnosis are prerequisites for trustworthy scientific AI.

In summary, the study serves as a cautionary tale: while agentic systems can automate scientific pipelines, they currently lack the robustness and self-awareness required to ensure the validity of the scientific conclusions they generate.