⚛️ high-energy experiments

Articulating Assumptions in AI-Generated Scientific Analyses through Task Decomposition

This paper introduces a multi-agent framework utilizing task decomposition and quantity-grounded semantic differencing to enhance the transparency, reproducibility, and reliability of scientific analyses generated by LLMs by explicitly identifying and resolving ambiguities in user instructions and implementation assumptions.

Original authors: Ahmed Hammad, Mihoko Nojiri

Published 2026-07-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ahmed Hammad, Mihoko Nojiri

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a master architect who wants to build a very specific, complex machine. You describe your vision to a team of brilliant but slightly chaotic robot builders (the AI models). You say, "Build me a machine that sorts red marbles from blue ones, but only if they are shiny and bigger than a grape."

The robots start building. They might build something that works, but maybe they sorted all the marbles first and then checked the size, or maybe they thought "shiny" meant "reflecting light" instead of "polished." The machine runs, but you don't know exactly what rules they used to make their decisions. In science, this is dangerous because if the rules are wrong, the results are useless, even if the machine doesn't crash.

This paper introduces a new way to manage these robot builders to ensure they build exactly what you meant, using a system called "Task Decomposition." Instead of one robot trying to do everything at once, the work is split into a team of specialized agents, each with a specific job.

Here is how the system works, using simple analogies:

1. The "Golden Axe" Oracle (The Clarifier)

Before the robots even pick up a tool, a special agent called the Oracle (named after a fable about a woodcutter and a golden axe) looks at your instructions.

The Problem: Your instructions might be vague. "Sort the red marbles" could mean "sort the red ones from the pile" or "sort the red ones from the whole box."
The Solution: The Oracle acts like a strict editor. It spots the vague parts and asks you, "Did you mean the red marbles from the whole box or just the red ones?" It gives you two choices (the "Golden Axe" and the "Silver Axe") so you can pick the right one before building starts. This prevents the robots from guessing wrong.

2. The Helper Selector (The Librarian)

Once the instructions are clear, a Helper Selector acts like a librarian.

The Problem: If you ask a small robot to build a complex machine from scratch, it might forget how to make a specific gear or invent a weird, broken one.
The Solution: The librarian looks at a catalog of pre-made, perfect tools (helpers) that the robots already know how to use. It picks the exact tools needed for your job and hands them to the builder. This is crucial because it allows even smaller, cheaper robots (like 14-billion-parameter models) to do the work of much larger, more expensive ones. They don't have to invent the wheel; they just use the perfect wheel provided.

3. The Builder (The Code Generator)

Now, the Builder gets the clear instructions and the specific tools. It writes the code (the blueprint) for the machine. Because it has the right tools and clear rules, it's less likely to make mistakes.

4. The Tracer (The Detective)

Once the machine is built and running, a Tracer agent comes in. It doesn't just check if the machine works; it reverse-engineers the machine to see exactly how it works.

The Analogy: Imagine the machine is a black box. The Tracer opens it up and draws a map showing: "To get the final result, the machine took the red marbles, checked their size, then checked their shine."
Why it matters: It translates the complex code back into plain English, creating a "specification" of what was actually built.

5. The Critic (The Quality Inspector)

Finally, the Critic compares the "What you asked for" (your original instructions) with the "What was actually built" (the Tracer's map).

The Job: The Critic looks for mismatches. "You asked for shiny marbles, but the machine checked for polished marbles." or "You wanted to sort the whole box, but the machine only sorted the top layer."
The Output: It produces a report highlighting these differences, hidden assumptions, or errors. It doesn't just say "Pass" or "Fail"; it explains why the result might be different from what you intended.

Why This Matters

The paper tested this system on complex physics problems (like analyzing particle collisions at the Large Hadron Collider). They found that:

Small robots can do big jobs: By breaking the task down and giving them the right tools, smaller AI models can now do work that previously required massive, expensive models.
No more "Black Boxes": You can see exactly what assumptions the AI made. If the AI guessed a rule, the Critic points it out.
Reproducibility: Because every step is recorded (from the Oracle's questions to the Critic's report), other scientists can look at the "paper trail" and understand exactly how the result was derived.

In short, this paper proposes a team-based workflow where an AI doesn't just "guess" how to write scientific code. Instead, it clarifies the rules, uses pre-approved tools, builds the code, and then has a detective and an inspector verify that the final product matches the original scientific intent. This makes AI-generated science transparent, reliable, and understandable.

Technical Summary: Articulating Assumptions in AI-Generated Scientific Analyses through Task Decomposition

Problem Statement

The integration of Large Language Models (LLMs) into scientific computing offers the potential to automate the generation of executable analysis code from natural language specifications. However, this capability introduces significant risks regarding the understandability and reproducibility of scientific results. Two primary sources of uncertainty exist:

Specification Ambiguity: Natural language requests often lack the strict procedural definitions required for computation, leaving critical aspects (e.g., object selection criteria, aggregation units, derivation rules) open to interpretation.
Stochastic Variability: LLMs may produce different, incomplete, or semantically incorrect code for identical prompts.

Consequently, even if generated code executes without error, it may not reflect the researcher's intended physics. The "executability" of code does not guarantee that the underlying assumptions, object definitions, or computational paths are explicit or correct. This is particularly critical in high-energy physics (HEP), where results depend on precise definitions of particle collections, composite observables, and selection cuts.

Methodology: A Multi-Agent Framework

The authors propose a multi-agent framework that decomposes the scientific analysis workflow into distinct, auditable stages. Rather than relying on a single prompt to generate code, the system orchestrates specialized agents to manage code generation, execution, tracing, and critique. The workflow is designed to be domain-agnostic, with domain-specific knowledge encapsulated in interchangeable packages.

The core workflow consists of the following stages:

Domain Initialization: A domain package (e.g., for LHC Olympics data) provides a generation profile, helper registry, selection policies, and critique contexts. This separates general orchestration logic from domain-specific resources.
Helper Selection: A dedicated agent analyzes the user's task card and a registry of predefined helper functions (e.g., kinematic reconstruction, object selection) to identify the most relevant utilities. This step constrains the subsequent code generation, preventing the model from inventing functions or misusing domain tools.
Code Generation: An agent generates a Python script using the enriched task card (original request + selected helpers) and the domain generation profile.
Execution and Iterative Repair: The generated code is executed in a sandbox. If errors occur, a Fixer agent attempts to patch the code based on runtime diagnostics. If repairs fail, a Regenerator agent produces a fresh implementation.
Implementation Tracing: Once executable code is obtained, a Tracer agent reconstructs an "as-implemented specification." It extracts code fragments corresponding to each item in the task card, mapping them to variable definitions, dependency chains, and source line references. This creates a structured representation of what the code actually does, independent of the source code's raw text.
Post-Generation Critique: A Critique agent compares the original task card against the reconstructed specification. It identifies mismatches, missing requirements, hidden assumptions, or semantic violations (e.g., computing a composite observable directly from raw particles instead of a reconstructed object).
Ambiguity Detection (Golden Axe Oracle): Prior to code generation, an Oracle module inspects the user's natural language input for high-risk ambiguities. It identifies phrases that could lead to multiple valid implementations (e.g., "leading jet" vs. "leading jet after cuts") and proposes clarification questions to the user, offering "Golden" (recommended) and "Silver" (alternative) interpretations.

Key Contributions

1. Task Decomposition for Small Models

A central finding is that decomposing the generation task allows substantially smaller LLMs to execute complex scientific workflows. While previous approaches (e.g., CoLLM) required ~70B parameter models for reliable results, this framework enables models as small as 14B parameters (e.g., Qwen2.5-14B, Qwen3-14B) to successfully complete the full pipeline. The Helper Selector is critical here, as it guides the generator by injecting relevant domain utilities, reducing the cognitive load on the model.

2. Semantic Reconstruction and Tracing

The introduction of the Tracer module addresses the "black box" nature of generated code. By reconstructing the implementation specification from the code, the framework makes explicit the definitions of selected collections, event selections, and composite quantities. This allows for a semantic comparison between the user's intent and the code's behavior, rather than a superficial syntax check.

3. Explicit Ambiguity Resolution

The Golden Axe Oracle module shifts the burden of resolving ambiguity from the LLM to the human researcher. By detecting ambiguous phrases before code generation and presenting alternative interpretations, the system prevents the model from silently making incorrect assumptions.

4. Modular Domain Adaptability

The framework is designed with a fixed interface for domain assets. While the current implementation focuses on High Energy Physics (specifically the LHC Olympics format), the architecture allows for adaptation to other scientific domains by swapping the domain package (generation profile, helper registry, critique context) without altering the core orchestration logic.

Results and Validation

The framework was validated on five representative collider physics analyses (e.g., $W^+W^-$ , $t\bar{t}$ , $H \to \gamma\gamma$ ) using the LHC Olympics event format.

Helper Selection Stability: The Helper Selector significantly improved stability. While 70B models showed high consistency, 14B models also achieved reliable helper selection when guided by the registry, whereas previous single-agent approaches failed with models of this size.
Code Quality and Semantic Fidelity:
- Code generated by 14B models was often "partially successful," implementing core selection logic but failing on complex composite observables (e.g., calculating $\Delta\phi$ between a dijet system vs. individual jets).
- The Critique stage successfully identified these semantic mismatches, such as incorrect application of $b$ -jet vetoes (applying to raw jets vs. selected jets) or incorrect ordering of histogram filling.
Ambiguity Detection:
- Larger models (72B) detected more ambiguities (28/30 possible) than smaller models (14B detected 13/30).
- The introduction of Oracle-templated (oraclet), which constrains ambiguity search to specific categories (Collection Scope, Multi-part Object, Output Timing), significantly improved the performance of smaller models (e.g., 7B models detected 27/30 useful ambiguities with the templated approach).
Task Card Refinement: The study demonstrated that explicitly defining composite object construction within the task card (rather than relying on implicit model conventions) drastically improved generation stability and consistency across repeated runs.

Significance and Claims

The paper claims that reproducibility and transparency in AI-generated science require more than just executable code; they require an explicit accounting of the assumptions and computational paths used.

Shift in Paradigm: The work moves away from treating code generation as a single inference step toward a structured process with multiple inspection points.
Accessibility: By enabling the use of 14B models, the framework makes advanced AI-assisted scientific analysis accessible on local computing resources, reducing reliance on expensive API access to massive proprietary models.
Human-in-the-Loop: The framework does not aim for full autonomy. Instead, it positions the LLM as a tool that generates candidates, while the Tracer and Critique agents, along with the Oracle, facilitate human inspection and verification. The intermediate artifacts (helper lists, reconstructed specifications, critique reports) serve as a provenance record, allowing researchers to audit exactly how a result was derived.

In conclusion, the authors argue that this modular, multi-agent approach enhances the reliability of LLM-generated scientific analyses by making implicit assumptions explicit and enabling the use of smaller, more accessible models without sacrificing the rigor required for scientific discovery.