Agents of Discovery

This paper demonstrates that a team of large language model (LLM) agents can autonomously generate code to solve complex particle physics data analysis tasks, specifically anomaly detection on the LHC Olympics dataset, achieving performance comparable to human state-of-the-art results.

The Gist

Imagine you are a master detective trying to solve a crime in a city so vast and chaotic that the police force is drowning in paperwork. This city is Particle Physics, and the crime is finding a tiny, hidden signal of "new physics" (like a new particle) buried inside a mountain of data generated by the Large Hadron Collider (LHC).

Traditionally, human detectives (physicists) have to manually sift through this mountain, writing complex rules and code to separate the "criminals" (signal) from the "innocent crowd" (background noise). It's slow, exhausting, and prone to human error.

This paper asks a bold question: What if we gave the detective a team of AI assistants who can think, write code, and check each other's work, just like a human research team?

Here is the story of their experiment, explained simply.

The Setup: A Team of Digital Agents

Instead of one giant AI doing everything, the researchers built a "digital agency" with four distinct roles, mimicking a real human lab:

1. The Researcher: The project manager. It looks at the data, comes up with a plan, and delegates tasks. It doesn't write code itself; it asks others to do it.
2. The Coder: The mechanic. When the Researcher says, "Write a program to find the anomaly," the Coder writes the Python code.
3. The Code Reviewer: The quality control inspector. It checks the Coder's work for bugs or mistakes before the code is run.
4. The Logic Reviewer: The critical thinker. It looks at the results and asks, "Does this actually make sense? Did we interpret the graph correctly?"

These agents talk to each other using a "toolbelt." They can write files, run programs, look at images, and even ask for feedback on their progress.

The Test: The "LHC Olympics"

To test this team, the researchers used a famous challenge called the LHC Olympics. Imagine a game where you are given a bag of marbles. Most are blue (background noise), but a few are secretly red (the new physics signal). You don't know which are which. Your job is to:

• Find the red marbles.
• Estimate how many there are.
• Guess their weight (mass).

The twist? The agents had to do this without being told which marbles were red. They had to figure it out on their own, just like a real physicist.

The Contenders: Which AI Model is the Best Detective?

The researchers tested four different "brains" (Large Language Models) from OpenAI to see which one could lead the team best:

• GPT-4o & GPT-4.1: The current standard, very capable.
• o4-mini: A "reasoning" model designed to think step-by-step.
• GPT-5: The newest, most advanced model (released in the paper's future timeline of 2025).

The Results: Who Won the Case?

1. The "Old Guard" Struggled
The older models (like GPT-4o) often got lost. They would write code that crashed, fail to format their reports correctly, or simply give up. They were like detectives who forgot to bring their notepads.

2. The "Reasoning" Models Improved
The models designed to "think" (like o4-mini and GPT-4.1) did better. They started using standard physics tricks, like looking for a "bump" in the data (a sudden spike in numbers that suggests a new particle).

3. GPT-5: The Super Detective
The star of the show was GPT-5. It didn't just follow instructions; it understood the spirit of the investigation.

• It knew the tricks: It used advanced statistical methods (like "Boosted Decision Trees") that human experts use.
• It avoided traps: It realized that looking at the wrong data could trick the algorithm, so it cleverly excluded certain variables to avoid false alarms.
• The Result: In its best runs, GPT-5 found the hidden signal with accuracy that matched the best human experts. It correctly identified the mass of the fake particle and estimated the number of signal events almost perfectly.

The "Feedback Loop" Experiment

In one version of the test, the researchers gave the agents a cheat sheet: after they made a guess, the researchers told them, "You're getting warmer, but your score is X."
This is like a teacher giving a student a hint during a test.

• The Result: When the agents could see their mistakes and try again, GPT-5 became even more impressive. In one run, it essentially "discovered" the hidden particle, correctly identifying its mass and decay mode, even though it started with zero knowledge.

The Catch: It's Expensive

While GPT-5 was amazing, it was also the most expensive. It used a lot of "tokens" (the currency of AI thinking). It was like hiring a team of brilliant, over-qualified consultants who charge by the hour. The older models were cheaper but less reliable.

The Big Picture: What Does This Mean?

This paper is a proof-of-concept. It shows that we are moving from a time where AI just helps humans write code, to a time where AI can run the whole experiment.

• The Good News: We might soon have AI teams that can automate the boring, repetitive parts of physics research (calibrating instruments, checking data, running standard tests). This frees up human scientists to tackle the really hard, creative problems.
• The Challenge: We still need to make these AI teams cheaper, more stable, and easier to trust. We can't just let them run wild; we need to make sure they don't hallucinate a new particle that doesn't exist.

In short: The paper demonstrates that a team of AI agents, led by a smart model like GPT-5, can act like a human physicist. They can write their own code, debug their own mistakes, and find hidden signals in data as well as a human can. We are taking the first steps toward a future where the "Discovery Machine" is a team of digital agents working alongside us.

Technical Summary

1. Problem Statement

Modern high-energy physics (HEP) experiments, such as those at the Large Hadron Collider (LHC), generate massive, high-dimensional datasets requiring increasingly complex, multi-stage analysis workflows. While machine learning (ML) tools are widely used, they are typically special-purpose algorithms requiring significant human effort for integration, calibration, and workflow coordination. This complexity threatens to bottleneck scientific output.

The paper investigates a novel, orthogonal approach: using Large Language Models (LLMs) as autonomous agents to automate the entire data analysis pipeline. The goal is to determine if a team of LLM-based agents can independently solve a complex physics problem (anomaly detection) by writing code, executing tools, and iterating on results, mimicking the workflow of a human researcher without direct human intervention in the coding loop.

2. Methodology

A. The Agentic Framework

The authors designed a multi-agent system consisting of four distinct roles, orchestrated by a central Researcher agent:

1. Researcher: The main actor that plans the workflow, manages tasks, and interprets results. It has access to tools for requesting code, executing Python scripts, viewing files/images, and submitting final reports.
2. Coder: An agent specialized in writing Python code based on the Researcher's requests. It has access to standard scientific libraries (NumPy, Pandas, Scikit-learn, etc.).
3. Code Reviewer: An agent that reviews the Coder's output for syntax errors, style compliance, and alignment with the task description before execution.
4. Logic Reviewer: An agent that critically evaluates the Researcher's interpretations of data and results to ensure logical consistency.

Tools: The agents interact with a local execution environment via tools to run Python scripts, view generated plots/text, and submit numeric values. The system operates autonomously after the initial prompt, iterating until a task is completed or a call limit is reached.

B. The Benchmark Task: LHC Olympics Anomaly Detection

The system was tested on the LHC Olympics (LHCO) R&D dataset, a standard benchmark for anomaly detection in HEP.

• Task: Identify a hidden "signal" (new physics) within a background of Standard Model QCD dijet events.
• Setup: The agent was provided with a "mixed" dataset (background + rare signal) and a pure "background" reference dataset.
• Constraints: The agent had no access to ground truth labels during the analysis (a "black box" scenario).
• Deliverables: The agent was required to:
  1. Determine if new physics exists.
  2. Report a p-value for the null hypothesis.
  3. Estimate the signal fraction and resonance mass.
  4. Submit a score file ranking events by anomaly likelihood.
• Metrics: Performance was evaluated using Significance Improvement Characteristic (SIC), p-values, signal mass accuracy, and technical metrics (token usage, cost, error rates).

C. Experimental Variables

The study varied several parameters to test robustness and capability:

• LLM Models: Four OpenAI models were tested: GPT-4o, GPT-4.1, o4-mini, and GPT-5.
• Prompting Strategies:
  • Default: Basic task description.
  • Ideas: Encouraged brainstorming multiple approaches.
  • ML: Explicitly suggested using machine learning.
  • Feedback Loop (FBL): Allowed the agent to submit scores and receive performance metrics (AUC, SIC) to iterate and refine its method.
  • Paraphrasing: Tested variations in tone (e.g., "survival of humanity," "CERN integration") and structure.
• Task Complexity: Variations included splitting the task into sub-questions and testing on the full mass range vs. a pre-defined signal region.

3. Key Contributions

1. First Systematic Agent-Based Physics Analysis: This work presents the first comprehensive framework for using agentic LLMs to perform end-to-end anomaly detection in high-energy physics.
2. Multi-Agent Architecture: Demonstrates a viable architecture where specialized agents (Researcher, Coder, Reviewers) collaborate to solve scientific problems, preserving reproducibility and transparency.
3. Benchmarking LLMs in Science: Provides a rigorous comparison of state-of-the-art models (including the newly released GPT-5) in a scientific context, highlighting differences in stability, coding capability, and physics intuition.
4. Prompt Engineering Insights: Quantifies the impact of prompting strategies (e.g., explicit ML hints, feedback loops, and narrative framing) on scientific performance.

4. Key Results

A. Model Performance

• GPT-5 emerged as the superior model. It achieved the highest Significance Improvement Characteristic (SIC) (average ~4.46), closely mirroring human state-of-the-art results.
  • It consistently employed advanced physics techniques, including bump hunts and Classification Without Labels (CWoLa).
  • Crucially, GPT-5 was the only model to correctly identify the need to exclude the signal mass variable from training to avoid "mass sculpting" (a common pitfall in anomaly detection).
  • It produced the most error-free code and required the fewest retries.
• GPT-4.1 showed strong performance (avg SIC ~3.36) and high stability, often matching GPT-5 in specific runs but with lower cost.
• GPT-4o and o4-mini struggled significantly. GPT-4o had a high failure rate (only 5/16 successful runs) and poor physics intuition. o4-mini failed to exclude the signal mass, leading to lower SIC scores.

B. Impact of Prompting and Feedback

• ML Hint: Explicitly telling the agent to use Machine Learning significantly improved physics performance compared to the default prompt.
• Feedback Loop (FBL): Providing the agent with performance metrics (SIC, AUC) allowed it to iterate and refine its models. In one specific ML+FBL+ run, the agent successfully discovered the hidden resonance, reporting a mass of 3.47 GeV (true value 3.5 TeV, noting a unit confusion in the agent's output) and a signal fraction of 0.53% (true value 0.6%).
• Prompt Phrasing: "Storytelling" prompts (e.g., framing the task as crucial for the "survival of humanity") yielded better results than concise prompts, suggesting that narrative context enhances agent focus.

C. Technical Stability

• Reproducibility: Runs over several days showed stable average performance, though API response times varied with server load.
• Cost: GPT-5 was the most expensive due to high token usage (longer reasoning and code generation), costing ~$1.21 per run, compared to ~$0.30 for GPT-4.1. However, the cost remains low enough for large-scale deployment.

5. Significance and Conclusion

The paper demonstrates that agentic LLM systems are capable of performing complex, routine data analysis tasks in particle physics at a level comparable to human experts.

• Automation Potential: While current models may not yet surpass human experts in novel discovery, they can effectively automate repetitive analysis components, calibration, and standard anomaly detection, freeing up human researchers for more creative tasks.
• Model Evolution: The results highlight a clear performance leap from GPT-4 series models to GPT-5, suggesting that future advancements in LLM reasoning and coding capabilities will directly translate to improved scientific discovery tools.
• Future Directions: The authors propose extending this framework to full LHC analysis workflows, including global statistical modeling and calibration procedures, marking a significant step toward "self-driving" scientific experiments.

In summary, "Agents of Discovery" validates the feasibility of using autonomous AI agents to navigate the complexity of modern physics data analysis, offering a promising path to mitigate the growing bottleneck of human effort in scientific research.