🤖 machine learning

A case study of evaluating AI agents on a neuroscience data-to-discovery pipeline

This paper presents an empirical study evaluating general-purpose AI coding agents on a complex neuroscience data-to-discovery pipeline, revealing that while agents can automate individual stages, they currently struggle with end-to-end execution, self-evaluation without predefined criteria, and generalization to new data.

Original authors: Kai A. Horstmann, Ethan Lin, Alice A. Robie, Jennifer J. Sun, Kristin Branson

Published 2026-08-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kai A. Horstmann, Ethan Lin, Alice A. Robie, Jennifer J. Sun, Kristin Branson

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where scientists don't just run experiments, but also spend months writing the complex computer code needed to analyze the results. In the field of neuroscience, researchers often study tiny creatures like fruit flies to understand how brains control behavior. To do this, they record hours of video, but the raw footage is just a blur of moving dots. To turn that blur into a scientific discovery, they need a "pipeline"—a long, winding assembly line of software steps. First, the computer must find the flies in the video. Then, it has to track their every move, measure their body parts, figure out if they are walking or standing still, and finally run statistical tests to see if a specific experiment changed their behavior. Traditionally, building this assembly line is a massive, tedious job that takes human experts weeks or even months of coding.

Recently, a new kind of artificial intelligence called an "AI agent" has emerged. Think of these agents as super-smart, tireless digital interns who can read instructions and write code for you. The big question on everyone's mind is: Can these AI interns take over the boring, time-consuming parts of scientific research? Can they build the assembly line themselves, or do they need a human to hold their hand every step of the way? This is the exact mystery a team of researchers set out to solve. They wanted to see if these AI agents could handle the messy, real-world challenges of neuroscience, or if they would get stuck in the details.

The researchers decided to test their AI interns on a very specific, high-stakes challenge: a "fly optogenetics data-to-discovery pipeline." Imagine this pipeline as a seven-stage obstacle course designed to turn raw video of fruit flies into a scientific conclusion. The course starts with Fly Body Tracking, where the AI must find and follow multiple flies in a video, keeping their identities straight even when they bump into each other. Next comes Registration, where the AI has to clean up the data and translate it from pixel coordinates into real-world measurements like millimeters. Then, the AI must perform Keypoint Tracking, which is like drawing a skeleton on every fly to pinpoint 21 specific body parts with extreme precision.

The course continues with Feature Computation, where the AI calculates how fast the flies are moving or turning. It then moves to Walking Behavior Classification, asking the AI to decide if a fly is walking or just standing still. The sixth stage is Gait Segmentation, a tricky step where the AI must determine if a specific leg is in the air (swinging) or on the ground (pushing). Finally, the AI reaches the finish line with Statistical Comparisons, where it must crunch the numbers to see if the experiment actually changed the flies' behavior. The researchers tested several different AI models on this course, both by asking them to solve just one stage at a time and by asking them to run the entire seven-stage marathon from start to finish.

The results were a mix of impressive success and frustrating failure, revealing that AI is ready for some jobs but not others. When the AI agents were given a single, well-defined task with a clear "scorecard"—like training a model to recognize walking behavior or tracking specific body parts—they did surprisingly well. In these "supervised learning" tasks, the agents could iterate, check their own work, and improve until they got it right. For example, they successfully learned to identify the 21 keypoints on a fly's body, matching the accuracy of human experts. This suggests that for specific, rule-based parts of the scientific pipeline, AI agents are already capable of doing the heavy lifting.

However, the story changes dramatically when the AI agents were asked to run the entire pipeline end-to-end without a detailed map. When the researchers gave the agents a broad goal like "analyze the flies" without breaking it down into steps, the agents struggled to keep their focus. They often forgot the earlier steps, merged complex stages into simple shortcuts, or made basic formatting errors that caused the whole chain to break. The most significant failure happened in the very first stage: Fly Body Tracking. This task required the AI to use its "scientific judgment" to figure out how to track flies without a pre-written rulebook or a clear score to aim for. The agents tried to look at the video to check their work, but they largely failed to understand what they were seeing. They would spot an error, misinterpret the visual clue, and then make the same mistake again, or even make it worse.

The paper suggests that the main bottleneck isn't that the AI can't write code; it's that they struggle with the "long game." They are great at solving a single puzzle if the rules are clear, but they get lost when they have to string together a long sequence of complex steps, manage computer resources, or use visual inspection to fix their own mistakes. The researchers found that while AI can automate specific stages of scientific discovery, the dream of a fully autonomous "AI scientist" that can run a whole experiment from raw data to a conclusion is still out of reach. The agents need better guidance, clearer goals, and a way to truly "see" and understand the data they are processing before they can be trusted to run the entire show on their own.

Technical Summary: Evaluating AI Agents on a Neuroscience Data-to-Discovery Pipeline

Problem Statement

Scientific research pipelines, particularly in behavioral neuroscience, involve complex, multi-stage workflows that transform raw experimental data into scientific insights. While "AI scientist" or fully autonomous lab approaches are emerging, they lack rigorous evidence of human-level performance and established metrics for assessing scientific reasoning, creativity, and validity. Current benchmarks for AI agents in science often fail to capture the scale, complexity, and domain-specific rigor required for real-world scientific bottlenecks. They typically involve small codebases, limited datasets, and evaluation criteria that do not reflect the high bar for correctness and robustness demanded by domain experts.

This paper addresses the gap by presenting an empirical study of general-purpose coding agents on a representative, end-to-end neuroscience pipeline: fly optogenetics data-to-discovery. The core challenge is to evaluate whether current agentic AI systems can not only solve individual, complex sub-tasks (ranging from computer vision tracking to statistical inference) but also successfully orchestrate these stages into a coherent, valid scientific workflow.

Methodology

The Fly Optogenetics Pipeline

The authors constructed a benchmark based on a 7-stage pipeline used to analyze fly behavior under optogenetic perturbation. The pipeline transforms raw video data into statistical conclusions:

Fly Body Tracking: Detecting and tracking multiple interacting flies in raw video, maintaining identity across frames.
Registration: Scaling and translating trajectories into a common coordinate system (millimeters) and cleaning tracking gaps.
Keypoint Tracking: Estimating the positions of 21 anatomical keypoints per fly with sub-pixel precision.
Behavior Feature Computation: Calculating interpretable movement features (e.g., velocity, orientation changes) from trajectories.
Walking Behavior Classification: Classifying whether a fly is walking or stationary in each frame using a trained classifier.
Gait Segmentation: Segmenting leg motion into "swing" (lifted) and "stance" (grounded) phases.
Statistical Comparisons: Performing Mann-Whitney U tests to determine if optogenetic perturbation significantly affects stance duration compared to a control group.

Evaluation Framework

The study evaluates agents under two settings:

Single-Stage Tasks: Isolating individual pipeline components to assess specific capabilities.
End-to-End (E2E) Tasks: Requiring agents to execute the full workflow. This includes a "Minimal Prompt" (only the final goal is given) and a "Maximal Prompt" (all stage-level prompts are provided).

Agents Tested: The study utilized general-purpose coding agents based on different models (e.g., claude-opus-4-6, gpt-5.4) and frameworks (claude-code, codex, terminus-2).

Evaluation Criteria:

Ground Truth Available: For tasks like tracking and classification, agents are compared against labeled ground truth. An agent passes if its score is within a tolerance ( $\epsilon$ ) of a scientist's reference solution.
Ground Truth Unavailable: For tasks like feature computation or statistical comparisons, the scientist's solution serves as the reference. The agent passes if its output deviates from the reference by less than a defined tolerance.
Task Size: Tasks are significantly larger than existing benchmarks (average ~447 lines of code vs. ~41 in ScienceAgentBench) and operate on datasets >250x larger (video and trajectory data vs. small tabular data).

Key Contributions

A Realistic, Large-Scale Benchmark: The authors introduce a comprehensive evaluation framework based on a real neuroscience pipeline. The tasks are designed to mirror the "time-consuming bottlenecks" scientists face, requiring days to months of human effort to build and demanding generalization to unseen, out-of-distribution data.
Rigorous Evaluation Design: The paper distills principles for constructing scientific tasks, emphasizing the need for:
- Avoiding over-prescription (allowing agents to choose methods) while avoiding under-specification (ensuring clear success criteria).
- Isolating evaluation from incidental implementation choices.
- Using robust evaluation metrics (e.g., tolerance-sensitivity analysis) rather than relying on LLM-as-a-judge, which is deemed unreliable for scientific validity.
Granular Failure and Success Mode Analysis: Beyond pass/fail rates, the authors manually analyzed agent code iterations and trajectories. They identified specific failure modes, such as the inability to iterate productively without pre-defined metrics and the failure to translate visual evidence into corrective actions.
Stage vs. Pipeline Feasibility: The study reveals a critical gap between stage-level and pipeline-level performance, highlighting that while individual stages may be tractable, the end-to-end orchestration remains a significant challenge.

Results

Quantitative Performance

Single-Stage Success: Agents achieved reliable performance on several stages, particularly Feature Computation and Behavior Classification, where clear metrics and training data were provided. Keypoint Tracking and Registration were also largely successful.
Single-Stage Failure: Body Tracking (requiring sustained visual reasoning and identity maintenance) and Gait Segmentation proved extremely difficult, with near-zero pass rates across most configurations.
End-to-End Degradation: Performance degraded markedly in E2E settings. Stages that passed in isolation often failed when composed. Notably, Statistical Comparisons failed in all E2E trials, even when upstream stages were provided with "gold" (perfect) inputs, suggesting the final stage's failure is due to compositional difficulty and the agent's inability to manage long-context instructions and sustained attention.

Qualitative Failure Modes

Lack of Self-Evaluation: Agents struggle most when tasks lack a pre-defined scalar metric (e.g., Body Tracking). Without a clear "correctness" signal, agents fail to iterate productively, often submitting solutions worse than their initial attempts.
Visual Reasoning Deficits: While agents frequently attempt to visualize intermediate outputs (mimicking scientists), they largely fail to interpret these visualizations correctly or act on them. Common failure modes include "Latent Miss" (seeing an error but misinterpreting it) and "Dismissed" (rationalizing an error as benign).
Resource and Time Management: Agents frequently misjudge runtime (e.g., confusing CPU time with wall-clock time), leading to premature termination of training processes. They also struggle with dependency management, sometimes reimplementing standard library functions instead of installing packages.
Attention and Effort: In maximal E2E tasks, agents exhibited basic formatting errors and simplified implementations (e.g., fewer training epochs, omitting data augmentation) compared to single-stage runs, indicating a loss of focus over long horizons.

Success Modes

ML-Style Tasks: Agents reliably solved tasks with clear supervised learning setups (Keypoint Tracking, Behavior Classification), iterating productively toward the provided metric.
Domain Knowledge Integration: Agents successfully applied implicit domain knowledge (e.g., understanding fly anatomy to define "swing" vs. "stance" phases) even when not explicitly prompted.
Parallelization: In E2E trials, some agents successfully identified opportunities to parallelize sub-tasks (e.g., training a model while developing another pipeline stage), though this was not universal.

Significance and Claims

The paper claims that stage-level automation for complex scientific pipelines is currently tractable for general-purpose agents, offering a promising near-term path to alleviating software development bottlenecks in research. However, end-to-end automation remains beyond current capabilities due to challenges in sustained attention, long-horizon task management, and the inability to perform the visual self-evaluation and iterative refinement that human scientists rely on.

The authors emphasize that their findings are specific to a single pipeline but hypothesize they generalize to scientific tasks sharing structural properties: high precision requirements, large-scale data, multi-stage workflows, and reliance on visual or data-based validation. They argue that future progress requires:

Better Evaluation: Moving beyond small-scale benchmarks to rigorous, domain-grounded criteria that can assess scientific validity.
Enhanced Reasoning: Improving agents' ability to close the loop from visual inspection to corrective action and to manage computational resources effectively.
Task Design: Constructing tasks that balance open-endedness with clear evaluation criteria, avoiding both over-prescription and under-specification.

Ultimately, the paper serves as a cautionary yet constructive assessment, suggesting that while "AI scientists" capable of full autonomy are not yet ready, agentic AI can effectively automate specific, high-value stages of the scientific workflow if evaluated with the rigor required by the scientific community.