Neurodata Without Boredom: Benchmarking Agentic AI for Data Reuse

This paper benchmarks agentic AI's ability to automate the reuse of fragmented neuroscience data by testing its performance on loading, understanding, and reformatting datasets from eight recent studies, revealing that while agents excel at individual sub-tasks, they currently struggle to produce fully error-free end-to-end solutions and require human-in-the-loop oversight.

The Gist

Imagine you are a chef who wants to cook a giant, delicious stew using recipes and ingredients from eight different kitchens. Each kitchen has its own way of organizing things: one uses jars labeled "Spicy," another uses boxes labeled "Hot," and a third just throws everything into a bucket with a sticky note that says "Maybe."

To make the stew, you first have to figure out what's in every single container, translate the labels so they all mean the same thing, and then mix them together. In the world of neuroscience, this "stew" is data about how mouse brains work, and the "kitchens" are different research labs.

This paper, titled "Neurodata Without Boredom," asks a simple but difficult question: Can a smart computer robot (an "Agentic AI") do this boring, messy translation work for us?

Here is the breakdown of what the researchers found, using simple analogies:

The Problem: The "Lost in Translation" Mess

Neuroscience data is incredibly fragmented. Some labs save data in a standard format (like a universal language), while others use custom formats (like a secret code only they understand).

• The Old Way: A human scientist has to read the lab's paper, look at their code, open their files, and manually figure out how to translate everything into a common format. This is slow, tedious, and prone to human error.
• The New Hope: Large Language Models (LLMs) are like super-fast, hyper-focused interns. They can read code and text faster than humans and don't get bored. The researchers wondered: Can these AI interns do the translation job perfectly?

The Experiment: The "Eight Kitchen" Challenge

The researchers set up a test with eight different neuroscience papers (the eight kitchens).

1. The Setup: They gave two different AI agents (named Claude Code and Codex) the raw data, the code, and the scientific paper for each kitchen.
2. The Task: The AI had to act like a translator. It needed to read the messy, unique files from each lab and convert them into a single, clean format that could be used to train a computer to predict mouse behavior (like "Will the mouse turn left or right?").
3. The Rules: The AI had to follow a strict checklist, write down its notes, and prove it understood the data before moving on.

The Results: Good at Steps, Bad at the Whole Journey

The results were a mix of impressive capability and frustrating inconsistency.

1. The AI is a Great "Step-Doer"
If you asked the AI to do just one small task—like "load this file" or "count the number of mice"—it usually did a fantastic job. It was often as good as, or even better than, a human expert at these isolated steps.

2. The AI Struggles with the "Marathon"
The problem happened when the AI had to string all those steps together into one long, error-free chain.

• The Analogy: Imagine a relay race. The AI is excellent at running its own leg of the race. But often, it drops the baton right before handing it off to the next runner, or it hands it to the wrong person.
• The Reality: In many cases, the AI would write code that ran (didn't crash), but the data inside was slightly wrong. For example, it might decide to count a "trial" (a single experiment) in seconds when the paper said minutes, or it might accidentally filter out important brain cells because it guessed the wrong rule.

3. The "Subtle Mistakes" Trap
The most dangerous errors were the ones that looked correct on the surface.

• Example: In one case, the AI decided to group data by "experiment ID" instead of "session ID." It sounded logical, but it split a single recording session into multiple fake sessions, ruining the data. The code ran perfectly, but the science was broken.
• The Takeaway: These mistakes were like a translator who swaps "left" and "right" in a recipe. The cake still bakes, but it tastes wrong.

The "Self-Check" Failure

The researchers also asked the AI to grade its own work. They asked, "Did you make any mistakes?"

• The Result: The AI was a terrible judge. It often missed its own big errors or flagged perfectly fine decisions as mistakes. It was like a student who thinks they got an 'A' on a test they actually failed.
• Conclusion: You cannot rely on the AI to check its own homework. A human still needs to look over the shoulder.

The Final Verdict

The paper concludes that Agentic AI is a powerful tool, but not a magic wand.

• What it can do: It can drastically reduce the "boredom" and time it takes to get started with a new dataset. It can do the heavy lifting of reading and initial translation.
• What it can't do yet: It cannot be trusted to work completely alone. It lacks the "common sense" and deep scientific intuition to catch subtle, high-stakes errors.
• The Future Workflow: The best approach is a human-in-the-loop system. Think of the AI as a very fast, very eager intern who does 90% of the work, and the human scientist as the supervisor who reviews the final product to catch the tricky 10% of errors that the AI missed.

In short: The AI can help us stop being bored by data formatting, but we still need to be the ones holding the steering wheel to make sure we don't drive off a cliff.

Technical Summary

Technical Summary: Neurodata Without Boredom: Benchmarking Agentic AI for Data Reuse

Problem Statement

Neuroscience data is inherently fragmented across laboratories, experimental paradigms, and recording modalities (e.g., electrophysiology, calcium imaging). While standardization initiatives like Neurodata Without Borders (NWB) and consortium-scale curation projects (e.g., International Brain Lab, Allen Brain Observatory) have improved interoperability, a fundamental tension remains: formats flexible enough to accommodate diverse experiments are rarely self-explanatory, while sufficiently descriptive formats require documentation burdens that few labs can sustain. Consequently, reusing shared data often demands substantial manual effort to decipher bespoke formatting choices, creating a bottleneck for cross-dataset integration and the training of foundation-scale models of brain and behavior.

Agentic AI, particularly Large Language Models (LLMs) capable of reading code and text with sustained attention to low-level details, presents a potential solution. However, it remains unclear whether current coding agents can reliably navigate the ambiguity of scientific data formats to produce error-free, end-to-end data conversion pipelines suitable for downstream analysis.

Methodology

The authors introduced a realistic benchmark framework to evaluate agentic AI on the task of scientific data reuse.

Dataset Selection:
The study utilized eight recent papers studying large-scale mouse neural population recordings that released both data and code. The datasets spanned:

• Modalities: Two-photon calcium imaging, Neuropixels electrophysiology, and mesoscope calcium imaging.
• Paradigms: Head-fixed decision making, virtual-reality navigation, and freely moving behavior.
• Formats: NWB files, specialized consortium APIs (IBL, ABO), and custom Python/MATLAB files.
• Scale: Varied by several orders of magnitude in terms of animals, sessions, and recorded neurons.

Task Design:
For each paper, agents were tasked with loading the raw data, understanding the methods and code, and reformatting the data into a common, prescribed structure suitable for training a linear decoder. The decoder's goal was to predict task or behavioral variables from neural activity. The output format followed a hierarchy of subjects, sessions, and trials, with specific requirements for input/output variable alignment and discretization.

Agent Setup:
Two general-purpose coding agents were evaluated: Claude Code (Opus 4.6) and Codex (GPT 5.4).

• Environment: Containerized with pre-installed dependencies (PyNWB, suite2p, dandi, ONE-api, allen-sdk).
• Prompting: Agents received the raw data, analysis code, paper PDF, and a structured prompt. The prompt enforced a rigorous 14-step workflow, requiring the agent to maintain a CONVERSION_NOTES.md worksheet to document decisions, verify consistency across sources (code, data, paper), and validate results on a small sample before processing the full dataset.
• Evaluation Protocol: Each dataset-agent pair was run three times. Evaluation occurred on two axes:
  1. Outcome-based: Checking if output files were created, if dataset statistics (sessions, trials, neurons) matched reference values (where available), and if downstream decoder performance reached a threshold (95% of reference accuracy).
  2. Process-based: A manual rubric scored the scientific and implementation decisions for each subtask (data loading, trial construction, preprocessing, variable construction, missing-data handling, code efficiency) on a five-level scale: incorrect, concerning, ok, match, better.

Self-Evaluation Test:
The authors also tested whether agents could act as judges, rating their own or other agents' solutions using the same manual rubric, to determine if automated quality control is feasible.

Key Contributions

1. Benchmark Framework: Introduction of a comprehensive benchmark for evaluating coding agents on scientific data reuse, covering eight heterogeneous neuroscience datasets with diverse formats and paradigms.
2. Dual-Mode Evaluation: A novel evaluation strategy combining outcome-level metrics (decoder performance, statistics) with a process-based manual rubric. This approach revealed that while agents often succeed on individual subtasks, they frequently fail to maintain correctness across the entire pipeline, with errors often being subtle and undetectable by aggregate metrics alone.
3. Failure Mode Characterization: A detailed taxonomy of agent errors, categorized into:
   • FILTER: Inconsistent or unjustified data filtering.
   • TIME_RES: Incorrect temporal resolution or alignment choices.
   • PROCESS: Flawed implementation of complex processing pipelines.
   • ASSUME: Incorrect assumptions about data semantics without verification.
   • VARNAME: Misinterpretation of ambiguous variable names.
     The study highlights that these failures are often "superficially reasonable" and difficult to detect without careful inspection.
4. Agent-as-Judge Reliability: Empirical evidence showing that agents are unreliable judges of data conversion quality, particularly without ground-truth references, exhibiting low precision and recall in detecting errors.

Results

• End-to-End Performance: While agents successfully completed the workflow for every run and often matched gold-standard solutions on individual metrics, they rarely produced a fully error-free end-to-end solution in a single trial. Complete agreement across all checks within a single run was rare.
• Subtask Performance: Agents performed well on individual subtasks (e.g., data loading) but struggled with the integration of these steps. Variability was high; approximately 25% of subtasks showed at least a one-grade swing in rating across repeated runs, and 5% showed a three-grade swing (e.g., from "match" to "incorrect").
• Format Independence: Agent performance did not significantly improve on datasets shared in more deliberate formats (e.g., NWB or consortium APIs) compared to custom formats. Agents frequently bypassed provided APIs to process raw data directly, even when instructed otherwise.
• Error Nature: Most errors were not simple execution failures but reflected mistakes in scientific interpretation (e.g., assuming a variable's meaning based on a "common-case" convention rather than verifying it against the specific dataset).
• Self-Evaluation: Agents acting as judges performed only slightly above chance on unsupervised datasets (balanced accuracy ~60%). Even with ground-truth references (supervised datasets), while balanced accuracy improved (up to 78.4%), precision remained low, indicating agents flagged many acceptable decisions as errors.

Significance and Claims

The paper claims that current agentic AI frameworks possess the capability to perform the necessary steps of scientific data reuse but lack the reliability to maintain correctness across long chains of processing steps required for scientific rigor.

• Human-in-the-Loop Necessity: The authors conclude that while agents can substantially reduce the tedious onboarding work for new datasets, human-in-the-loop review remains necessary. Specifically, human review should focus on the subtasks where agents are most prone to subtle, scientifically significant errors.
• Data Sharing Best Practices: The study suggests practical guidelines for data sharing in the agentic-AI era:
  • Releases should include working analysis code that directly interacts with data to serve as an executable reference.
  • Variable names should be clear, unambiguous, and consistent with their intended meaning.
  • Datasets should provide processed variables expected by downstream users rather than requiring reconstruction from raw data.
  • Providing expected dataset statistics and sanity-check targets could improve agent reliability.
• Limitations: The authors acknowledge the modest scale of the benchmark (eight datasets, two agents) and the reliance on manual ratings. They emphasize that their findings support the continued need for expert judgment in scientific data reuse, positioning agentic AI as a powerful assistant rather than an autonomous replacement for human curation.