🤖 AI

Imaging-101: Benchmarking LLM Coding Agents on Scientific Computational Imaging

This paper introduces Imaging-101, a benchmark of 57 expert-verified computational imaging tasks across six scientific domains designed to evaluate LLM coding agents, revealing that current frontier models struggle with domain-specific challenges like algorithm selection and physical convention handling, thus highlighting the need for specialized, skill-augmented agents to enable reliable scientific discovery.

Original authors: Siyi Chen, Jiahe Ying, Yixuan Jia, Yuxuan Gu, Enze Ye, Weimin Bai, Zhijun Zeng, Shaochi Ren, Binhong Gao, Yubing Li, Tianhan Zhang, He Sun

Published 2026-07-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Siyi Chen, Jiahe Ying, Yixuan Jia, Yuxuan Gu, Enze Ye, Weimin Bai, Zhijun Zeng, Shaochi Ren, Binhong Gao, Yubing Li, Tianhan Zhang, He Sun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a magical box that can turn a blurry, noisy photo of a secret object into a crystal-clear picture. In the real world of science, this isn't magic; it's called computational imaging. Scientists use it to see things they can't touch directly, like the inside of a black hole, the structure of a virus, or what's happening deep underground. But building the "recipe" to turn that blurry mess into a clear picture is incredibly hard. It requires a PhD-level understanding of physics, math, and coding, plus a lot of patience to fix tiny mistakes that ruin the whole image.

Enter Imaging-101, a new "driving test" for Artificial Intelligence. The researchers created a challenge with 57 different puzzles (tasks) taken from real scientific papers across six fields: astronomy, biology, chemistry, earth science, medicine, and physics. They didn't just ask the AI to write code; they asked it to build a complete, working machine that could solve these puzzles from start to finish.

The Three-Part Test

To see if the AI was truly smart or just lucky, the researchers gave it three different ways to take the test:

The Plan: First, the AI had to write a blueprint. Could it figure out the right math and steps before writing a single line of code?
The Lego Blocks: Next, it had to build specific parts of the machine (like the "preprocessing" or "solver" blocks) and pass tiny, strict unit tests for each one.
The Full Drive: Finally, it had to put the whole machine together, run it, and produce a picture that was accurate enough to pass a strict quality check.

The Results: AI is Good at Talking, Bad at Physics

The researchers tested seven of the smartest AI models available. Here is what they found:

The "Plan" was okay: The AIs were surprisingly good at writing the blueprint. About 52.6% of the time, the top models (GPT-5.4 and Kimi-k2.5) got the general idea right. They could name the right type of math to use.
The "Lego Blocks" were shaky: When it came time to actually build the parts, things got messy. The success rate for building all the parts correctly at once dropped to between 1.8% and 22.8%.
The "Full Drive" was the hardest: Only 29.8% of the time did the best AI (Claude-4.6-Opus) manage to build a machine that actually worked and produced a good picture. The others did even worse, with some only succeeding 7% of the time.

The "Invisible" Traps

The most interesting part of the study is why the AIs failed. They didn't fail because they couldn't write code; they failed because they missed the "secret rules" of physics that scientists learn through years of experience.

The paper calls this "numerical-convention drift." Imagine you are baking a cake. The AI knows the recipe says "add flour," but it forgets that in this specific kitchen, you must measure the flour in grams, not cups, and you must sift it first. The AI's cake looks like a cake and smells like a cake, but it's the wrong size and texture.

In the study, the AIs made these specific mistakes:

Wrong Units: In a task involving light scattering, the AI forgot to convert "intensity" (how bright the light is) into "amplitude" (how strong the wave is). It was like trying to measure a car's speed in "brightness" instead of "miles per hour."
Wrong Solvers: The AI would pick a generic math tool (like a standard hammer) when the job required a specialized tool (like a laser scalpel). It would say, "I'll use this common method," even though the paper it was following specifically required a rare, complex one.
Missing Normalization: In MRI scans, the AI forgot to adjust for the fact that some parts of the body are closer to the sensor than others. This made the final image look like it had a weird shadow, even though the code ran without errors.

The "Black Box" Surprise

The researchers also tested a "black-box" agent (Claude Code) that has access to a real computer terminal, can install software, and can look at files. This agent did much better, succeeding 56.1% of the time. This suggests that giving AI more freedom to explore and fix its own mistakes helps, but even with that advantage, it still isn't perfect.

What This Means

The paper makes it clear that while AI is getting better at writing code, it still struggles with the deep, tacit knowledge of scientific imaging. It's like a student who can recite the rules of baseball but doesn't know how to actually catch a curveball.

The researchers found that the AIs often fail because they rely on what is "statistically common" in their training data, rather than what is physically correct for the specific problem at hand. They suggest that to fix this, we might need to build AI assistants that come with a "toolbox" of pre-verified, domain-specific skills, rather than expecting them to figure out every physics rule from scratch.

In short, the AI can write the story of how to solve the puzzle, but it still needs a human expert to check the math and make sure the physics actually add up. The dream of a fully automated scientific discovery machine is still a work in progress.

Technical Summary: Imaging-101

Problem Statement

Computational imaging, the process of recovering hidden signals from indirect, noisy, and incomplete measurements, is fundamental to quantitative discovery across scientific disciplines ranging from astronomy to medicine. However, constructing a correct reconstruction pipeline is laborious, requiring deep domain expertise to handle data cleaning, forward physics modeling, algorithm selection, and iterative debugging. While Large Language Models (LLMs) have demonstrated proficiency in general coding and parts of the scientific workflow, it remains an open question whether frontier models can reliably solve end-to-end computational imaging tasks. These tasks require the joint reasoning of physical measurement processes, mathematical formulations, and numerical solvers, translating them into executable code that passes strict numerical verification. Existing benchmarks often fail to expose the specific failure modes inherent to scientific inverse problems, such as subtle physical convention errors or inappropriate solver selection.

Methodology

To address this gap, the authors introduce Imaging-101, a benchmark comprising 57 expert-verified computational imaging tasks spanning six scientific domains: astronomy, biology, chemistry & material science, earth science, medicine, and physics.

Benchmark Construction

The benchmark is constructed via an expert-in-the-loop pipeline designed to ensure scientific authenticity and reproducibility:

Source Grounding: Every task is derived from a peer-reviewed paper and its open-source reference implementation.
Standardization: Tasks are canonicalized into a standardized four-stage pipeline:
- Preprocessing: Converting raw measurements into a form compatible with the forward model.
- Forward Physics Modeling: Implementing the forward operator $A$ (and often its adjoint $A^*$ ).
- Inverse Solver: Choosing and executing an inversion algorithm to recover the latent quantity.
- Visualization: Rendering the reconstructed signal.
Verification: Domain experts (PhD level) validate the reproduction of reference runs against original publications. The pipeline utilizes AI assistance (Claude Code) for refactoring and unit-test synthesis, but experts retain final authority over correctness.
Evaluation Metrics: Tasks are graded using calibrated numerical tolerances. End-to-end success is measured by Normalized Cross-Correlation (NCC) and Normalized Root Mean Square Error (NRMSE), which are robust to global amplitude scaling, unlike PSIM or SSIM.

Evaluation Tracks

Imaging-101 employs three complementary evaluation tracks to probe distinct agent capabilities:

Planning: Agents generate a structured plan (algorithm description and module design) based on the problem statement. Correctness is judged by domain experts against reference plans.
Function-Level: Agents implement specific functions within the pipeline. Correctness is determined by passing a suite of pytest unit tests with strict numerical tolerances (deterministic vs. stochastic).
End-to-End: Agents generate the full pipeline code, execute it, and produce a reconstruction. Success is defined by meeting image quality thresholds (NCC/NRMSE) relative to the reference.

Key Contributions

Imaging-101 Benchmark: A collection of 57 tasks grounded in peer-reviewed literature, supporting planning, function-level, and end-to-end evaluation across six scientific domains.
Expert-Supervised Construction Pipeline: A methodology that canonicalizes research papers into a standardized four-stage layout, enabling scalable, reproducible, and execution-verifiable benchmark tasks.
Empirical Characterization of Failure Modes: The work provides evidence of specific capability gaps in current LLMs when applied to scientific computational imaging, distinguishing them from errors found in general coding benchmarks.

Results

The authors evaluated seven frontier LLMs (including Claude-4.6-Opus, GPT-5.4, and DeepSeek-V3) on the 57 tasks.

Quantitative Performance

Planning: GPT-5.4 and Kimi-k2.5 achieved the highest overall pass rates (~52.6%), followed closely by Claude-4.6-Opus (50.9%). Preprocessing was the most difficult planning dimension (67.9% mean pass rate), while inverse solver selection was the easiest (88.5%).
Function-Level: Performance dropped significantly at the implementation stage. The overall module pass rate (requiring all functions in a module to pass) ranged from 1.8% to 22.8% across models. Inverse solvers were the hardest to implement correctly (30.6% mean module pass rate).
End-to-End: The top-performing model, Claude-4.6-Opus, achieved a 29.8% success rate. GPT-5.4 followed at 19.3%. A black-box agent (Claude Code) achieved 56.1%, suggesting that richer execution affordances (e.g., unrestricted package installation, shell feedback) significantly improve performance.

Failure Mode Analysis

The evaluation revealed systematic failure modes distinct from general coding errors:

Scientific Reasoning Failures: The dominant errors were not surface-level syntax mistakes but failures in scientific reasoning.
- Measurement-to-Physics Misalignment: Agents often omitted critical preprocessing steps, such as converting sensor intensity ( $1/r^4$ ) to wavefield amplitude ( $1/r$ ) in non-line-of-sight imaging.
- Inappropriate Solver Selection: Models frequently substituted specialized, physics-coupled solvers (e.g., qNewton-PIE for Fourier ptychography) with generic optimizers (e.g., Adam), citing general heuristics that ignored the specific structure of the inverse problem.
Numerical Convention Drift: The most common code bug (42.1% of failures) involved using incorrect sign, scale, axis ordering, or normalization conventions (e.g., omitting MVUE coil normalization in MRI). These errors produced structurally valid code that passed visual checks but failed tight numerical parity tests.
Agent Non-Completion: 28.2% of failures occurred because agents exhausted their reflection budgets without producing code, often when dealing with specialized libraries (e.g., ehtim, deepwave), indicating a bottleneck in API familiarization rather than mathematical reasoning.

Significance

The paper argues that while LLMs show promise in scientific workflows, they currently lack the reliability required for autonomous computational imaging. The findings highlight that the "dark matter" of these pipelines—tacit domain knowledge regarding unit conventions, normalization factors, and measurement-model coupling—is not explicitly captured in written specifications and is systematically missed by current models.

The authors conclude that bridging this gap requires moving beyond general coding agents toward skill-augmented, domain-specialized agents. They propose that codifying tacit knowledge into reusable, verified modules (skill libraries) is a practical path toward reliable scientific assistance. Imaging-101 serves as a foundational benchmark to track progress in this direction, emphasizing that true reliability in scientific computing demands more than just code generation; it requires faithful adherence to physical laws and numerical precision.