💻 computer science

Designing Any Imaging System from Natural Language: Agent-Constrained Composition over a Finite Primitive Basis

This paper introduces an autonomous multi-agent system and a structured specification format that translate natural language descriptions into validated computational imaging designs, achieving expert-level quality and enabling novel compositional modalities while providing a theoretical framework for bounding reconstruction errors.

Original authors: Chengshuai Yang

Published 2026-03-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chengshuai Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to build a custom camera. In the past, if you wanted to design a new type of medical scanner or a super-powerful microscope, you needed a team of PhD physicists and engineers. They would spend weeks or even months writing thousands of lines of complex code, tweaking numbers, and testing hardware to make sure the math actually worked. If you made a tiny mistake, the whole image would be blurry or useless.

This paper introduces a "Magic Architect" that does all that work for you in minutes.

Here is how it works, broken down into simple concepts:

1. The "Lego" of Light (The Finite Primitive Basis)

Think of every imaging system (CT scans, MRI machines, cameras, microscopes) as a giant, complex machine. The authors discovered that you don't need to invent a new machine for every job. Instead, there are only 11 basic "Lego bricks" (called primitives) that make up all imaging systems.

These bricks are things like:

Projecting a shadow (like a CT scan).
Spreading out colors (like a prism).
Wiggling waves (like sound or radio waves).
Detecting the signal.

No matter how complex the machine is, it's just these 11 bricks snapped together in a specific order. The paper proves that if you can describe a system using these 11 bricks, you can build it.

2. The Three Robot Workers (The Agents)

The system uses three AI "agents" (robots) that work together like a construction crew:

The Planner (AP): You type a simple sentence like, "I need a camera that can see inside a fruit to find rot without cutting it open." The Planner translates this into a strict, technical blueprint called spec.md. It picks the right Lego bricks and snaps them together.
The Judge (AJ): This is the strict inspector. Before anything is built, the Judge checks the blueprint against three "Safety Gates":
1. Recoverability: "Did we take enough pictures to actually solve the puzzle?"
2. Carrier Budget: "Is the light/signal strong enough to be seen?"
3. Mismatch: "Does our math match the real physics?"
  If the blueprint fails, the Judge sends it back to the Planner to fix it. They do this automatically until it passes.
The Executor (AE): Once the blueprint is approved, this robot builds the actual image reconstruction algorithm and runs the test.

3. The "Error Receipt" (The Theorem)

Usually, when an image is blurry, you don't know why. Is it the camera? The math? The lighting?

This paper introduces a 5-part Error Receipt. It breaks down exactly why an image might be imperfect into five specific buckets:

The Lego Limit: Did we use the right bricks?
The Blueprint: Was the instruction clear?
The Translation: Did we misunderstand the sentence?
The Settings: Did we tune the knobs (parameters) correctly?
The Unknown: Is there some physics we didn't account for?

If the image is bad, the system tells you exactly which bucket is leaking so you can fix just that part. It's like a mechanic telling you, "Your car isn't running because the battery is dead," rather than just saying, "The car is broken."

4. The Results: From "Expert Only" to "Anyone"

The team tested this on 173 different types of imaging systems (from X-rays to electron microscopes).

Speed: What used to take experts weeks now takes the AI minutes.
Quality: The images produced by the AI are 98% as good as those made by human experts.
New Creations: The AI didn't just copy old cameras; it invented 10 new types of cameras that no human had ever designed before, capable of capturing 5D data (3D space + color + time) in a single snapshot.

The Big Picture Analogy

Imagine that designing an imaging system used to be like writing a symphony from scratch. You needed a master composer who knew every instrument, every note, and every rule of music theory.

This paper gives you a smart music app. You just hum a tune (natural language), and the app:

Picks the right instruments (Lego bricks).
Checks if the song is playable (The Judge).
Writes the sheet music (The Spec).
Plays the song perfectly (The Executor).

It democratizes science. Now, a biologist or a doctor can design their own custom imaging tool without needing to be a math wizard. They just need to know what they want to see.

1. Problem Statement

Designing computational imaging systems (selecting operators, setting parameters, and validating consistency) currently requires weeks of specialized effort per modality. This creates an expertise bottleneck that prevents the broader scientific community from prototyping new imaging instruments. While a companion paper established that 11 canonical "primitives" can represent any imaging forward model across 170+ modalities, the challenge remained: how to automate the translation from a natural-language intent to a validated, real-world system design with bounded reconstruction error.

2. Methodology

The authors propose a framework that bridges natural language and physical reality using a structured specification format and a three-agent autonomous pipeline.

A. The Finite Primitive Basis (FPB)

The system relies on a companion theorem stating that 11 canonical primitives (e.g., Projection, Fourier Encode, Modulate, Convolve, Detect) are sufficient to represent any imaging forward model across 5 carrier families (photon, X-ray, electron, acoustic, spin/particle) with a representation error $\epsilon_{FPB} < 0.01$ . These primitives are implemented at four fidelity tiers (Tier-0 geometric to Tier-3 full-physics).

B. The `spec.md` Specification Format

The core artifact is spec.md, a structured 8-field Markdown format that decouples the imaging problem from implementation:

Modality, Carrier, Geometry, Object, Forward Model, Noise, Target: Physics definitions.
System Elements: Hardware constraints (source power, detector QE, optics) linking the spec to feasibility and cost.
Significance: This format serves as a shareable, peer-reviewable scientific artifact, similar to how FASTA standardized genomics.

C. The Three-Agent Pipeline

The design process is automated by three specialized agents:

Plan Agent ( $A_P$ ): Translates a one-sentence natural language prompt into a spec.md. It selects FPB primitives, parameters, and noise models from a registry of 173 modalities.
Judge Agent ( $A_J$ ): Validates the design through a rigorous three-stage process:
- Structural Compiler (6 Checks): Verifies DAG acyclicity, chain fidelity, complexity bounds, nonlinear constraints, adjoint consistency, and representation error.
- Triad Gates (3 Deterministic Checks):
  - Gate 1 (Recoverability): Ensures measurement geometry captures sufficient information (Compression Bound).
  - Gate 2 (Carrier Budget): Ensures SNR exceeds the noise floor based on hardware specs.
  - Gate 3 (Operator Mismatch): Checks if the forward model faithfully represents intended physics given calibration tolerances.
- Feasibility/Cost: Estimates hardware cost and realizability.
- Feedback Loop: If checks fail, $A_P$ regenerates the spec (up to 3 rounds) or returns diagnostics to the user.
Execute Agent ( $A_E$ ): Selects the reconstruction algorithm (e.g., FISTA-TV, GAP-TV), executes the reconstruction, and predicts quality using analytical estimates (condition numbers) and calibrated lookups.

D. Design-to-Real Error Theorem (Theorem 1)

The paper introduces a formal error decomposition theorem that bounds the total reconstruction error ( $\|\hat{x} - x^*\|$ ) into five independently measurable terms:

$\epsilon_{FPB}$ : Basis representation error (bounded by the primitive library).
$\epsilon_{spec}$ : Specification error (zero if compiler checks pass).
$\epsilon_{trans}$ : Translation error (zero if the chain matches the canonical registry).
$\epsilon_{param}$ : Parameter mismatch error (reducible via calibration).
$\epsilon_{unmod}$ : Unmodeled physics error (reducible via "Tier-lifting" to higher fidelity models).
$\epsilon_{recon}$ : Irreducible noise floor/bias.

This theorem acts as a "design-to-deployment diagnostic," mapping specific error terms to corrective actions (e.g., if $\epsilon_{param}$ dominates, calibrate; if $\epsilon_{unmod}$ dominates, lift the physics tier).

3. Key Results

Validation Scope: The pipeline was tested on 173 modalities across all 5 carrier families.
- 100% passed structural compiler checks.
- 39 modalities had quantitative reconstruction validation.
- 6 modalities (CT, MRI, CASSI, Ultrasound, E-ptychography, PET) were validated on real measured data.
Performance vs. Experts:
- On real data, the automated pipeline achieved 98.1 ± 4.2% of the quality of expert-designed systems (measured by PSNR).
- Time Efficiency: The agents reduced design time by 190x to 420x (minutes vs. weeks/days).
- Error Bounds: The predicted error bounds were tight, with a tightness ratio $\tau$ (predicted/observed) of median 2.9, indicating the bound captures the dominant failure mechanisms.
Novel System Designs: The agents successfully designed 10 novel systems not previously in literature, composing primitives into chains up to 5 operators deep to recover 5D datacubes (x, y, z, $\lambda$ , t) from single 2D snapshots.
Ablation Study: Removing any single agent (Plan, Judge, or Execute) resulted in significant failure rates (e.g., without the Judge, 4/12 designs were physically invalid; without Execute, PSNR dropped by 2.1 dB).

4. Significance and Contributions

Democratization of Imaging Design: By converting natural language into validated forward models, the framework removes the expertise bottleneck, allowing non-specialists to prototype complex imaging instruments.
Standardization (spec.md): It establishes a "FAIR" (Findable, Accessible, Interoperable, Reusable) standard for imaging system specifications, separating physics from code and hardware.
Formal Error Guarantees: The Design-to-Real Error Theorem provides a mathematical contract between the user's intent and the system's performance, offering a diagnostic tool to pinpoint exactly why a system might underperform (e.g., calibration vs. unmodeled physics).
Compositional Reach: The system demonstrates that complex, high-dimensional imaging problems (5D) can be solved by composing a finite set of primitives, proving the universality of the Finite Primitive Basis.
Tier-Lifting Strategy: The framework introduces a protocol to handle scattering-dominated systems by dynamically switching from geometric to full-physics models during reconstruction, bridging the gap between simple design and complex reality.

Conclusion

This work represents a paradigm shift in computational imaging, moving from manual, modality-specific engineering to an automated, specification-driven design process. By combining a finite primitive basis, a structured specification language, and a multi-agent validation pipeline, the authors have created a system that matches expert quality while drastically reducing development time and enabling the discovery of novel imaging modalities.