Technical Summary: MIND THE ALIGNMENT GAP

Problem Statement

Scientific coding agents are difficult to benchmark because research tasks often require executable code but produce outputs that are ambiguous or hard to verify. While existing benchmarks (e.g., HumanEval, SWE-bench) rely on clear automated verification via unit tests or regression suites, real scientific workflows are iterative, open-ended, and often lack precise ground truth. Furthermore, constructing benchmarks for these tasks is resource-intensive. The authors argue that current evaluation frameworks often prioritize "hill-climbing" (improving scores on static metrics) over a user-centric workflow where scientists inspect agent traces to assess the scientific soundness of the approach. Specifically, there is a gap in understanding how providing agents with specialized scientific context (package names, environments) affects their ability to solve complex domain-specific problems like spatial transcriptomics alignment.

Methodology

The authors introduce an interactive framework to construct scientific-agent benchmarks from peer-reviewed papers and diagnose agent behavior through trace inspection.

Benchmark Construction:

• Source: The study repurposes SABench, a recent benchmark for spatial transcriptomics alignment methods, converting 40 of its 295 tasks into executable, hidden-evaluation tasks for coding agents.
• Task Definition: Agents are given two-dimensional tissue slices with observed spatial coordinates and gene-expression profiles. The goal is to submit a coordinate table aligning these slices.
• Evaluation: A hidden scorer evaluates submissions based on a schema check, expression agreement, label/landmark consistency, and (where available) alignment error against hidden gold-standard coordinates. Metrics are standardized to a 0–1 scale (higher is better).

Experimental Conditions:
The authors evaluated a single model configuration (Codex CLI with gpt-5.3-codex and xhigh reasoning effort) across three distinct execution conditions to test the impact of prompt richness and environment support:

1. Basic: The agent receives only the task prompt, with no paper-method context and no pre-bootstrapped environments.
2. Package-aware (PA): The prompt includes paper-method names and package suggestions. The agent must install and probe packages itself.
3. Full + prior (FPP): The agent receives pre-bootstrapped virtual environments with installed packages and high-level method priors (summarizing favored methods for related task families) in the prompt.

Analysis:
The study involved 120 runs (3 independent replicates per condition per task). Evaluation included:

• Aggregate Scoring: Comparing mean composite scores across conditions.
• Trace Inspection: Using an independent LLM call to extract failure stages, error messages, and package-use signals from run logs.
• Geometric Baselines: Synthesizing simple geometric heuristics (center/scale, PCA-similarity, trimmed ICP) derived from agent submissions to determine if agents were outperforming basic geometry.

Key Results

1. Increased Context Did Not Improve Performance
Contrary to the hypothesis that richer context would improve performance, the Basic condition achieved the highest mean composite score (0.43), followed closely by PA (0.42). The FPP condition, despite having the strongest package-use signals, achieved the lowest score (0.36).

• Statistical Significance: FPP underperformed Basic by −0.067 (95% CI [−0.11, −0.03]) and PA by −0.059.
• Package Usage: FPP agents utilized specialized packages (e.g., PASTE2, Spateo) in 43–72% of runs, whereas Basic agents rarely used them (<20%). Despite this behavioral shift, the aggregate accuracy declined.

2. Analysis of Failure Modes
Trace inspection revealed why increased package use led to lower scores:

• Competitive Custom Strategies: Under the Basic condition, agents often wrote custom Python code using simple geometric ideas (e.g., Procrustes rotation) that matched or exceeded package-backed solutions.
• Variance and Fragility: Invoking specialized packages required parameter tuning and dependency management. Poorly calibrated choices led to transformations worse than simple geometric alternatives.
• Infrastructure Failures: FPP runs suffered from longer package loops, environment failures, and "prior-driven" submissions where agents accepted suboptimal results after failed package calls rather than reverting to simpler approaches.
• Degradation of Good Inputs: In several cases (e.g., SA-004, SA-014), the input coordinates were already partially aligned. FPP agents actively degraded these inputs by applying poorly calibrated transformations, resulting in scores lower than the "Identity" baseline (submitting input unchanged).

3. Geometric Heuristics vs. Specialized Tools
Synthesized geometric baselines (center/scale, PCA-similarity, trimmed ICP) achieved scores comparable to or better than the FPP condition. The strongest geometric probe (trimmed ICP) averaged 0.41, nearly matching PA (0.41) and exceeding FPP (0.36). This suggests that agents in the Basic condition were effectively utilizing simple, robust heuristics that specialized packages failed to replicate reliably in this context.

4. Task-Specific Variability
No single package family was a universal winner. Rigid/affine custom geometry performed best on some serial-alignment families, while packages like Spateo or PASTE excelled in others. The benchmark rewarded local method selection over simply increasing package availability.

Significance and Claims

The paper claims that scientific coding agents are difficult to steer toward effective specialized tooling solely through prompting and environment provisioning.

• The "Alignment Gap": There is a disconnect between an agent's ability to use specialized tools and its ability to use them effectively. Adding scaffolding (package hints, pre-installed environments) often induces unnecessary transformations, fragile workflows, and infrastructure failures that lower overall performance.
• Evaluation Philosophy: The authors argue that scientific-agent benchmarks must evaluate agent traces and workflows in addition to final outputs. The "black box" of final scores hides the fact that agents may be falling back on surprisingly performant heuristics or failing due to brittle package integration.
• Benchmark Utility: The study demonstrates that benchmark papers can be repurposed as sources of executable, scorable tasks for AI agents, providing a concrete case study for evaluating method soundness in computational genomics.
• Modest Scope: The authors explicitly state they do not claim spatial alignment exhausts scientific agent evaluation. Instead, they view this as a case study illustrating how specialized tooling can alter agent behavior, necessitating future work to assess generalizability across other domains and model families.

In conclusion, the paper advocates for a shift from "hill-climbing" on aggregate metrics to a user-centric workflow where scientists inspect agent traces to understand how solutions are derived, ensuring that the scientific soundness of the approach is not compromised by the complexity of the tools employed.