Technical Summary: What Process Evaluation of Coding Agents Actually Measures

1. Problem Statement

Current evaluations of Large Language Model (LLM) coding agents increasingly focus on process—how an agent executes a task—rather than just the final outcome. However, the paper argues that existing process-level evaluations are conceptually underspecified. They often conflate three distinct levels of analysis:

1. Action Level: Predicting the agent's next operation (e.g., "What tool will it use next?").
2. Task Level: Assessing the uncertainty or recoverability of the overall execution (e.g., "Is this task still solvable?").
3. Step Level: Attributing causal contribution to specific actions (e.g., "Did this specific step cause the failure?").

The paper posits that treating these as a single notion of "process quality" leads to misinterpretation. Specifically, standard methods like full-trace LLM judges or process reward models often condition on information (like future steps or final outcomes) that invalidates causal claims about individual steps. The core problem is determining what these evaluation signals actually measure and whether they can reliably identify step-level causal effects.

2. Methodology: The SCAE Framework

To address this, the authors introduce a measurement framework grounded in a Structural Causal Model (SCM) of agent execution. They instantiate this with SCAE (Step-level Causal Attribution Estimator), a replay-based estimator designed to measure process evaluation at the three distinct levels.

2.1 Structural Causal Model

The agent's execution is modeled as:

• Exogenous Variables: Latent task difficulty ($Z$), decoding noise ($G^{(t)}_\pi$), and environment noise ($U^{(t)}_{env}$).
• Policy: Defined via a Gumbel-max formulation, where actions are selected by perturbing log-probabilities.
• Outcome: A verifiable terminal outcome (e.g., recall of a gold file set).

2.2 Identification and Estimation

The goal is to estimate the prefix-conditioned causal effect of replacing an action $a$ with $a'$ at step $t$:
$$\theta_t(a, a') = E[Y_{A_t=a'} - Y_{A_t=a} \mid \tau_{1:t-1}]$$
where $\tau_{1:t-1}$ is the realized execution prefix.

Key Assumptions:

1. Context Observability: The state is a deterministic function of the issue and the logged prefix.
2. Exogenous Decoding Noise: Decoding noise is independent of task difficulty and future noise.
3. Replayable Environment: The environment can be deterministically reproduced from the logged state and action.

Hybrid Estimation Strategy (SCAE):
Since coding agents often exhibit deterministic behavior (violating the "positivity" assumption required for purely observational estimation), SCAE uses a hybrid approach:

• Observational Branch: If replaying the prefix yields diverse next actions (high entropy), the effect is estimated by averaging outcomes over multiple replays.
• Intervention Branch: If the policy is deterministic (low entropy), the system substitutes an alternative action, executes it in the real environment to get a genuine tool output, and then replays the rest of the trajectory. This avoids fabricating tool outputs.

2.3 Measurement Levels

The framework produces three types of measurements from the same fitted object:

• Action Level: Characterizes what predicts the next operation.
• Task Level: Measures execution uncertainty via the variance of replayed continuations.
• Step Level: Estimates the "anytime value" increments ($\Delta_t$) to attribute causal contribution.

3. Experimental Setup

• Task: File localization (identifying files to modify for a given issue), derived from SWE-bench-Verified.
• Data: 499 episodes across 12 repositories (dominated by django).
• Protocol: The agent is rolled out once. Selected prefixes are reconstructed, and continuations are replayed (8 times per point) to estimate value functions and uncertainty.
• Evaluation: The study focuses on measurement reliability rather than agent performance improvement.

4. Key Results

4.1 Action Level: Provenance vs. Graph Structure

• Finding: The strongest predictor of an agent's next action is its execution provenance (the paths and files it has just seen in tool outputs), not the code dependency graph.
• Evidence: On "first-visit" steps (where the target file hasn't been seen before), a predictor using only the most recent tool output achieved a top-3 accuracy of 0.326. In contrast, a strengthened code-dependency graph predictor achieved only 0.058.
• Implication: Agents navigate based on immediate observational context rather than global repository structure.

4.2 Task Level: Uncertainty is Global, Not Local

• Finding: Execution uncertainty is structured primarily at the task (instance) level, not the step level.
• Evidence: Variance in replayed outcomes was strongly explained by the specific instance (ICC = 0.640) but not by step position or tool type. Furthermore, 0 out of 190 estimated step-level effects survived False Discovery Rate (FDR) correction.
• Implication: Whether an execution succeeds or fails is largely determined by the difficulty of the specific task instance, not by the cumulative effect of individual steps. This makes step-level causal attribution empirically infeasible at feasible replay costs.

4.3 Step Level: Collider Bias in Full-Trace Judges

• Finding: Observational attribution methods, particularly full-trace LLM judges, suffer from collider bias.
• Evidence: When downstream steps were visible to the judge, the "blamed" step shifted significantly later in the trajectory (median normalized position 0.88) compared to when downstream steps were hidden (median normalized position 0.17). The mean displacement was +0.537 later in normalized position.
• Implication: Full-trace judges do not measure certified causal contribution. Instead, they measure semantic relevance enriched by downstream evidence. They select steps that are semantically related to the failure (e.g., touching a gold file) but are biased toward the end of the trajectory because they condition on descendants of the action.

5. Contributions and Significance

The paper's primary contribution is a measurement framework that disentangles action prediction, task uncertainty, and step attribution.

• Conceptual Clarification: It establishes that "process evaluation" is not a single problem. Signals useful for predicting the next action (provenance) or characterizing task difficulty (instance variance) should not be conflated with step-level causal attribution.
• Methodological Rigor: By introducing SCAE, the authors provide a principled way to test what process evaluators can and cannot identify. They demonstrate that formal identifiability does not guarantee empirical measurability when uncertainty resides at the task level.
• Critique of Current Practices: The paper provides empirical evidence that current process evaluators (like full-trace judges) are systematically biased by downstream information, measuring semantic relevance rather than causal contribution.

Limitations and Scope:
The authors are modest about their claims. They acknowledge that:

• The findings are specific to file localization, a setting chosen for its replayability and verifiable outcomes.
• Step-level causal attribution remains empirically weak (underpowered) in this setting, not necessarily because it doesn't exist, but because the uncertainty is too global to be localized at feasible computational costs.
• The uncertainty decomposition (task vs. policy vs. model) is validated on synthetic data, not fully on real traces.

Conclusion:
The paper concludes that process evaluation should not be treated as a monolithic benchmark. Instead, evaluators must be validated against the specific level they measure. Action predictors can be strong, task uncertainty can be structured, but step-level causal localization may remain infeasible in practical settings. Recognizing these distinct levels is essential for interpreting process signals correctly.