🤖 machine learning

The Replay Gap: Static Evaluation of Model Switching in LLM Agents Scores the Wrong World

This paper demonstrates that static replay-based evaluation methods fundamentally fail to assess LLM agent routers because substituting model outputs in logged trajectories ignores the cascading divergence of subsequent agent actions, leading to misleading performance metrics that do not reflect real-world outcomes.

Original authors: Ashritha Gonuguntla

Published 2026-08-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ashritha Gonuguntla

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the captain of a spaceship navigating a treacherous asteroid field. You have a fleet of different AI pilots: some are fast and cheap but make small mistakes, while others are slow, expensive, and incredibly precise. To save fuel, you decide to switch pilots mid-journey based on what's happening right now. If the path is clear, you use the cheap pilot; if a massive asteroid appears, you swap to the expert. This is the dream of "agentic routing" in the world of Large Language Models (LLMs). These models are the brains behind AI agents that can write code, solve math problems, or play games by taking a series of steps. The big question for engineers is: How do we know our pilot-switching strategy actually works?

Traditionally, scientists have tested these strategies like a movie director reviewing a script. They look at a recorded journey (a "trajectory") where one pilot flew the whole way. Then, they pretend, "What if we had swapped the pilot at step 10?" They simply paste the new pilot's recorded answers into the old script and see if the ending looks good. This is called "replay evaluation." It's cheap, fast, and assumes that if you change the pilot at step 10, the rest of the ship's journey would have stayed exactly the same as the original recording. But in the real world, a spaceship is a closed loop: the pilot's action at step 10 changes the view at step 11, which changes the decision at step 12, and so on. If the new pilot sees a different asteroid field because of their own earlier move, the whole script changes. This paper asks a simple, terrifying question: Is our "movie script" method of testing actually measuring the wrong reality?

The researchers at Carnegie Mellon University decided to stop guessing and start testing. Instead of editing a movie script, they built a time-travel machine. They took real AI agents solving software engineering problems and, at specific moments, "forked" the timeline. They created two parallel universes: one where the original pilot continued, and another where they swapped in a different model. Crucially, they didn't just paste new answers into the old story; they let the new pilot actually take control, interact with the computer environment, and see what actually happened next. They ran about 900 of these parallel experiments to see how much the stories diverged.

The results were a shock to the system. The assumption that the rest of the journey stays the same was completely wrong. When they swapped models, the new pilot didn't just take over the existing path; they rewrote the entire future. In the early stages of a task, swapping models caused the agent to change its mind on 74% to 77% of its very next action. By the time the researchers looked at the whole journey, the new pilot had rewritten 61% to 94% of all the steps that followed the swap. To put it in perspective, if you swapped a pilot in a video game, the game world would change so drastically that the "script" you were trying to read would be describing a level that no longer exists.

The study also found that the "replay" method is dangerously blind to success. In their experiments, they saw five specific moments where swapping a pilot actually changed the outcome of the mission—saving a failed task or losing a solved one. The old "movie script" method missed every single one of these critical moments. It predicted failure when the new pilot actually succeeded, and it predicted success when the new pilot actually failed. The patch of code the new pilot actually wrote looked nothing like the patch the replay method predicted; they were essentially unrelated.

Furthermore, the researchers discovered that the "noise" in the system depends heavily on how the models are run. Even when using the exact same model twice, the results weren't always identical. If the model was served using one type of hardware compression (FP8), the "control" runs (where no swap happened) diverged 90% of the time. But with a different compression method (AWQ), the runs stayed nearly identical. This means that even the baseline for comparison is shaky, making the "replay" method even less reliable.

The paper concludes that the current way we test AI routing is scoring the wrong world. The "replay" method is like judging a chef by reading a recipe they wrote yesterday, ignoring the fact that the ingredients in the kitchen have changed. The author suggests that to truly understand how to route AI agents, we need to stop pretending the future is fixed and start testing by actually running the agents in branching, parallel realities. They have released their tools and data so others can stop guessing and start seeing the true, chaotic, and fascinating way these AI agents actually behave when the rules of the game change.

Technical Summary: The Replay Gap in Agentic Model Routing

Problem Statement
Model routing for Large Language Models (LLMs) has evolved from single-turn query selection to per-step decision-making within multi-step agents (e.g., software engineering, tool use). Current evaluation methodologies for these "agentic routers" rely on static replay: logging trajectories generated by a base model and simulating a router's decision by substituting a different model's recorded output at a specific step, assuming the remainder of the trajectory remains unaffected.

This paper argues that this assumption is fundamentally flawed for agents. Unlike single-turn queries, agent trajectories form a closed loop where an action at step $k$ directly determines the observation at step $k+1$ . Static replay implicitly treats the trajectory as an open loop, ignoring that a model switch alters the state distribution, leading to divergent future actions. The paper posits that replay-based benchmarks "score the wrong world" because the states they evaluate against never actually occur in the counterfactual scenario.

Methodology: Branching Rollouts
To measure the magnitude and structure of this divergence, the author introduces a branching-rollout protocol that moves beyond static replay to counterfactual evaluation:

Setup: Experiments utilize the mini-SWE-agent scaffold on the SWE-bench Verified dataset. The environment is a minimal bash-only ReAct loop with a 50-step budget and 28k-token context.
Models: Two models are used: a small model (S: Qwen3-4B-Instruct, FP8 quantized) and a large model (L: Qwen3-14B, AWQ quantized). Both are served by vLLM at temperature 0 on a single 24GB GPU.
Forking Protocol:
- A base trajectory is run to termination.
- At controlled fork points (30% and 70% of the trajectory length), the process forks into a fresh Docker container.
- Replay Fidelity: The pre-fork action prefix is re-executed in the new container to ensure the environment state matches the base trajectory (verified with 99.99% return-code agreement).
- Seeding: The message history is seeded with the recorded prefix so the branch model sees the exact same context.
- Arms: Each fork point generates two arms: a swap arm (switching to the other model) and a same-model control arm (continuing with the base model). The control arm isolates noise from sampling, batching, and environment replay, allowing the author to attribute divergence specifically to the model switch.
Scale: The study comprises six run pairs (varying difficulty and swap direction), yielding approximately 900 rollouts and 717 scored branch pairs.

Key Contributions

Methodology: A reproducible harness for counterfactual evaluation of per-step model switching via live branching, paired with same-model controls to isolate noise.
Quantifying the Replay Gap: Empirical evidence that model swaps cause immediate and significant divergence, rendering static replay invalid.
Structural Insights: Characterization of how divergence depends on fork depth (early vs. late) and swap direction (upgrade vs. downgrade).
Audit of Determinism: A demonstration that "temperature-0" determinism is configuration-dependent, with some serving stacks (FP8) introducing significant noise even in same-model controls.

Key Results

Action-Level Divergence: Swaps rewrite 61–94% of post-fork actions. The normalized edit distance between swap arms and matched controls ranges from +0.25 to +0.66 (95% CIs exclude zero).
Immediate Divergence: 74–77% of early swaps diverge at the very first post-fork action, compared to only 6–35% of control forks.
Invalidity of Replay States: For early swaps, a static replay evaluator would score 92–97% of post-fork decisions against states that never occur (only 3–8% of replayed states remain valid).
Outcome Flips: All five observed outcome flips (instances where the solution status changed due to the swap) occurred exclusively in swap arms. Zero flips occurred across 359 control forks. Specifically, upgrades rescued unsolved instances, and a downgrade caused the loss of the sole solved instance. Static replay failed to predict any of these success-relevant outcomes.
Patch Similarity: Patches predicted by a log-stitching replay evaluator had a similarity of 0.00–0.11 to the actual patches produced by the switched model.
Noise Floor: Even same-model controls are not perfectly deterministic. FP8-served controls diverged on 90–96% of forks (edit distance 0.49–0.67), whereas AWQ-served controls remained near-identical (edit distance 0.16–0.23).
Thoroughness Tax: Under tight budgets, the stronger model (14B) exhausted its step budget without submitting a solution more often than the weaker model (4B), suggesting a "thoroughness tax" that budget-aware routers must consider.

Significance and Claims
The paper claims that replay-based benchmarks score the wrong world for agentic routing. The structural blindness of static replay prevents it from observing outcome flips and evaluating decisions against valid states.

For Evaluation: The author asserts that agentic router benchmarks must adopt live or branched evaluation. They provide their harness and dataset as a starting point for validating off-policy estimators (e.g., importance sampling, doubly robust estimators) which are proposed as necessary corrections for the distribution shift.
For Routing Policies: The results suggest that fork position and direction are critical features: handoffs should occur late for downgrades (to inherit completed work) and early for upgrades (to capitalize on immediate re-decisions).
Modesty: The author explicitly states that their findings are based on a controlled pilot (one scaffold, specific quantization stacks, low resolution rates) and do not claim to demonstrate end-to-end router mis-ranking in "competent regimes" yet. They frame their work as establishing the invalidity of the inputs to replay evaluation rather than proving a specific routing strategy is superior. The paper concludes that the sign of the result (replay is wrong) is uniform, but the magnitude of the error in high-performance regimes requires further validation.

Technical Summary: The Replay Gap in Agentic Model Routing

More like this