💻 computer science

QuoteBench: How Matched Scores Can Hide Command-Path Failures

The paper introduces QuoteBench, a benchmark demonstrating that matched execution scores in LLM coding agents often mask significant command-generation failures caused by execution-path serialization and parsing, revealing that model rankings and performance are heavily dependent on specific deployment configurations rather than intrinsic capabilities.

Original authors: Shangao Li, Yao Zhang, Volker Tresp, Yuanyuan Yang

Published 2026-08-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shangao Li, Yao Zhang, Volker Tresp, Yuanyuan Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a robot chef. You give it a recipe written in plain English, and it's supposed to turn that into a set of precise instructions for a kitchen machine to cook a meal. In the world of artificial intelligence, these "robots" are Large Language Models (LLMs), and the "kitchen machine" is a computer system that runs code. For a long time, scientists have been testing these robots by seeing if the final meal tastes good. If the robot says, "Chop the onions," and the computer chops the onions, everyone cheers.

But there's a tricky part: how the message gets from the robot to the machine. Sometimes, the message travels through a translator, a wrapper, or a middleman that rewrites the sentence before the machine sees it. Think of it like sending a letter. If you write "Don't forget the salt!" and a middleman reads it, gets confused by the punctuation, and accidentally changes it to "Don't forget the salt!" (with a different meaning), the machine might add too much salt. The problem is, if you only check the final dish, you might think the robot is perfect, even though the middleman messed up the instructions. This paper asks a simple but vital question: Is the robot actually good at writing instructions, or is it just lucky that the middleman didn't ruin them?

This is where QuoteBench comes in. The researchers created a special test to see if AI models can write Bash commands (which are like the specific, technical instructions computers use to manage files and folders) without getting tripped up by "quoting" rules. In computer language, quotes and special symbols are like traffic signs; if you miss one, the car crashes. The team found that many AI models look amazing on paper but fail miserably when their instructions have to pass through a "middleman" (like a remote server or a container) that re-reads the text.

Here is the twist: The researchers discovered that the standard way of testing these AIs is hiding a massive problem. They took the exact same instructions an AI wrote and ran them through two different paths. In the first path, the instructions went straight to the computer. In the second, they went through a "middleman" that added an extra layer of reading (like putting the instructions inside double quotes).

When they did this, the results were shocking. For some of the top AI models, the success rate dropped by a huge amount—between 55.4% and 73.2%—just because the instructions had to pass through that extra layer. It's like a robot chef who can perfectly chop onions when you hand them the knife directly, but if you ask them to shout the instructions through a megaphone, they forget how to hold the knife.

However, the story gets even more interesting. The researchers found that if they told the AI in advance, "Hey, your instructions are going to go through a middleman," the smartest models could actually fix their own mistakes. They would rewrite their instructions to be extra careful. When they did this, they recovered 30.4% to 60.7% of the lost points.

This leads to a confusing situation. If you just look at the final score, one model might look almost perfect, hiding the fact that it lost 64.3% of its potential success due to the middleman, only to gain 60.7% back because it learned to adapt. The paper calls this a "matched gap" of just −3.6 points. It looks like a tiny difference, but it's actually a massive rollercoaster of failure and recovery that the standard test completely missed.

The paper argues that we can't just look at a single "success score" to judge an AI. That score is like a grade on a test that doesn't tell you if the student used unauthorized help or if the teacher helped them. The researchers show that depending on how you set up the test (the "command path"), the ranking of which AI is "best" can completely flip. For example, one model might be the clear winner in a direct test, but fall behind when the instructions have to go through a remote server.

In short, the paper proves that the way we test AI is broken because it ignores the journey the instructions take. It's not enough to know if the AI can write a command; we need to know if that command survives the trip to the computer. The researchers suggest that anyone building or buying these AI tools needs to stop looking at simple scores and start asking: "How did the instructions get there? Was there a middleman? Did the AI know it was coming?" Because without those answers, a high score might just be an illusion.

Technical Summary: QuoteBench – How Matched Scores Can Hide Command-Path Failures

1. Problem Statement

Large Language Model (LLM) coding agents often issue Bash commands through interfaces that serialize, wrap, and reparse model output before execution. Current benchmarks typically evaluate "matched" scores, where the model's generation and the execution environment are aligned (e.g., a raw shell script executed directly). However, these scores fail to distinguish between errors generated by the model and failures introduced by the execution transport layer (e.g., double-quoting in ssh host "..." or docker exec sh -c "...").

Bash quoting failures can corrupt literals, break routine agent actions, and trigger unnecessary repair loops. Existing benchmarks either conflate command construction with planning and recovery capabilities or score emitted programs under a fixed transport, obscuring whether a command preserves its payload when deployed in real-world scenarios involving nested parsers or remote execution.

2. Methodology: The QuoteBench Benchmark

2.1 Task Design

QuoteBench introduces a benchmark of 56 one-shot Bash tasks derived from 14 operation families (e.g., writing files, heredocs, JSON manipulation, Git operations, and SSH-like simulations).

Structure: Each family includes one benign control task and three hazardous payload variants containing quotes, backticks, dollar signs, newlines, and glob characters.
Validation: Unlike benchmarks that rely on exit codes (which can be misleading as 23.4–47.0% of failures exit with code 0), QuoteBench uses exact final-state validators. These check file bytes, argument vectors, JSON parsing, directory states, or Git history.
Scope: The benchmark focuses on POSIX/Bash command construction, excluding PowerShell, network failures, and multi-turn recovery.

2.2 Crossed Experimental Design

The core innovation is a 2×2 crossed design that independently varies the Generation Contract and the Execution Transport:

Generation Contracts:
- Raw (R): The model emits a Bash program executed verbatim via bash -c.
- Disclosed-Boundary (N): The model is explicitly told its reply will be interpolated into bash -c "R".
Execution Transports:
- Raw (R): Direct execution of the reply.
- Nested (N): The reply is re-parsed through an added double-quoted parser (simulating remote/CI execution).

This design creates four conditions for every stored model reply:

RR: Raw contract on Raw transport (Baseline).
RN: Raw contract on Nested transport (Measures Transport Damage).
NR: Disclosed contract on Raw transport (Measures cost of over-escaping).
NN: Disclosed contract on Nested transport (Matched score).

2.3 Decomposition of Scores

The paper decomposes the conventional "matched gap" ($NN - RR$) into two distinct components:
$(NN - RR) = \underbrace{(RN - RR)}_{\text{Transport Damage}} + \underbrace{(NN - RN)}_{\text{Contract-Conditioned Compensation}}$

Transport Damage: The loss in success when a fixed raw reply is re-parsed by a downstream parser.
Compensation: The recovery in success when the model is explicitly informed of the boundary and adapts its generation.

3. Key Results

3.1 Transport Damage is Severe and Universal

Replaying fixed raw-generated replies through an added double-quoted parser causes a catastrophic drop in performance across all tested configurations.

Magnitude: Success rates drop by 55.4 to 73.2 percentage points in every same-window configuration.
Scope: This damage affects even benign control tasks (losing 28.6–57.1 points), as models often emit double-quote-active characters even for simple commands.
Retention: Only 25.0–35.4% of direct-path successes survive the nested transport.

3.2 Matched Scores Can Mask Large Opposing Effects

Conventional matched evaluations ($NN$ vs. $RR$) often show small differences, hiding the underlying volatility.

Example (GPT-5.6-sol): The matched gap is only −3.6 points. However, this masks a −64.3 point transport damage that is almost entirely offset by +60.7 points of contract-conditioned compensation.
Rank Reversals: The ranking of models changes significantly depending on the path. For instance, GPT-5.6-sol trails Gemini-3.5-Flash by one task on the raw path but leads by 18 tasks on the nested path.

3.3 Compensation is Model-Dependent and Contract-Specific

Adaptation: Six of eight configurations recover 30.4–60.7 points when the boundary is disclosed. Two configurations (Qwen3.5-27B and Gemini-3.1-Flash-Lite) show zero or negative compensation, failing to adapt even when the boundary is disclosed.
Specificity: Compensation is not generic robustness; it is conditioned on the declared grammar. Models that adapt to a double-quote wrapper often fail on a single-quote wrapper, indicating they are not applying a universal fix but adapting to the specific contract.
Effort vs. Path: Increasing "effort" (token budget) improves matched scores for some models but does not necessarily improve the portability of the command across paths. The nested-replay pass rate of raw-conditioned replies remains stable (varying by at most 5.4 points) across effort rungs.

3.4 Robustness of the Mechanism

Payload Independence: The damage persists across public and private hostile payloads.
Environment Independence: The effect holds across GNU/Linux and BSD/macOS userlands, though minor dialect-specific shifts occur.
Fixes: Two trivial fixes—correct escaping at the interpolation point or using a temporary script—completely eliminate the transport damage, confirming the failure is in the transport layer, not the model's fundamental capability.

4. Significance and Claims

The paper argues that matched execution scores are not intrinsic model properties but are heavily dependent on the deployment configuration (generation contract and execution transport).

Decomposition of Failure: QuoteBench provides the first method to separate generation errors from transport-induced failures. It demonstrates that a high matched score can be the sum of a large transport loss and a large compensatory generation, rendering the aggregate score misleading for deployment planning.
Evaluation Protocol: The authors claim that evaluations of command-issuing agents must report the generation contract, execution path, operating point, and final-state validator. Treating a matched score as a standalone metric obscures critical portability risks.
System Design Implications: The command interface is part of the evaluated system, not neutral plumbing. Vendors should disclose how their tools handle boundaries (e.g., escaping), and users should select models based on their specific deployment path rather than raw benchmark scores.
Modest Scope: The paper explicitly limits its claims to one-shot Bash generation under quoting hazards. It does not claim to measure general coding capability, multi-turn recovery, or other shell environments (e.g., PowerShell), nor does it estimate the prevalence of these failures in the wild, only their mechanism and impact within the benchmark.

In conclusion, QuoteBench reveals that "reliability" in LLM agents is often an illusion created by a specific, favorable transport path, and that true robustness requires explicit handling of the command path boundary.