Layer-Isolated Evaluation: Gating the Deterministic Scaffold of a Production LLM Agent with a No-LLM, Regression-Locked Test Harness

Technical Summary: Layer-Isolated Evaluation of Production LLM Agents

Problem Statement
Current evaluation paradigms for Large Language Model (LLM) agents rely heavily on end-to-end task-success metrics. While these aggregate numbers indicate whether an agent has regressed, they fail to diagnose where the failure occurred within the agent's complex architecture (e.g., intent resolution, planning, safety validation, or memory). Identifying the specific broken subsystem requires re-running stochastic, live agent episodes and bisecting changes, a process that is slow, noisy, and expensive. Furthermore, aggregate metrics often "mask" localized regressions; a significant failure in a single component can be diluted by the success of the rest of the system, rendering the drop in the overall pass-rate indistinguishable from normal variance.

Methodology
The authors propose Layer-Isolated Evaluation, a deterministic, no-LLM testing harness designed for a production food-and-beverage ordering agent. The methodology involves three core components:

1. Architectural Decomposition: The agent is decomposed into a fixed taxonomy of 23 architectural layers (e.g., ontology pre-resolution, intent signals, routing, decomposition, escalation, safety, memory, and cross-cutting defense).
2. Pure-Mode Assertion Slices: Each layer is exercised by a dedicated "assertion slice" that runs in a deterministic, no-LLM "pure" mode. Instead of invoking a generative model, these slices assert contracts on deterministic outputs (e.g., canonical IDs from an ontology resolver, rule-based escalation decisions, dictionary rewrites, or server-side price recalculations).
   • The full suite consists of 238 baseline cases (225 pure cases + 13 legacy end-to-end cases).
   • Execution time is approximately 2.39 seconds (≈10 ms per case), enabling it to run on every Pull Request (PR) in Continuous Integration (CI).
3. Coverage-Honesty Baseline: The system employs a "locked per-slice baseline." Crucially, it enforces a coverage-honesty criterion: if a layer has zero test cases, its pass-rate is reported as null (uncovered) rather than 100%. This prevents a "green" aggregate score from hiding untested components.

Validation and Results
The authors validated the method through controlled regression injection, degrading one non-safety layer at a time across seven distinct layers and measuring the impact on both the aggregate pass-rate and individual slice rates.

• Masking Phenomenon: The study confirmed that aggregate metrics mask local failures. When a single layer was degraded, the aggregate pass-rate dropped only marginally (between -1.7 and -5.9 percentage points), a change small enough to be dismissed as dashboard noise.
• Localization: In contrast, the specific slice corresponding to the injected fault "cratered," dropping between -25 and -91 percentage points.
• Fault Localization Accuracy: The injected layer's own slice was the single worst-hit slice in 5 out of 7 cases and ranked in the top 3 worst-hit slices in all 7 cases (mean rank 1.29 out of 19). The "off-diagonal" impact (damage to other slices) was near-flat, except for foundational layers (like ontology resolution) where a small drop in the foundational layer caused downstream slices to fail, correctly signaling a foundational issue.
• Cross-Tenant Replication: The localization results were replicated on a second, structurally different tenant (Starbucks SG). All seven matching slices cratered under their respective injections, confirming the method is not an artifact of a single catalog.

Key Contributions
The paper claims the following specific contributions:

1. A Deterministic Harness: A fully decomposed, sub-second, no-LLM per-layer harness for a production agent that gates CI against a locked baseline.
2. Coverage-Honesty Criterion: A test-adequacy rule that refuses to score an unexercised layer as passing, ensuring that "green" aggregate metrics cannot launder untested risks.
3. Empirical Validation of Masking: A regression-injection demonstration proving that per-slice baseline-locked gates can localize regressions that aggregate metrics completely miss.
4. Operationalization of Theory: The work provides a concrete, runnable instantiation of the "evaluation-driven agent operations" process model prescribed by Xia et al. (2024), which had previously lacked a full implementation.

Significance and Claims
The authors position this work not as an invention of component-level evaluation, but as a concrete, deterministic instantiation of existing theoretical frameworks. They explicitly state:

• The method is a drift detector, not a correctness oracle. A green slice certifies that the layer has not changed from the baselined contract, not that the contract itself is "correct."
• The approach complements, rather than replaces, end-to-end stochastic testing. While the pure mode gates the deterministic core of the agent, stochastic whole-workflow mutation testing (e.g., Bhardwaj, 2026) remains necessary for generative behaviors.
• The primary value is development-time signal: it transforms the slow, noisy process of finding broken subsystems into a fast, deterministic, and actionable CI gate.

The paper concludes that while the injection study is limited to author-chosen faults (N=7) and a single domain (F&B ordering), the demonstrated ability to isolate faults that aggregate metrics hide offers a robust mechanism for maintaining agent stability during rapid iteration.