Every-successful-replay route admission changes first-token latency rankings in clinical question answering

Technical Summary: Every-successful-replay route admission changes first-token latency rankings in clinical question answering

Problem Statement
Current benchmarks for clinical question answering (CQA) often measure latency based on the time to the first generated token without verifying whether the complete answer-and-evidence route satisfies the clinical evidence task. A route may appear fast because it omits provenance, utilizes out-of-window evidence, suppresses registered contradictions, or reuses stale dependencies. Crediting such routes in the same latency distribution as valid routes conflates evidence-validity failures with performance gains. The unresolved systems question is not merely how quickly a route runs, but which completed routes qualify for benchmark credit. Existing approaches to Clinical Retrieval-Augmented Generation (RAG), adaptive routing, and provenance systems typically protect different objects (e.g., retrieved context, generated output, or physical plans) at different times, failing to define a complete route as the unit of benchmark membership while retaining the cost of failed attempts.

Methodology
The authors introduce MedRouteGuard, a system that treats the complete answer-and-evidence route as the benchmark unit. The methodology is built on a four-clause admission contract and a retrospective evaluation framework:

1. Admission Architecture:

   • Pre-execution Admission: Candidates are screened before execution to ensure they can expose resolvable support, enforce the requested time scope, enumerate registered conflicts, and validate every reused dependency.
   • Post-execution Admission: Returned packets are validated for evidence alignment, claim-evidence consistency, conflict visibility, and a final freshness watermark.
   • Timing Protocol: The system uses a single clock across failed attempts and fallbacks. Failed routes do not restart the request clock; latency is measured from request acceptance to the first token of the route that eventually passes post-admission.

2. Retrospective Evaluation (Q3):

   • The study conducted an exhaustive replay of 2,541 question–route candidates for 847 clinician-authored questions.
   • A "retrospective oracle" was used to recompute latency rankings under two conditions: (a) enforcing the "every-successful-replay" rule (where a candidate is admitted only if all successful replays satisfy the four clauses) and (b) ignoring admission labels (allowing any executed route into the benchmark).
   • This design isolates the effect of route eligibility on latency rankings while holding the graph, prompts, workers, and cache state fixed.

3. Validation and Comparators:

   • Q1 (Output Quality): A blinded audit of 500 pairs compared MedRouteGuard against a text-only RAG baseline (TextRAG) for correctness, completeness, safety, and provenance.
   • Q2 (Deployed Performance): MedRouteGuard was compared against a matched fixed graph-serving comparator (OfflineGraph-Tuned) on the same hardware and evidence graph.
   • Independent Workload: A held-out medication-safety workload (326 questions) using time-versioned public evidence was used to test transferability and packet-level consequences.
   • Clinical Validity: A blinded review by board-certified specialists (320 routes) assessed the sensitivity and specificity of the admission labels against a clinical reference standard.

Key Contributions
The paper contributes three primary elements:

1. A Benchmark-Membership Estimand: A definition for complete clinical answer-and-evidence routes that requires satisfying a four-clause evidence contract (provenance, temporal validity, conflict preservation, and freshness) before receiving latency credit.
2. An Executable Architecture: A pre/post-admission system with dependency-aware reuse and a single-clock timing mechanism that retains the cost of failed attempts and fallbacks.
3. Empirical Evidence: Fixed-candidate, specialist, independent-workload, and matched-deployment evidence demonstrating that route eligibility changes both tail-latency rankings and the specific evidence packets selected for review.

Results

• Latency Ranking Sensitivity: Under the "every-successful-replay" rule, admission changed the request-to-first-generated-token winner for 61/847 (7.2%) of questions. The retrospective-oracle P95 latency shifted from 2.89 s (ignoring admission) to 3.24 s (enforcing admission), a +0.35 s increase.
• Independent Replication: In the independent medication-safety workload, admission changed winners for 21/326 (6.4%) of questions, with a P95 shift of +0.45 s.
• Clinical Validity: Specialist validation of admission labels achieved 92.3% sensitivity (rejecting clinically invalid routes) and 93.3% specificity (admitting valid routes).
• Error Reduction: In the 14/180 reviewed questions where membership changed the frozen output, admission-based selection reduced material evidence-validity errors from 13/14 to 2/14 (an absolute reduction of 78.6 percentage points) and unsafe omissions from 10/14 to 1/14 (a reduction of 64.3 points).
• Deployed Performance: Compared to the matched fixed graph-serving comparator, MedRouteGuard reduced first-generated-token P95 from 4.18 s to 3.24 s and complete-answer P95 from 8.06 s to 6.82 s, while maintaining 98.9% answered coverage.
• Output Quality: In the blinded 500-pair audit, MedRouteGuard showed higher mean scores for correctness, completeness, safety, and actionability compared to the text-only RAG baseline, with an overall acceptability rate of 94.2% versus 87.4%.

Significance and Claims
The paper claims that the "every-successful-replay" rule exposes a measurable denominator error in current first-token benchmarks. By crediting routes that omit provenance or use stale evidence, benchmarks can artificially inflate performance metrics. The study demonstrates that enforcing a complete route membership rule:

1. Changes the Benchmark: It alters the tail-latency distribution and the identity of the "fastest" route, particularly for safety, temporal, and aggregation questions.
2. Improves Clinical Safety: It selects evidence packets with significantly fewer material validity errors and unsafe omissions.
3. Maintains Efficiency: Despite the stricter admission criteria, the system achieves faster recorded processing milestones (first-token and complete-answer) compared to a matched fixed-policy baseline, proving that strict evidence contracts are compatible with high performance.

The authors conclude that for mutable clinical evidence services, route eligibility must be specified before latency credit is assigned to prevent benchmarks from crediting evidence omission, temporal mismatch, conflict suppression, or stale reuse as speed. The work operationalizes this boundary through pre-execution capability checks, post-execution validation, and same-clock timing across failed attempts.