🤖 machine learning

ContinuityBench: A Benchmark and Systems Study of Stateful Failover in Multi-Provider LLM Routing

This paper introduces ContinuityBench, a benchmark and stateful proxy architecture that utilizes a history-forwarding strategy to achieve near-perfect conversational continuity during multi-provider LLM failover events, addressing the critical limitation of stateless systems that discard conversation history.

Original authors: Vishal Pandey, Gopal Singh

Published 2026-07-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Vishal Pandey, Gopal Singh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are talking to a very smart, friendly voice assistant. You've been chatting for ten minutes, sharing your favorite movie, your weird dream about a flying toaster, and the secret code to your imaginary treehouse. Suddenly, the voice assistant's brain gets a little dizzy and needs to swap to a backup brain to keep talking. In the world of computer science, this is called "failover." Usually, engineers just make sure the new brain answers the question immediately. But here's the catch: the new brain has no idea who you are or what you just said. It's like if you walked into a new room, started a conversation, and the person you were talking to suddenly forgot your name and asked, "Who are you again?" You'd have to start your whole story over from the beginning. This paper, written by researchers at Metriqual, dives into this specific problem of "conversational continuity." It asks a simple but crucial question: When a computer system switches from one AI provider to another during an outage, does it actually remember the conversation, or does it just pretend to be alive while having amnesia?

The researchers discovered that the standard way of doing things is broken. Most systems today are "stateless," meaning they treat every single message as a brand-new, isolated event. If the main AI provider crashes, the system instantly switches to a backup, but it only sends the backup the very last sentence you typed. It throws away the entire history of your chat. The paper argues that this is a disaster for user experience. Even if the system is technically "up" and running, the conversation is dead because the context is gone. To prove this, the authors built a new testing tool called ContinuityBench. They created 150 fake conversations where they planted secret "fact anchors"—like a specific date, a made-up name, or a favorite food—early in the chat. Then, they simulated a crash right before the user asked a question about that secret fact. They compared two systems: the old "stateless" way and a new "stateful" way they designed, which they call History-Forwarding.

The results were dramatic. The old system failed completely. In all 750 test crashes they simulated, the backup AI remembered 0% of the context. It was as if the conversation had never happened. The user would ask, "What is my secret code?" and the new AI would honestly say, "I don't know, you haven't told me." However, the new History-Forwarding system was a total game-changer. Instead of just sending the last message, this system grabbed the entire conversation history and handed it to the backup AI like a complete storybook. This new method achieved a 99.20% success rate in preserving the context. In the rare cases where it failed (about 6 times out of 750), it wasn't because the system forgot to send the history; it was just because the backup AI model itself made a small mistake in following instructions.

The paper also tackled some tricky engineering nightmares that happen when you try to do this with hundreds of people talking at once. They found that if you aren't careful, the system can accidentally mix up two different people's conversations, giving Person A the secrets of Person B. They also discovered that if the backup AI gets too busy, a simple "retry" button can cause a "thundering herd" problem, where thousands of requests crash the backup server all at once. To fix this, they used a technique called "exponential backoff with jitter," which is like telling a crowd of people to wait a random amount of time before trying to push through a door, rather than all pushing at the exact same second.

In short, the paper proves that keeping a conversation alive during a computer crash isn't just about keeping the lights on; it's about keeping the memory alive. By forwarding the full history to the backup, they showed that you can maintain a seamless, continuous chat with 99.20% reliability, with almost no extra delay for the user. They've even released their testing tool, continuity-bench, to the public so other engineers can build systems that don't just answer questions, but actually remember the story.

Technical Summary: ContinuityBench

Problem Statement

In production Large Language Model (LLM) deployments, high API availability does not guarantee conversational continuity. Current multi-provider gateways (e.g., LiteLLM, Portkey, OpenRouter) operate on a stateless, request-level abstraction. When a primary provider fails or hits rate limits, these systems successfully reroute the request to a fallback provider to maintain uptime. However, they discard the preceding conversation history. Consequently, the fallback model receives only the most recent user message, devoid of context.

This creates a "continuity gap": the system returns a valid HTTP 200 response (availability), but the user experience is shattered because the agent forgets prior turns, forcing users to repeat themselves or causing agentic pipelines to fail on multi-step tasks due to truncated context. This failure mode is pervasive, consequential, and currently lacks rigorous measurement or standardized mitigation.

Methodology

The authors introduce ContinuityBench, an open evaluation harness designed to stress-test context preservation under controlled failover conditions.

Evaluation Design

Test Suite: A synthetic dataset of $N=150$ multi-turn conversations (7–11 turns each). Each conversation embeds a specific "factual anchor" (e.g., a name, date, or preference) in the first turn, followed by unrelated filler dialogue, and concludes with a probe question that can only be answered correctly if the model recalls the anchor.
Fault Injection: A deterministic fault injector simulates three failure modes (TIMEOUT, API_ERROR, RATE_LIMIT) at the final probe turn. This ensures the fallback provider must answer the context-dependent question immediately after the primary fails.
Scoring (LLM-as-Judge): An automated judge (GPT-4o) evaluates whether the fallback provider's response correctly incorporates the factual anchor. The judge is calibrated against human labels and uses a fast-path heuristic for exact substring matches.
Concurrency: Experiments are conducted at $C=100$ concurrent conversations to stress-test race conditions and rate-limit handling.
Replication: The study consists of 5 independent runs, totaling $N=750$ failover events per system.

Systems Compared

The study compares two architectures sharing identical infrastructure, differing only in the payload forwarded during failover:

Baseline (Stateless): The proxy extracts only the last user message and forwards it to the fallback provider.
Treatment (History-Forwarding): The proxy forwards the complete messages[] array (full conversation history) to the fallback provider.

Key Contributions

Formalization of Continuity: The paper defines conversational continuity as a property orthogonal to availability, distinct from simple uptime metrics.
New Metrics:
- Continuity Preservation Rate (CPR): The fraction of failover events where the fallback model successfully accesses the full preceding context.
- Continuity Latency Overhead (CLO): The additional latency cost incurred by reconstructing and forwarding state.
Stateful Proxy Architecture: A design utilizing a "History-Forwarding" strategy to reconstruct conversational state across heterogeneous endpoints.
Open Benchmark: Release of continuity-bench, a reproducible harness for evaluating multi-provider systems.
Systems Failure Characterization: Identification of two critical failure modes in high-concurrency failover scenarios:
- Concurrency Race Conditions: Shared state corruption where parallel failover events mix conversation histories.
- Retry Storms (Thundering Herd): Naive fixed-interval retry logic against rate-limited fallbacks creates a self-sustaining request storm that locks out the fallback endpoint.

Results

Continuity Preservation: The History-Forwarding treatment achieved a CPR of 99.20% [95% Wilson CI: 98.27%, 99.63%] across 750 events. The stateless baseline achieved 0.00%. The six treatment failures were attributed to the fallback model's instruction-following limitations, not the forwarding mechanism.
Latency Overhead: The median Continuity Latency Overhead (CLO) was -450 ms, indicating the treatment was often marginally faster than the baseline due to variance in third-party API response times. The mean CLO was +59 ms. The P95 CLO was +13,614 ms, driven by the time required for the fallback model to process large context payloads (9–11 turns).
Stability: CPR remained stable (98.7%–99.3%) across all five independent runs.
Concurrency Fixes: The study demonstrated that exponential backoff with jitter is necessary and sufficient to prevent retry storms against strict-rate-limit fallbacks, and that strict per-session state isolation is required to prevent context bleeding.

Significance and Claims

The paper argues that the industry's current focus on "availability" (returning an answer) conflates two distinct properties. A robust multi-model inference system must preserve continuity (returning an answer informed by the full context).

The authors claim that:

Stateless failover is insufficient for interactive voice agents and agentic pipelines, as it functionally resets the conversation.
History-Forwarding is a viable solution, achieving near-perfect continuity preservation with negligible median latency overhead.
Reliability is a distributed systems problem. Simply routing requests is not enough; production gateways must implement explicit concurrency controls (to prevent state corruption) and intelligent backoff protocols (to prevent retry storms) when managing stateful failover.

The work provides a principled foundation for building state-preserving multi-model systems, shifting the reliability paradigm from "request survival" to "conversation survival."