Beyond Continuity: Challenges of Context Switching in Multi-Turn Dialogue with LLMs

Technical Summary: Beyond Continuity: Challenges of Context Switching in Multi-Turn Dialogue with LLMs

Problem Statement
In real-world multi-turn conversations, users frequently pivot to new topics or refine existing ones without explicitly starting a new session. However, Large Language Models (LLMs) often exhibit a "continuity bias," carrying over stale, irrelevant context from previous turns even when the user intent has shifted. This leads to inaccurate responses and a failure to reset context appropriately. While prior work has addressed long-context management and query rewriting, the specific capability of out-of-the-box LLMs to detect topic shifts (pivots) and reset context in a zero-shot setting remains understudied. This paper investigates the inherent ability of LLMs to distinguish between a user pivoting to a new topic and refining the current topic, and their subsequent ability to select only the relevant prior turns for the current response.

Methodology
The authors construct a stress-test framework using synthetic benchmarks derived from two real-world datasets: TopiOCQA (knowledge-seeking, entity-centric QA) and MSC (Multi-Session Chat, persona-based chit-chat). To create controlled "hard switches," the authors concatenate head segments of different conversations, establishing ground-truth labels for every user turn as either PIVOT or REFINE, along with a ground-truth shortlist of relevant prior turn indices.

The study evaluates three experimental variants:

1. V1 (Hard Concatenation): Conversations are joined without any bridging text, creating a difficult, implicit pivot.
2. V2 (Cue-Bridged): A lexical cue phrase (e.g., "switching gears") is inserted at the pivot turn to provide an explicit signal.
3. V3 (Position Study): A single pivot is inserted at varying positions (turns 2–20) within a session to analyze position bias.

The authors evaluate ten LLMs across three categories: open-weight non-reasoning, closed-source non-reasoning, and reasoning models. The evaluation is conducted in a zero-shot setting, prompting models to output a JSON object containing a predicted label (PIVOT/REFINE) and a list of relevant prior turn indices.

Key Contributions

• Formalization of Pivot Detection and Context Resetting: The paper formalizes the task as a dual problem: (1) classifying the user's intent (pivot vs. refine) and (2) selecting the correct subset of prior turns.
• New Evaluation Metric (Over-Carry Rate): The authors introduce the Over-Carry (OC) metric to quantify context carry-over errors specifically at pivot turns. Unlike standard F1 scores, OC directly measures a model's ability to "reset" its retrieval policy when a topic switch occurs (i.e., the predicted shortlist should be empty for a pivot).
• Comprehensive Zero-Shot Benchmark: The study provides one of the first direct comparisons of diverse LLM families (open/closed, reasoning/non-reasoning) on pivot detection and context resetting without task-specific fine-tuning.
• Analysis of Failure Modes: The paper identifies specific failure patterns, including "always-carry" behaviors in open-weight models, conservative resetting in others, and significant position bias where later pivots are harder to detect.

Results
The evaluation reveals distinct performance tiers and failure modes:

• Reasoning Models: Models with explicit reasoning capabilities (e.g., o3, Claude-3.7-sonnet, Gemini-2.5-pro) demonstrate superior performance. They achieve high F1_PIVOT scores and near-zero OC, indicating they effectively "gate" context and drop irrelevant information at topic switches.
• Closed-Source Non-Reasoning Models: Models like gpt-4o perform well on knowledge-seeking tasks (TopiOCQA) with low over-carry but struggle more on persona-based chat (MSC), exhibiting higher OC. gpt-4o-mini shows strong detection but suffers from dramatic over-carry, revealing a strong continuity bias.
• Open-Weight Models: These models generally struggle. Some (e.g., Gemma-3, Llama-3.1) show reasonable detection but high OC, indicating a misalignment between their classification head and context selection mechanism (they acknowledge a pivot but still carry stale context). Others (e.g., Mistral-7.2B) underperform on both metrics. Phi-4 shows very low OC but poor detection, suggesting an overly aggressive reset policy that misses actual pivots.
• Position Bias: A clear trend emerges where F1_PIVOT degrades as the pivot position increases (later in the conversation). This indicates that as the conversation history grows, models become less reliable at detecting shifts, a phenomenon exacerbated in non-reasoning models.
• Impact of Cues: The presence of explicit cue phrases (V2) improves performance across most models, particularly for those with strong instruction-following capabilities, suggesting that lexical boundaries act as effective triggers for context resetting.

Significance and Claims
The paper claims that while some advanced LLMs (specifically reasoning and strongly instructed models) possess the inherent capability to detect pivots and reset context, many state-of-the-art open-weight models fail to do so reliably without fine-tuning. The authors argue that the "stickiness" of stale context is a critical failure mode for long-term multi-turn robustness.

The study concludes that improving multi-turn capabilities requires addressing specific gaps:

1. Targeted Fine-tuning: To align classification heads with context selection mechanisms.
2. Two-Stage Pipelining: Separating pivot detection from context selection.
3. Position-Aware Decaying: Mitigating the degradation of performance as conversation length increases.
4. Inference-Time Guards: Implementing rules to enforce context resets at detected boundaries.

The authors position this work as a diagnostic lens for understanding the inherent limitations of out-of-the-box LLMs in dynamic dialogue, emphasizing that current models often prioritize coherence over context discipline, leading to errors in real-world applications where topic shifts are frequent.