Technical Summary: MT-OSC

Problem Statement

Large Language Models (LLMs) exhibit significant performance degradation when task instructions and critical context are fragmented across multiple conversational turns. While single-turn benchmarks show strong results, real-world dialogue naturally distributes information, clarifications, and intents over several exchanges. This fragmentation creates two primary challenges:

1. Information Dispersion: LLMs struggle to accurately recall and integrate crucial details scattered across turns, leading to inconsistencies and errors compared to single-message queries.
2. Context Window Exhaustion: The standard approach of concatenating full chat history into prompts rapidly consumes context windows. As conversations lengthen, this leads to increased latency, higher computational costs, and diminishing returns, often forcing systems to truncate history or fail.

Existing solutions, such as ad-hoc summarization, risk omitting essential context needed for complex reasoning. Other methods like retrieval-augmented dialogues or hierarchical compression often require model fine-tuning, complex selection schemes, or introduce latency via real-time retrieval, limiting their scalability and ease of integration.

Methodology: MT-OSC Framework

The authors propose MT-OSC (Multi-Turn One-off Sequential Condensation), a framework designed to automatically condense chat history in the background without disrupting the user experience or requiring model retraining. The system operates via three core components:

1. Condenser Agent

The Condenser is an LLM-based agent (using Llama-3.3-70B-Instruct in experiments) that employs a few-shot inference-based approach. Unlike generic summarization, it is tuned with curated exemplars that demonstrate specific condensation policies:

• Retaining key user text verbatim for future reference.
• Consolidating comprehensive information.
• Safely excluding irrelevant turns.
  The agent outputs a condensed user-assistant pair $(C_{ju}, C_{ja})$ along with reasoning for the strategy used. This allows the model to adapt dynamically to diverse dialogue patterns observed in public datasets.

2. Decider Module

To prevent information loss in dense or repetitive dialogues, a lightweight Decider component adaptively determines whether to apply condensation. It employs a rule-based function to quantify redundancy and novelty:

• It represents user messages and assistant responses as sets of normalized content words.
• It calculates the overlap between assistant turns relative to prior user queries.
• Logic: If the average word overlap between assistant turns exceeds a threshold $\gamma$ (e.g., 0.2) and the total user content tokens exceed a threshold $\tau$ (e.g., 1000) across a window of $w$ turns, condensation is withheld ($D_w = 0$). Otherwise, condensation is applied ($D_w = 1$).
  This safeguard ensures that information-rich or highly repetitive conversations retain necessary fidelity.

3. One-off Sequential Condensation Strategy

MT-OSC processes conversations sequentially in a sliding window of size $w$ (e.g., $w=4$).

• As the conversation progresses, once $w$ turns accumulate, the Condenser generates a condensed representation of that window.
• This condensed history replaces the raw history for subsequent inputs, while the most recent turns remain in their original form.
• The process repeats as new turns are added, ensuring the model input always contains a compact, contextually salient history.
• Asynchronous Execution: The condensation runs as a background process. Since the output is not required for the immediate next turn (it is used for the next-to-next turn), it adds no latency to the user experience.

Key Contributions

1. Task-Agnostic Framework: MT-OSC achieves up to 72% token reduction in 10-turn dialogues without disrupting user experience, operating entirely in the background without model modifications or fine-tuning.
2. Robust Condensation Agent: The framework introduces a Condenser Agent using curated few-shot examples to preserve nuanced context, coupled with a configurable Decider to minimize information loss in dense dialogues.
3. Scalable Evaluation: Extensive experiments across 13 state-of-the-art LLMs and 10 diverse multi-turn benchmarks demonstrate the framework's broad applicability and robustness against distractions and redundant turns.

Results

The evaluation utilized 10 datasets, including sharded versions of standard benchmarks (GSM8K, BFCL, HumanEval, Spider, ToTTo, Summary of Haystack) and MT-EVAL datasets (Recollection, Expansion, Follow-up, Refinement).

• Performance Preservation and Improvement: Across 13 LLMs, MT-OSC consistently narrowed the multi-turn performance gap. In many cases (e.g., GSM8K, BFCL, HumanEval), it improved accuracy compared to the baseline of passing full chat history. For instance, on the Sharded BFCL dataset, MT-OSC improved accuracy from 81.13% to 86.79% while reducing tokens.
• Token Reduction: MT-OSC reduced chat history tokens by an average of 41% across datasets, with reductions reaching 72% in 10-turn dialogues. For conversations with 6+ turns, the average reduction was 44.56%.
• Robustness: The framework remained resilient when tested against synthetic noise, including repetition infusion, filler injection, and contextual diversion. MT-OSC often showed similar or higher performance gains on noisy data compared to clean data, effectively mitigating the impact of irrelevant information.
• Latency and Cost: By reducing prompt length, MT-OSC lowers Time To First Token (TTFT). The authors estimate a latency reduction of approximately 1.1–1.2 seconds for chats with 4,000 tokens, alongside significant operational cost savings.
• Ablation Studies:
  • Removing the Decider led to performance drops on complex datasets, confirming its role in safeguarding information.
  • Replacing the Condenser with simple summarization prompts resulted in lower success rates, highlighting the necessity of the curated few-shot approach for preserving critical instructions and context.

Significance

The paper positions MT-OSC as a practical, scalable solution to the multi-turn context bottleneck. It addresses the trade-off between context retention and efficiency by enabling richer context within constrained input spaces. Unlike prior work that relies on complex architectures or manual intervention, MT-OSC offers a drop-in solution that balances performance, latency, and cost. The authors claim it enables broader applications of conversational AI in chat systems by allowing models to handle extended, fragmented dialogues without the performance degradation typically associated with long contexts.

Limitations Noted: The study relies on datasets with fewer than 12 turns and primarily clean, single-topic dialogues. The authors note that real-world conversations often involve greater linguistic complexity, topic diversity, and agent-driven tool use, which represent areas for future exploration.