TurnWise: The Gap between Single- and Multi-turn Language Model Capabilities

This paper introduces the TurnWiseEval benchmark and the TurnWiseData synthetic pipeline to address the gap between single- and multi-turn language model capabilities, demonstrating that incorporating as few as 10,000 multi-turn conversations during post-training significantly improves multi-turn performance.

The Gist

Imagine you are hiring a new employee for a job that requires talking to customers all day.

The Problem: The "One-Question" Interview
Right now, most companies (and AI researchers) are hiring these "employees" (Large Language Models) based on a very strange interview process. They ask the candidate a single question, get an answer, and then immediately stop. They never ask a follow-up. They never say, "That's interesting, but can you tell me more?" or "Wait, I meant something slightly different."

The paper calls this the Single-Turn Gap. It's like hiring a chef based only on how well they can make a single omelet, but then expecting them to run a busy restaurant where customers keep changing their orders, asking for substitutions, and having long conversations about the menu.

The authors found that even the smartest AI models (like GPT-5) get confused when the conversation gets long. They forget the context, lose their train of thought, or just give a worse answer than they would have if you had just asked the question once.

The Solution: A New Test and a New Training Method

To fix this, the authors created two things:

1. The New Test: "TURNWISEEVAL" (The "Same Question, Different Format" Test)

Instead of just giving the AI a hard conversation and seeing if it passes, they created a fair comparison.

• The Analogy: Imagine you ask a student, "What is the capital of France?" (Single Turn). Then, you ask them, "I'm planning a trip. First, tell me the capital of France. Then, tell me the best time to visit. Finally, suggest a hotel." (Multi-Turn).
• The Trick: The authors take the exact same core question and ask it in both ways. They then compare the AI's answer in the long conversation against its answer in the short one.
• The Goal: If the AI gives a great answer to the short question but a terrible, confused answer to the long one, the test reveals a specific weakness: it's bad at keeping a conversation going. It proves the AI isn't "dumb," it just gets lost in the flow of chat.

2. The New Training: "TURNWISEDATA" (The "Imaginary Friend" Method)

The big problem is that real, long conversations with humans are hard to get. You can't just ask 10,000 people to chat with a robot for free.

So, the authors invented a way to make fake but realistic conversations using a "synthetic pipeline."

• The Analogy: Think of it like a writer's room.
  1. They start with a single prompt (the seed).
  2. They use a super-smart AI to pretend to be a user who is unsatisfied or curious. This AI says, "That's not quite what I meant, try again," or "Oh, by the way, what about X?"
  3. They stack these fake user turns on top of each other to create a long, messy conversation.
  4. They feed this fake conversation to the student AI (like Olmo 3) to teach it how to handle the "back-and-forth."

The Results: A Little Practice Goes a Long Way
The authors tested this on an open-source AI model called Olmo 3.

• Before: The model was great at answering single questions but fell apart in long chats.
• The Experiment: They took the model and gave it just 10,000 of these fake, multi-turn conversations to study. That's a tiny amount of data compared to the billions of words the model usually sees.
• The Result: It was like a magic trick. The model's ability to handle long conversations jumped by 12%. It learned to remember what was said earlier and keep the conversation on track.

The Big Takeaway
The paper concludes that being good at a single question and being good at a long conversation are two different skills.

Just because an AI is smart doesn't mean it's a good conversationalist. To build AI that feels like a real human friend, we can't just train it on one-off questions. We need to give it practice in the "gym" of long, messy, multi-turn chats. And the good news? We don't need real humans to do the training; we can generate these practice sessions ourselves, and even a little bit of practice makes a huge difference.

Technical Summary

1. Problem Statement

Despite the prevalence of multi-turn conversations in real-world Large Language Model (LLM) applications, current training and evaluation paradigms are heavily skewed toward single-turn settings. This creates a significant mismatch between model capabilities and actual usage.

• Evaluation Gap: Existing benchmarks (e.g., MT-Bench, MTEval) are becoming saturated and often fail to distinguish whether poor performance in multi-turn settings stems from a lack of conversational ability or general task difficulty. They do not isolate the specific degradation caused by the multi-turn context.
• Training Gap: Open-source model recipes (e.g., Olmo) primarily rely on single-turn data. When multi-turn data is included, it is often done without systematic analysis of its contribution to conversational skills.
• Data Scarcity: Collecting high-quality, scalable multi-turn data is difficult due to the challenges of realistic user simulation and the high cost of manual collection.

2. Methodology

The authors propose a two-pronged approach: a new evaluation benchmark to measure the gap and a synthetic data pipeline to bridge it.

A. TURNWISEEVAL (Evaluation Benchmark)

This benchmark is designed to isolate multi-turn conversational ability from general task performance. It uses pairwise comparisons between a model's responses in a multi-turn setting and a baseline.

• Design: The benchmark generates multi-turn conversations (up to 8 user turns) based on seed instructions from AlpacaEval.
• Two Evaluation Modes:
  1. TURNWISEEVAL-Absolute: Compares the evaluated model's multi-turn response against a strong reference model (GPT-4) responding to the equivalent single-turn prompt. This measures absolute multi-turn performance relative to a high-quality single-turn baseline.
  2. TURNWISEEVAL-Self: Compares the evaluated model's multi-turn response against its own response to the equivalent single-turn prompt. This isolates the specific degradation (or improvement) caused by the multi-turn context, independent of the model's general capability.
• Metric: Win-rate determined by an LLM judge (GPT-4.1), using length-controlled comparisons.

B. TURNWISEDATA (Synthetic Data Pipeline)

To address the lack of training data, the authors introduce a scalable method for generating synthetic multi-turn conversations from single-turn seeds.

• Process:
  1. Seed Selection: Randomly select single-turn prompts (from the WildChat subset of Dolci Instruct).
  2. User Turn Generation: Independently generate 2–8 user turns using a strong model (GPT-4.1). These turns are designed to be topically related but self-contained, simulating user behaviors like seeking clarification, iterating on dissatisfaction, or exploring related topics.
  3. Structure: The original seed prompt is placed at the end of the conversation. This preserves the original dataset's preference pairs and ensures the assistant's final response addresses the core instruction while leveraging context.
• Comparison: The authors compare this method against "Self-talk" (where the model plays both user and assistant), finding TURNWISEDATA produces higher quality data.

3. Key Contributions

1. TURNWISEEVAL Benchmark: A novel evaluation framework that quantifies the "multi-/single-turn gap" by controlling for task difficulty and general model ability through pairwise comparisons.
2. TURNWISEDATA Pipeline: A scalable, cost-effective method for generating high-quality synthetic multi-turn training data without relying on expensive user simulation.
3. Empirical Analysis of the Gap: Demonstration that even frontier models (e.g., GPT-5 Chat) and open models (e.g., Olmo 3) exhibit significant performance drops in multi-turn settings compared to single-turn settings.
4. Training Insights: Evidence that training with even a small amount of synthetic multi-turn data (e.g., 10k conversations) significantly narrows the performance gap.

4. Key Results

Experiments were conducted using Olmo 3 7B Instruct as the base model.

• The Gap Exists:
  • Open models (Olmo 3) showed a large performance gap (e.g., -7.7% difference between single-turn AlpacaEval and multi-turn TURNWISEEVAL-Absolute).
  • Even advanced models like GPT-5 Chat underperformed in multi-turn settings compared to their single-turn baseline (-5.0% gap).
• Training Effectiveness:
  • Fine-Tuning (SFT): Adding 10k synthetic multi-turn conversations improved the TURNWISEEVAL-Self score by 12.8 points.
  • Preference Tuning (DPO): Adding 10k synthetic conversations improved the score by 9.2 points.
  • Data Efficiency: Significant gains were achieved even when multi-turn data comprised less than 5% of the total training mixture.
  • Data Quality: Models trained on pure TURNWISEDATA outperformed those trained on a mix of TURNWISEDATA and "Self-talk" data, suggesting TURNWISEDATA is a higher-quality signal.
• Stability:
  • Preference Tuning: Improved multi-turn capabilities without degrading single-turn instruction following (IFEval) or knowledge (MMLU) scores.
  • Supervised Fine-Tuning (SFT): While effective for multi-turn, SFT with synthetic data slightly degraded single-turn knowledge benchmarks, likely due to sensitivity to the generator's quality.
• Context Length: Training with multi-turn data flattened the performance decay curve, allowing models to maintain high win-rates even as conversation length increased to 8 turns.

5. Significance and Conclusion

• Distinct Capability Dimension: The paper establishes that multi-turn conversational ability is a distinct dimension of LLM capability that is not captured by traditional single-turn evaluations.
• Training Imperative: The results demonstrate that multi-turn data is vital for achieving robust chat performance. Relying solely on single-turn data leaves a critical capability gap.
• Scalable Solution: The TURNWISEDATA pipeline offers a practical solution for the open-source community to generate necessary training data without the prohibitive costs of human annotation or complex user simulation.
• Recommendation: The authors advocate for the systematic inclusion of multi-turn data in open model training recipes and the adoption of benchmarks like TURNWISEEVAL to better assess conversational robustness.