Technical Summary: Clarification Is Not Enough: Post-Clarification Answering Remains the Bottleneck in Multi-Turn QA

Problem Statement

While Large Language Models (LLMs) have achieved fluency in conversational tasks, they struggle with under-specified language, a common feature of human communication where users omit key details or rely on shared contextual assumptions. In multi-turn Question Answering (QA), this creates a tension: an assistant must decide whether to answer directly (risking generic or incorrect responses) or to ask clarifying questions (risking user annoyance).

Existing research has largely focused on training models to ask clarifying questions. However, it remains unclear whether the primary bottleneck in end-to-end multi-turn performance lies in the decision to clarify (the clarification policy) or in the ability to answer correctly once the missing information is provided (post-clarification answering). This paper posits that understanding and correctly interpreting the user's response to a clarifying question is the critical, yet under-addressed, gap in current systems.

Methodology

The authors decompose the multi-turn QA problem into two distinct components to isolate performance bottlenecks:

1. Clarification Policy: The decision to either ask a clarifying question or answer directly.
2. Post-Clarification Answering: The generation of the correct final answer after the user provides the missing information.

Experimental Setup

• Dataset: The study utilizes PACIFIC, a benchmark of naturally occurring multi-turn financial QA interactions containing both ambiguous and unambiguous user turns. This dataset allows for the evaluation of both the decision to clarify and the correctness of the final answer.
• Models: Two instruction-tuned models were evaluated: Meta-Llama-3-8B-Instruct and Qwen2.5-7B-Instruct.
• Training: The authors performed Supervised Fine-Tuning (SFT) on these models using full-parameter fine-tuning with DeepSpeed ZeRO-3.
• Evaluation Metrics:
  • Action-Level Metrics: Measure whether the model chooses the correct next action (Ask vs. Answer). Metrics include Action Accuracy, Weighted F1, and Macro F1.
  • Final-Answer Metrics: Measure the correctness of the final output. Crucially, the authors report Final Acc (cond.), which measures final answer accuracy conditioned on the model taking the correct action. This metric isolates failures in post-clarification answering from failures in the clarification policy.
• Simulation: GPT-4o was used to simulate user responses to model-generated clarifying questions and to act as a binary judge for the semantic correctness of final answers.

Additional Analyses

The authors conducted experiments on synthetic data augmentation, varying the number of augmented turns, the clarification rate (proportion of clarification-seeking turns), and the inclusion of reasoning traces generated by GPT-4o.

Key Results

1. SFT Rapidly Improves Clarification Policy

Supervised Fine-Tuning yields substantial improvements in action-level performance.

• Action Accuracy: SFT improved Action Accuracy by over 10 points for both models.
• Ambiguity Handling: Before SFT, models defaulted to answering directly, resulting in near-zero accuracy on ambiguous turns. After SFT, action accuracy on ambiguous turns rose significantly (to 0.68 for LLaMA and 0.71 for Qwen) while maintaining high accuracy on unambiguous turns.
• Conclusion: Models can quickly learn when to ask for clarification.

2. Post-Clarification Answering Remains the Bottleneck

Despite improved decision-making, the ability to generate the correct final answer after clarification remains significantly weaker.

• The Gap: While Final Acc (overall) improved, Final Acc (cond.) showed much more modest gains (e.g., +0.05 for LLaMA, +0.10 for Qwen).
• Specific Performance: On ambiguous turns, even when the model took the correct action (asking for clarification), the final answer accuracy remained low (e.g., 0.28 for LLaMA and 0.18 for Qwen).
• Interpretation: The gap between high action accuracy and low conditional final answer accuracy indicates that the primary failure mode is not deciding to clarify, but rather integrating the user's disambiguating information with the dialogue context to produce the correct answer.

3. Limitations of Synthetic Augmentation

• Turn Augmentation: Adding synthetic conversational turns provided only modest gains that quickly saturated.
• Clarification Rate: Varying the proportion of clarification turns in synthetic data had limited impact on performance.
• Reasoning Traces: Naive augmentation with reasoning traces during training actually degraded performance. However, providing reasoning traces at inference time to the fine-tuned model resulted in substantial improvements, suggesting the models can utilize reasoning if guided, but struggle to learn it solely from noisy synthetic traces.

Significance and Claims

The paper claims that the field of multi-turn QA has reached a point where better clarification policies alone are insufficient for solving under-specified queries.

• Primary Contribution: The study explicitly identifies post-clarification answering as the main remaining bottleneck. The ability to correctly interpret a user's response to a clarifying question and synthesize it with context is a harder problem than deciding to ask the question in the first place.
• Implication for Future Work: Progress in multi-turn QA requires a shift in focus from solely optimizing the "ask vs. answer" decision to improving the model's capability to combine user feedback with dialogue history to generate accurate final answers.
• Modesty: The authors acknowledge that their findings are based on the PACIFIC dataset (financial domain, table-based, extractive/calculation answers). While the interaction between clarification behavior and answer correctness is a generalizable concept, the specific forms of ambiguity and reasoning may differ in open-ended generative tasks.

In summary, the paper argues that while LLMs can be effectively taught to recognize ambiguity and ask the right questions, they currently lack the robustness to fully resolve that ambiguity and deliver the correct answer, making post-clarification answering the critical frontier for future research.