Can Small Language Models Handle Context-Summarized Multi-Turn Customer-Service QA? A Synthetic Data-Driven Comparative Evaluation

1. Problem Statement

Customer-service Question Answering (QA) systems require high contextual continuity and dialogue management across multiple turns. While Large Language Models (LLMs) excel in these tasks, their deployment is often hindered by:

• High Computational Costs: Significant latency and infrastructure requirements.
• Privacy Concerns: Reliance on cloud APIs for proprietary models raises data governance issues, especially with sensitive customer data (PII).
• Resource Constraints: Many organizations require on-premise or edge deployment, which is difficult for massive models.

Small Language Models (SLMs) (typically <10B parameters) offer a more efficient alternative but remain underexplored for multi-turn customer-service scenarios. Existing research focuses largely on single-turn QA or lacks public, English-language benchmarks for multi-turn interactions with context summarization. Furthermore, evaluation practices are often inconsistent, relying solely on automatic metrics or isolated human assessments without analyzing performance across different conversation stages.

2. Methodology

The study employs a rigorous experimental framework involving synthetic data generation, model fine-tuning, and a multi-faceted evaluation strategy.

A. Synthetic Data Construction Pipeline

To address the lack of public multi-turn customer-service datasets (due to privacy), the authors constructed a custom dataset:

1. Source: Started with the TalkMap Banking Conversation Corpus (single-turn pairs).
2. Aggregation: Converted isolated turns into structured multi-turn dialogues (5–100 turns).
3. Context Summarization: Used GPT-4o-mini to summarize prior conversational history into concise representations. The summary preserves critical entities (names, account IDs, dates, status) while removing noise, enabling SLMs to maintain context within limited token windows.
4. Response Refinement: Used GPT-4.1 to refine agent responses for naturalness, clarity, and coherence, ensuring high-quality ground truth for training.
5. Dataset Split: 183,337 total conversations split into Training (70%), Validation (10%), and Test (20%).

B. Model Selection and Training

• SLMs Evaluated: Nine instruction-tuned SLMs ranging from 1B to 8B parameters, including:
  • 1-2B: Qwen-3-1.7B, LLaMA-3.2-1B.
  • 3-4B: Qwen-3-4B, Phi-4-Mini, LLaMA-3.2-3B, Gemma-3-4B, SmolLM3-3B.
  • 8B: Qwen-3-8B, LLaMA-3.1-8B.
  • Note: Reasoning capabilities (e.g., "thinking" tags) were explicitly disabled to ensure fair comparison of instruction-following.
• Fine-Tuning: All SLMs were fine-tuned using QLoRA (4-bit quantization + Low-Rank Adaptation) on NVIDIA RTX A100 GPUs.
• Baselines: Three commercial LLMs were used for comparison: GPT-4.1, Gemini-2.5-Flash, and Virtuoso-Large (based on Qwen-2.5-72B).

C. Evaluation Framework

The study introduces a Conversation Stage-Based Analysis, categorizing interactions into Early (issue identification), Mid (core exchange), and Late (resolution/closure) stages.

1. Quantitative Metrics: ROUGE-L, METEOR, BERTScore, BARTScore, and Cosine Similarity.
2. LLM-as-a-Judge: Used Claude Sonnet 4.5 to score responses on a 1–5 Likert scale across four dimensions: Human-Likeness, Continuity/Context Understanding, Tone/Clarity, and Task Appropriateness.
3. Human Evaluation: Three human evaluators assessed a subset of models (3-4B range + LLMs) using the same dimensions.
4. Pairwise Evaluation: Direct comparison of SLM vs. LLM responses using Claude Haiku 4.5 to determine win/loss/tie rates.

3. Key Contributions

1. Systematic Evaluation: A comprehensive benchmark of fine-tuned, instruction-tuned SLMs for context-summarized multi-turn customer-service QA, a previously underexplored area.
2. Synthetic Data Pipeline: A novel pipeline for generating privacy-preserving, multi-turn training data that integrates context summarization and response refinement.
3. Stage-Wise Analysis: A methodological contribution that segments evaluation into Early, Mid, and Late conversation stages, revealing specific strengths and weaknesses of models at different interaction phases.
4. Comparative Insights: A detailed comparison showing that specific fine-tuned SLMs can rival or outperform commercial LLMs in specific domains and stages.

4. Key Results

Quantitative Performance

• Top SLMs: Qwen-3-4B-Instruct and LLaMA-3.1-8B-Instruct achieved the highest scores in lexical and semantic similarity (e.g., Qwen-3-4B: ROUGE-L 0.3959, BERTScore 0.9137).
• Outperforming LLMs: Surprisingly, several fine-tuned SLMs outperformed all three commercial LLMs (GPT-4.1, Gemini-2.5-Flash, Virtuoso-Large) on automatic metrics, suggesting that domain-specific fine-tuning is more critical than raw parameter count for this specific task.

Qualitative & Stage-Wise Performance

• Human-Likeness & Tone: Fine-tuned SLMs (e.g., LLaMA-3.2-3B, Qwen-3-4B) scored competitively with LLMs in human-likeness and tone.
• Context Continuity: While strong, SLMs showed slightly lower scores in "Continuity and Context Understanding" compared to top-tier LLMs, indicating moderate gaps in long-range coherence.
• Stage-Based Insights:
  • Late-Stage: SLMs demonstrated exceptional performance in resolution-focused turns, often scoring higher than Gemini-2.5-Flash.
  • Mid-Stage: Top SLMs remained competitive with LLMs in core information exchange.
  • Early-Stage: Performance was generally consistent across models for issue identification.
• Pairwise Wins:
  • Against Gemini-2.5-Flash, top SLMs (Qwen-3-8B, LLaMA-3.1-8B) achieved win rates of 52.9% – 55.8%.
  • Against GPT-4.1, SLMs struggled (win rates <25%), though they performed better in late-stage resolutions.
  • Model Variance: Not all SLMs performed well; SmolLM3-3B and Gemma-3-4B showed persistent limitations in dialogue continuity and task appropriateness, highlighting that architecture matters more than just parameter count.

5. Significance and Conclusion

This study demonstrates that Small Language Models, when properly fine-tuned and equipped with context summarization strategies, are viable and efficient alternatives to massive LLMs for customer-service QA.

• Practical Impact: Organizations can deploy 3-8B parameter models on-premise or on standard hardware, significantly reducing computational costs and latency while maintaining high-quality, context-aware responses.
• Privacy: The synthetic data approach allows for robust training without exposing sensitive real-world customer data.
• Future Directions: The authors suggest extending this to other domains (healthcare, telecom) and exploring Reinforcement Learning from Human/AI Feedback (RLHF/RLAIF) to further improve dialogue continuity and tone control.

In summary, the paper argues that for specific, domain-constrained tasks like customer service, specialized fine-tuning of SLMs can bridge the performance gap with LLMs, offering a scalable, cost-effective, and privacy-compliant solution.