⚡ electrical engineering

Call2Instruct: Automated Pipeline for Generating Q&A Datasets from Call Center Recordings for LLM Fine-Tuning

This paper introduces Call2Instruct, an automated end-to-end pipeline that transforms noisy call center audio recordings into high-quality Q&A instructional datasets for LLM fine-tuning through a multi-stage process of audio processing, text cleaning, and semantic matching, successfully demonstrated by fine-tuning a Llama 2 7B model.

Original authors: Alex Echeverria, Sávio Salvarino Teles de Oliveira, Fernando Marques Federson

Published 2026-01-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Alex Echeverria, Sávio Salvarino Teles de Oliveira, Fernando Marques Federson

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive library of old, dusty audio recordings from a busy call center. These recordings are full of real people talking, but they are messy: there's static, people talking over each other, automated menus, and the conversations are unstructured. You want to teach a super-smart computer (a Large Language Model, or LLM) how to answer customer questions based on these recordings, but the computer can't learn from raw, noisy audio. It needs a clean, organized textbook of "Questions" and "Answers."

The paper "Call2Instruct" describes a robotic factory line (an automated pipeline) built to take that messy audio and turn it into a perfect textbook for the computer.

Here is how the factory works, step-by-step:

1. The Sound Cleaning Station (Audio Processing)

First, the raw audio goes into a cleaning station.

The Problem: The recordings are like a noisy room where two people are talking, but you can't tell who is who, and there's a lot of background static.
The Fix: The system acts like a sound engineer. It separates the voices (so it knows which parts are the customer and which are the agent), removes the static noise, and cuts out the boring automated menus (like "Press 1 for sales").
The Result: It then uses a "super-listener" (a speech-to-text tool) to turn the clean audio into a rough draft of text.

2. The Editor's Desk (Text Cleaning & Privacy)

Now the system has a rough transcript, but it's still messy. It might have weird typos, repeated words ("um, um, um"), or sensitive info like real names and account numbers.

The Problem: Computers get confused by messy text, and it's illegal to share private customer data.
The Fix: An automated editor steps in. It fixes punctuation, removes the "umms" and "ahhs," and acts like a privacy guard. It swaps out real names for placeholders like <NAME> or <ACCOUNT_ID>, ensuring no one's secrets are leaked.
The Result: A clean, safe, and easy-to-read story of the conversation.

3. The Librarian with a Magic Map (Semantic Extraction)

This is the smartest part of the factory. The system needs to figure out: "What did the customer actually want?" and "How did the agent fix it?"

The Problem: In a real conversation, a customer might ramble for five minutes before asking a simple question, and the agent's answer might be buried in the middle of the chat.
The Fix: The system uses a "magic map" (vector embeddings). It translates the customer's rambling into a clear, direct question (e.g., "How do I reset my password?"). It does the same for the agent's answer. It then turns these questions and answers into mathematical coordinates on a map.
The Result: Now, the system can find the perfect match. Even if the customer asked about "resetting" and the agent talked about "restarting," the magic map knows they are the same thing and pairs them up.

4. The Textbook Writer (Dataset Generation)

Now that the system has matched the right questions to the right answers, it needs to format them for the computer to learn.

The Problem: The computer needs to learn in a specific format: "Here is an instruction, here is the answer."
The Fix: The system acts like a textbook author. It takes the matched pairs and writes them into a perfect format. It might add a little instruction like, "Based on the customer's request, what is the solution?" followed by the agent's answer.
The Result: A brand-new, high-quality dataset ready for training.

5. The Final Exam (Validation)

To make sure the factory actually works, the authors built a test.

The Test: They took the new dataset and used it to teach a computer model (specifically, a model called Llama 2 7B).
The Outcome: The computer successfully learned from the data. When they asked it simulated customer questions, it gave answers that made sense for that specific call center. This proved the pipeline works: it successfully turned messy audio into a useful training tool.

The Bottom Line

The paper concludes that this "Call2Instruct" pipeline is a working solution. It takes the chaotic, unstructured reality of call center recordings and automates the entire process of turning them into a clean, private, and structured dataset. This allows companies to train AI assistants that are actually good at handling customer service, using their own real-world data without needing humans to manually type out thousands of questions and answers.

The authors have also made the code for this factory available to the public so others can build upon it.

Technical Summary of Call2Instruct

Problem Statement
The adaptation of Large Language Models (LLMs) to specific domains, particularly for customer service, relies heavily on high-quality, instruction-format (Question-Answer) fine-tuning datasets. While call centers possess vast amounts of valuable, real-world conversational data, this information exists in unstructured audio formats. Transforming these noisy, disorganized recordings into structured instructional datasets presents a significant challenge. Existing approaches often address isolated stages—such as Automatic Speech Recognition (ASR), text normalization, or semantic extraction—independently, lacking an integrated, end-to-end automated pipeline capable of converting raw call center audio directly into LLM-ready Q&A pairs.

Methodology: The Call2Instruct Pipeline
The paper proposes "Call2Instruct," an automated, five-stage pipeline designed to transform raw call center audio into instructional Q&A datasets. The process flows as follows:

Audio Acquisition and Preprocessing:
- Raw audio is retrieved from storage (e.g., AWS buckets) and standardized.
- Channel Separation & Denoising: Speaker diarization isolates agent and customer speech (though in the specific case study, channels were pre-separated). A denoising framework mitigates background noise.
- Segment Filtering: An unsupervised approach using K-Means clustering on acoustic features identifies and removes non-human segments, such as Interactive Voice Response (IVR) or automated attendant messages.
- Transcription: Preprocessed audio is converted to text using ASR systems leveraging OpenAI's Whisper Large model (optimized with LoRA and quantization).
Textual Preprocessing and Cleaning:
- Raw transcripts undergo normalization, including punctuation restoration and the correction of ASR artifacts (e.g., hallucinations in silent segments).
- Anonymization: Personally Identifiable Information (PII) is detected and masked using placeholder tokens (e.g., <NAME>, <ACCOUNT_ID>) to ensure compliance with data protection regulations like GDPR and LGPD.
Semantic Information Extraction and Vectorization:
- Demand Rewriting: Customer utterances are processed via an LLM (ChatGPT) to rewrite them into objective, direct questions, removing oral markers and prolixity.
- Vectorization: Both the rewritten customer demands and the corresponding agent responses are converted into dense vector embeddings using the text-embedding-ada-002 model.
- Storage: These vectors, along with metadata (speaker persona, original file ID), are stored in a vector database (Elasticsearch) to facilitate semantic search.
Question-Answer (Q&A) Dataset Generation:
- Semantic Matching: For each rewritten customer demand, the system performs a semantic similarity search against the entire corpus of agent responses to retrieve the most relevant answers, regardless of the original interaction context.
- Refinement and Synthesis: The top $N$ candidate responses (where $N=3$ ) are individually refined by an LLM to improve clarity and objectivity. Subsequently, these refined candidates are synthesized by the LLM into a single, coherent, and comprehensive answer that aggregates relevant information while discarding noise.
- Formatting: The final pairs are structured into an instruction-tuning format (e.g., "Based on the customer's query, what was the recommended solution?").
Validation:
- The pipeline includes automatic checks for coherence and privacy leakage.
- Functional Validation: The generated dataset is used to perform Instruct Fine-Tuning on a pre-trained LLM (Llama 2 7B) using the Lamini platform to verify the data's utility.

Key Contributions

End-to-End Automation: The paper presents a complete, automated workflow that bridges the gap between raw, noisy call center audio and structured, instruction-ready datasets, addressing a critical bottleneck in domain-specific LLM training.
Semantic Aggregation: Unlike simple transcription-to-Q&A methods, the pipeline utilizes semantic search to cross-reference demands with the entire corpus of agent responses, followed by LLM-based synthesis to create high-quality, comprehensive answers.
Privacy-Preserving Processing: The integration of robust anonymization steps ensures that sensitive customer data is handled compliantly before model training.
Open Source: The authors have made the developed code publicly available to promote reproducibility and future research.

Results
The pipeline was successfully implemented and tested using over 3,000 call center recordings from the telecommunications sector.

Dataset Generation: The system successfully processed the audio, cleaned and anonymized the text, and generated a structured Q&A dataset.
Functional Validation: A fine-tuning experiment was conducted on the Llama 2 7B model. The resulting model demonstrated the ability to understand the call center context and respond to simulated customer queries without deviating from the domain.
Limitations Observed: While functional, qualitative analysis revealed that some model responses were occasionally too generic or lacked the specificity required for nuanced queries, suggesting room for improvement in dataset diversity and the synthesis process.

Significance
The paper claims that the proposed approach is a viable solution for converting unstructured conversational data into valuable resources for training LLMs. By automating the transformation of raw call center recordings into instructional datasets, the work offers a systematic path to unlock operational knowledge that is typically dispersed and underutilized. This development has the potential to facilitate the creation of more effective, context-aware AI systems for customer service Q&A tasks. The authors emphasize that while the current work establishes a promising foundation, rigorous quantitative evaluation and further optimization of pipeline components (such as ASR accuracy and pairing algorithms) are necessary future steps.