Investigating the Representation of Backchannels and Fillers in Fine-tuned Language Models

1. Problem Statement

Modern transformer-based Large Language Models (LLMs) are typically trained on text corpora where backchannels (e.g., "uh-huh," "yeah") and fillers (e.g., "uh," "um") are often treated as noise and removed during preprocessing. Consequently, these models lack the ability to represent or generate these crucial linguistic elements.

• The Gap: Backchannels and fillers are not semantically "bleached"; they carry rich pragmatic functions (e.g., turn-holding, agreement, signaling cognitive load, and managing conversation flow).
• The Consequence: Current LMs produce text-like outputs that lack the fluidity and social cues of natural human dialogue, failing to act as competent conversational agents.
• Research Goal: To investigate whether fine-tuning strategies can enable LMs to learn and improve the semantic representations of backchannels and fillers, thereby transforming general LMs into more human-like conversational agents.

2. Methodology

Data Selection

The study utilized three annotated dialogue corpora containing preserved backchannels and fillers:

• English: Switchboard (Godfrey & Holliman, 1993) and MapTask (Anderson et al., 1991).
• Japanese: BTSJ 1000 Person Japanese Natural Conversation Corpus (Usami, 2023).
• Focus: The top 15 most frequent backchannels/fillers in each language were selected for analysis (e.g., uh, yeah, uh-huh in English; un, hai, a in Japanese).

Models Evaluated

Four transformer-based models were used, ranging from smaller to larger architectures:

• BERT (English & Japanese versions).
• GPT-2 (English & Japanese versions).
• LLaMA-3 8B (Multilingual).
• Qwen-3 8B (Multilingual).

Fine-Tuning Strategies

Three distinct downstream tasks were employed to teach the models the context of these linguistic units:

1. Masking (MASK): Inspired by BERT's pre-training. Specific backchannels/fillers were masked (80% replaced with [MASK], 10% random tokens, 10% unchanged), and the model was trained to predict the original tokens using bidirectional context.
2. Next Token Prediction (NTP): A standard autoregressive task where the model predicts the next token (which could be a filler/backchannel or a content word) given the preceding context.
3. Turn-Taking Prediction (TTP): Using the TurnGPT framework, the model predicts the probability of a turn-shift occurring after a specific token. This task leverages the strong correlation between backchannels and turn management.

Note: For MASK and NTP, utterances from both speakers were merged into a single sequence with speaker IDs (<s1>, <s2>) to preserve context.

Context Settings

To analyze the role of context, embeddings were extracted under three conditions:

• No Context: Only the utterance containing the backchannel/filler.
• One Context: The target utterance plus the immediately preceding and following utterances.
• Full Context: All preceding utterances in the dialogue (applicable only to LLaMA-3 and Qwen-3 due to context window limits).

Evaluation Metrics

• Representation Quality (Unsupervised):
  • Clustering Analysis: K-means clustering was applied to the embeddings of the 15 target words.
  • Silhouette Score (SC): Used to measure clustering quality (higher scores indicate better separation between different backchannel types).
  • Confusion Matrices: Visualized distances between embeddings to show distinctiveness.
• Generation Quality (Supervised/Generative):
  • Models were tasked with continuing a dialogue.
  • Metrics: Frequency and diversity of generated backchannels, frequency-weighted Perplexity (PPL), BERTScore (F1), and BLEU Score against ground truth.

3. Key Contributions

1. Empirical Evidence of Representation Learning: The study provides the first systematic evidence that fine-tuning significantly improves the internal representation of backchannels and fillers in LMs, moving them from "random" or "bleached" vectors to distinct, functionally clustered representations.
2. Comparative Analysis of Strategies: It compares MASK, NTP, and TTP strategies, finding that while all improve representation, NTP generally yields the highest silhouette scores, though MASK performs exceptionally well for Japanese BERT.
3. Contextual Impact Analysis: The paper reveals a nuanced relationship between context size and representation quality. While some context helps, excessive context (Full Context) can sometimes "dilute" the specific representation of functional words by compressing them into limited semantic regions, leading to lower silhouette scores in some cases.
4. Model Architecture Insights: It demonstrates that larger models (LLaMA-3, Qwen-3) generally achieve better clustering, but smaller models (BERT) can also learn these representations effectively with the right fine-tuning task (MASK).

4. Results

• Silhouette Scores: Across all models and languages, fine-tuning consistently increased the average silhouette scores.
  • Example: For LLaMA-3 (English), the average SC increased from ~0.45 (pre-trained) to ~0.59 (NTP fine-tuned).
  • This indicates that fine-tuned models distinguish between different pragmatic functions (e.g., "uh" for hesitation vs. "yeah" for agreement) much better than pre-trained models.
• Clustering Behavior:
  • Pre-trained models showed high overlap (low SC) and random clustering.
  • Fine-tuned models showed clearer boundaries in t-SNE visualizations and more distinct clusters in K-means analysis.
• Generation Performance:
  • Fine-tuned models generated backchannels/fillers with higher frequency and greater diversity compared to pre-trained baselines.
  • Perplexity on generated backchannels decreased significantly (e.g., English LLaMA-3 PPL dropped from 197.67 to 5.30), indicating the model learned the statistical likelihood of these tokens in context.
  • Similarity: BERTScore and BLEU scores improved, showing the generated dialogues were closer to human ground truth.
• Qualitative Findings: Fine-tuned models demonstrated the ability to use fillers for cognitive load signaling (e.g., "um" before a difficult word) and backchannels for grounding (e.g., "uh-huh" to confirm understanding), mimicking human conversational flow.

5. Significance and Implications

• Beyond "Noise": The paper challenges the NLP convention of treating backchannels and fillers as noise to be cleaned. It proves they are learnable, representable linguistic units with distinct semantic and pragmatic roles.
• Conversational AI: The findings suggest a pathway to building more natural, attentive, and socially aware dialogue agents. By fine-tuning on dialogue data, LMs can learn to manage turn-taking and provide appropriate feedback, essential for human-computer interaction.
• Under-represented Tokens: This serves as a case study for how LMs can learn to represent tokens that are rare or poorly represented in general pre-training corpora, provided they are exposed to rich contextual data via fine-tuning.
• Future Directions: The authors highlight the need to explore vocal signals (prosody, pitch) in speech models and the potential of advanced fine-tuning techniques (e.g., surgical fine-tuning) to further refine these representations without compromising general language capabilities.

In conclusion, the study demonstrates that fine-tuning is a critical mechanism for transforming general-purpose LMs into conversational agents capable of producing human-like, pragmatically rich dialogue by effectively learning the representations of backchannels and fillers.