💬 NLP

Reading Between the Lines: The One-Sided Conversation Problem

This paper introduces the one-sided conversation problem (1SC) as a framework for inferring and learning from single-speaker dialogue recordings, demonstrating that combining future-turn context with utterance length improves reconstruction and that high-quality summaries can be generated without fully reconstructing missing turns, thereby advancing privacy-aware conversational AI.

Original authors: Victoria Ebert, Rishabh Singh, Tuochao Chen, Noah A. Smith, Shyamnath Gollakota

Published 2026-04-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Victoria Ebert, Rishabh Singh, Tuochao Chen, Noah A. Smith, Shyamnath Gollakota

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are sitting in a crowded coffee shop, trying to listen to a conversation between two people at the next table. You can clearly hear Person A talking, but Person B is whispering, or perhaps their voice is muffled by the noise of the espresso machine. You only have half the story.

This is the real-world problem that the paper "Reading Between the Lines: The One-Sided Conversation Problem" tackles.

In many situations—like a doctor talking to a patient, a customer service agent on a call, or a person wearing smart glasses—we can only legally or technically record one side of the conversation. The other side is a "black box." The researchers ask: Can we use AI to fill in the blanks of that missing voice?

Here is a breakdown of their work using some everyday analogies.

The Two Main Challenges

The researchers treated this problem like a detective solving a mystery with two specific goals:

1. The "Ghost Writer" Task (Reconstruction)

The Goal: Try to guess exactly what the missing person said, turn by turn, as the conversation happens.
The Analogy: Imagine you are watching a play where the actor playing the villain is invisible. You can hear the hero's lines perfectly. The AI's job is to act as a ghost writer, improvising the villain's lines in real-time so the story makes sense.

The Catch: The AI must be careful not to make things up. If the hero asks, "What time is the meeting?", the AI shouldn't guess "3:00 PM" if it doesn't know. Instead, it should say, "The meeting is at [TIME]."
The Findings:
- Big Brains Win: Massive AI models (like the ones powering advanced chatbots) are surprisingly good at this "ghost writing" without needing much training. They can guess the vibe and tone well.
- Small Models Need Help: Smaller, cheaper models are like a student trying to write a novel; they need a lot of practice (fine-tuning) to get it right.
- Context is King: The more the AI knows about what happened just before and just after the missing line, the better it guesses. It's like trying to finish a sentence: knowing the next word helps you guess the current one.

2. The "News Editor" Task (Summarization)

The Goal: Instead of guessing the missing words, just write a summary of the whole conversation based only on what you heard.
The Analogy: Imagine you are a news editor who only has the reporter's notes (Person A) but not the interviewee's answers (Person B). Your job is to write a headline and a short article about what happened.

The Big Surprise: The researchers found that you don't need to guess the missing words to write a good summary.
- If you try to "ghost write" the missing parts first and then summarize, you often introduce errors (hallucinations). It's like a game of "Telephone" where the message gets distorted.
- It is actually better to act like a smart editor who looks at the available notes and says, "The user asked for a hotel, the agent suggested one, and they booked it," without needing to know the agent's exact phrasing.
- Result: Summaries created directly from the "one-sided" recording were often more accurate and trustworthy than summaries created after trying to fill in the gaps.

Why Does This Matter?

Think about Privacy as a pair of sunglasses.

The Problem: In the past, to make a helpful AI assistant (like one that reminds you of meeting details or helps a doctor remember a patient's history), we needed to record everyone. But in many places (like California or the EU), recording a conversation without everyone's consent is illegal.
The Solution: This research shows we can build "privacy-aware" assistants that only listen to you (the user) but still understand the whole context.
- Real-World Use: Imagine a smart earbud for a doctor. It listens only to the doctor. It can whisper to the doctor, "The patient mentioned they are allergic to penicillin," even though the AI never heard the patient say it directly. It infers the missing piece from the doctor's reaction.

The Verdict in Plain English

Filling in the blanks is hard: Trying to perfectly reconstruct what the other person said is difficult. Big AI models can do a decent job, but they aren't perfect. They are great at guessing the vibe but bad at guessing specific facts (like names or dates) without making things up.
Summarizing is easier: If your goal is just to get the "gist" of the conversation (e.g., "They agreed to meet Tuesday"), you don't need to reconstruct the missing voice. You can get a high-quality summary just by listening to one side.
Privacy is possible: We can build AI tools that respect privacy laws (by only recording one person) while still being smart enough to help us remember, organize, and understand our conversations.

In short: The paper teaches us that while we can't perfectly "read minds" to hear the missing speaker, we can definitely build AI that understands the story well enough to be helpful, without breaking the rules of privacy.

1. Problem Definition: The One-Sided Conversation (1SC)

The paper formalizes the One-Sided Conversation (1SC) problem, defined as the challenge of inferring and learning from a dialogue where only one speaker's utterances are observed. This scenario arises frequently in real-world applications due to:

Privacy Constraints: Laws in "all-party consent" states (e.g., California, Washington) and GDPR regulations often prohibit recording all participants without explicit consent.
Technical Limitations: Wearable devices (smart glasses, in-ear assistants) often capture only the wearer's speech to preserve battery life and privacy.

The authors identify two core tasks within 1SC:

Other Party Recreation: Reconstructing the missing speaker's turns in an online, turn-by-turn manner.
One-Sided Summarization: Generating faithful summaries of the entire conversation using only the available side (with or without reconstructed turns).

2. Methodology

Datasets

The study evaluates models on four datasets representing different levels of difficulty and modality:

Text-based: MultiWOZ (task-oriented) and DailyDialog (open-domain).
Audio-based (Transcribed): SpokenWOZ (task-oriented, simulated) and Candor (open-domain, unscripted video-chat transcripts with interruptions and natural speech nuances).

Task Formulations

A. Other Party Recreation:
The goal is to predict the masked speaker's turn ( $Turn_N$ ) given the user's side. The authors investigate several context variations:

Full Prior Context: All previous turns.
Lookahead (Turn $N+1$ ): Including the user's next turn to provide context for the masked turn (simulating a slight delay).
Turn Lengths: Providing the word count of the masked turn as a proxy for timing.
Local Context: Using only the immediate surrounding turns ( $N-1, N, N+1$ ) for computational efficiency.

B. Summarization:
Two strategies are compared:

Reconstruction-Free: Summarizing the dialogue with masked turns hidden (using placeholders like xxxx).
Reconstruction-Heavy: Reconstructing the missing turns first, then summarizing the "complete" dialogue.

Models & Training

Large Models: Claude-4-Sonnet and Llama-3.2-1B-Instruct used via prompting.
Small Models: Llama-3.2-1B fine-tuned on the limited context task (autoregressive infilling).
Anti-Hallucination Strategy: Models are instructed to use placeholders (xxxx or XXXXXXX) for any specific details (names, dates, numbers) not present in the context, rather than fabricating facts.

Evaluation Framework

The authors introduce a rigorous evaluation framework combining:

Human A/B Testing: 16 human judges compare ground truth vs. model reconstructions.
LLM-as-a-Judge: GPT-4O evaluates outputs based on a 5-point rubric covering:
- Semantic Similarity
- Intent Preservation
- Specific Information Hallucination (penalizing fabricated facts)
- Contextual Appropriateness
- Summary Alignment
Precision-Recall: Extracting specific details from ground truth and predictions to calculate overlap, treating placeholders as correct matches for unknown info.

3. Key Results

Reconstruction Findings

Context is Critical: Access to the immediately following user turn (Turn $N+1$ ) and turn length information significantly improves reconstruction quality.
Model Size Matters: Large pretrained models (Claude) generate plausible reconstructions "out-of-the-box" with prompting. Smaller models (1B parameters) require fine-tuning and still fail to match the performance of larger models, even with fine-tuning.
Text vs. Speech: Reconstruction performance drops on audio-based transcripts (SpokenWOZ, Candor) compared to text, primarily due to the complexity of unscripted speech, interruptions, and lack of structure.
Human Perception: In A/B tests, humans often cannot distinguish between ground truth and high-quality model reconstructions, particularly with large models.

Summarization Findings

Reconstruction is Not Necessary for Summaries: Contrary to intuition, high-quality summaries can be generated directly from one-sided input without reconstructing missing turns.
Faithfulness vs. Plausibility: While reconstructed turns are often plausible, they introduce hallucinated details that degrade the factual accuracy of the final summary. Masked inputs (leaving turns hidden) yield summaries that are more faithful to the ground truth.
Dataset Dependency: In task-oriented settings (MultiWOZ), reconstruction offered slight benefits. However, in open-domain and unscripted settings (Candor), reconstruction-free summaries performed better or equally well.
Audio Performance: Surprisingly, summarization performance on spoken transcripts was comparable to text-based conversations, suggesting that summarization is more robust to the "missing speaker" problem than turn-by-turn reconstruction.

4. Key Contributions

Formalization of 1SC: The paper defines a new, privacy-aware NLP challenge that moves beyond standard text infilling to reasoning over entire absent dialogue streams.
Novel Evaluation Metrics: Introduction of a rubric specifically designed to penalize hallucination of specific details (names, times) while rewarding contextual coherence, alongside precision-recall metrics for dialogue details.
Empirical Insights:
- Demonstrated that lookahead context (Turn $N+1$ ) is a powerful signal for reconstruction.
- Proved that reconstruction-free summarization is often superior to reconstruction-heavy approaches for maintaining factual integrity.
- Highlighted the performance gap between large and small models in this specific generative task.

5. Significance and Future Directions

This work lays the foundation for privacy-aware conversational AI. It demonstrates that AI assistants can operate effectively in environments where only one side of a conversation is recorded (e.g., telemedicine, call centers, personal assistants) without violating privacy laws.

Implications:

Proactive Guidance: Systems can infer missing context to provide real-time support (e.g., reminding a user of a missed question) without needing the other party's audio.
Documentation: Organizations can generate accurate meeting summaries from single-sided recordings, ensuring compliance with privacy regulations.

Limitations & Future Work:

Uncertainty: Reconstruction inherently introduces uncertainty; even with placeholders, models may subtly shift intent.
Model Scale: Smaller models struggle with the task, suggesting a need for specialized fine-tuning or larger parameter counts.
Multi-party: The current work focuses on two-party dialogues; extending this to multi-party conversations is a future direction.
Privacy Risks: The authors note that highly accurate reconstruction could theoretically replicate private conversations, necessitating robust safeguards and transparent communication about AI-generated content.