💬 NLP

The distribution of discourse relations within and across turns in spontaneous conversation

This study adapts a written-language discourse relation system for spontaneous dialogue using crowdsourced annotations to demonstrate that discourse relation distributions vary significantly across different conversational contexts, with single-turn annotations presenting the most uncertainty, while confirming that these annotations are of sufficient quality to be predicted from discourse unit embeddings.

Original authors: S. Magalí López Cortez, Cassandra L. Jacobs

Published 2026-02-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: S. Magalí López Cortez, Cassandra L. Jacobs

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a lively dinner party where two strangers are trying to get to know each other. They are talking about everything from childcare to recycling. Now, imagine you are a detective trying to figure out the "secret logic" behind every sentence they say. Why did Person A say that? Was it to explain something? To make a joke? To ask for clarification?

This paper is about building a map of those "secret connections" (called Discourse Relations) in real, messy, spontaneous conversations, rather than in polished, written articles.

Here is the breakdown of their journey, using some everyday analogies:

1. The Goal: Mapping the Invisible Threads

Think of a conversation like a tapestry. In a written book, the threads (the connections between sentences) are usually neat and obvious. But in a live conversation, the threads are tangled, fast, and sometimes invisible.

The researchers wanted to see if the rules for how we connect ideas in writing (like "Explanation" or "Contrast") work the same way when people are just chatting naturally. They took 19 real conversations from a famous database (the Switchboard Corpus) and asked students to act as "thread-mappers."

2. The Experiment: The "Thread-Mappers"

Instead of hiring expensive linguistics experts, they recruited 114 college students (novices) and split them into teams.

The Task: They showed the students pairs of sentences (or parts of sentences) and asked, "What is the relationship between these two?"
The Twist: They tested these pairs in three different scenarios:
1. Same Speaker, Same Turn: One person saying two things in a row without stopping.
2. Same Speaker, Different Turns: One person speaking, pausing, and then speaking again later.
3. Different Speakers: Person A says something, and Person B replies.

3. The Big Surprise: The "Single-Turn" Confusion

Here is the most interesting part of the story.

The researchers expected that it would be hardest to figure out the connection between two different people (because you have to guess what the other person is thinking).

But the opposite happened.
The students were actually most confused when trying to link two sentences spoken by the same person in a single breath.

The Analogy: Imagine you are watching a magician. When the magician talks to the audience, the connection is clear ("I will pull a rabbit out of this hat"). But when the magician talks to themselves while setting up a trick, it's hard to tell if they are explaining the trick, making a joke, or just muttering.
The Result: The students marked more connections in single-turn speech, but they were also more unsure about them. It seems that when one person is talking to themselves, the "logic" is fuzzier. When two people are talking to each other, the logic is clearer (like a Question followed by an Answer).

4. The "AI Detective" Test

To see if the students were actually doing a good job, the researchers built a computer program (an AI) to play the same game.

They fed the AI the text and asked it to guess the connections.
The Result: The AI was terrible at guessing the exact label the students picked (it got the details wrong). However, it was pretty good at guessing the general vibe or the "top choice" of the students.
What this means: Even though the students were confused, their confusion wasn't random. The patterns in the conversation were real enough that a computer could learn to spot them. This proves that the "messy" data is actually usable for teaching computers how to understand human chat.

5. The Takeaway

The paper concludes that context is king.

If you want to understand how people connect ideas, you can't just look at the words; you have to look at who is speaking and when.
Connections between different people are like a clear game of tennis (serve and return).
Connections within one person's speech are like a solo jazz improvisation—beautiful, but harder to predict and categorize.

In short: We thought it would be hard to map connections between strangers, but it turns out it's actually harder to map the tangled thoughts of a single person talking to themselves. And that's okay—it just means human conversation is a lot more complex and interesting than we thought!

1. Problem Statement

Discourse Relations (DRs) are fundamental to text coherence, yet most existing annotated corpora and computational models focus on written language (e.g., news articles) or highly structured, constrained conversations (e.g., game-based dialogues). There is a significant lack of annotated data for spontaneous, multi-party spoken dialogue.

The authors identify a gap in understanding how DRs function in spontaneous conversation, where participants face time pressure and must negotiate topics dynamically. Key research questions include:

Can discourse relation systems designed for written text (specifically Segmented Discourse Representation Theory, SDRT) be effectively adapted to spontaneous dialogue?
Do the patterns of DR usage differ significantly based on the discourse context (i.e., within a single turn, across turns by the same speaker, or across turns by different speakers)?
Is the annotation quality sufficient to train predictive models, or does the inherent ambiguity of spontaneous speech lead to high uncertainty?

2. Methodology

Data Source

The study utilizes a subset of 19 dialogues from the Switchboard Corpus, a well-known dataset of informal telephone conversations between strangers discussing assigned topics (e.g., childcare).

Annotation Framework

Theory: The authors adapted Segmented Discourse Representation Theory (SDRT) and guidelines from the STAC corpus.
Labels: From an original set of 16, 11 relation labels were selected based on a pilot study (e.g., Elaboration, Background, Explanation, Contrast, Acknowledgement, Clarification Question). An "Other" category was included.
Unit Definition: Elementary Discourse Units (EDUs) were defined similarly to written text but modified to handle spoken language noise (e.g., laughter, restarts, disfluencies like "uh/um"). Complex Discourse Units (CDUs) were used when multiple EDUs functioned as a single argument.

Annotation Process

Annotators: 114 undergraduate students in a computational linguistics course, grouped into 19 teams.
Procedure: Annotators worked individually to label pairs of EDUs/CDUs ( $\pi_1$ and $\pi_2$ ) presented in the Prodigy annotation tool. They were provided with 2 preceding and 2 subsequent turns for context.
Multi-labeling: Crucially, annotators were allowed to select multiple relations for a single pair if applicable.
Contexts Analyzed: The study specifically compared three contexts:
1. Within-turn: Relations between EDUs in the same speaker's turn.
2. Across-turn (Same Speaker): Relations between EDUs in different turns by the same speaker.
3. Across-turn (Different Speakers): Relations between adjacent turns of different speakers.

Evaluation & Modeling

Inter-annotator Agreement: Calculated using the Marchal et al. (2022) framework for multi-label tasks (including bootstrapped kappa, precision, recall, and F1).
Statistical Modeling: A multiclass logistic regression was used to predict relation labels based on discourse context variables (speaker identity, turn boundaries).
Classification: A BERT-based classifier (using Next Sentence Prediction architecture) was trained to predict DR labels from embeddings of the EDU pairs. The model used a "leave-one-conversation-out" cross-validation strategy to prevent overfitting to specific dialogue turns.

3. Key Results

Distribution and Uncertainty

Annotation Volume: Annotators selected significantly more relations per pair in single-turn contexts (avg. 1.60 per annotator) compared to across-turn contexts (same speaker: 1.45; different speakers: 1.30).
Interpretation: While this initially suggested annotators found more relations in single-turn speech, post-hoc polling revealed that 91.4% of annotators felt uncertain about distinguishing relations in single-turn contexts. This suggests the high number of labels in single-turn cases reflects ambiguity rather than clear multi-relation signals.
Contextual Differences:
- Relations like Acknowledgement, Clarification Questions, and Question-Answer Pairs were more frequent across speakers.
- Relations like Continuation, Elaboration, Explanation, and Narration were more frequent within a single speaker.

Inter-annotator Agreement

Agreement metrics were modest (e.g., Adjusted Kappa $\approx$ 0.18–0.36 depending on the metric).
The authors attribute this to the inherent difficulty of the task and the uncertainty of annotators regarding DR boundaries in spontaneous speech, rather than a failure of the guidelines.

Predictive Modeling

Logistic Regression: Adding discourse context variables significantly improved model fit ( $p < .001$ ), confirming that DR distributions are distinct across contexts.
BERT Classifier:
- Strict Accuracy: Performance was poor (Macro F1 = 0.19), likely due to the multi-label nature and annotation noise.
- Recall: When checking if the model's top guess was among the set of labels provided by annotators, recall was high (0.76).
- Uncertainty Quantification: Cross-entropy between model predictions and annotator distributions was highest for single-turn pairs (mean = 0.43) and lowest for cross-speaker pairs (mean = 0.38), quantitatively confirming that single-turn relations are the most ambiguous.

4. Key Contributions

First Annotated Corpus for Spontaneous Dialogue: This work presents one of the first attempts to systematically annotate DRs in spontaneous, multi-party conversation using SDRT guidelines.
Novice Annotation Validation: It demonstrates that crowdsourced/novice annotators can produce data of sufficient quality to train predictive models and reveal distinct distributional patterns, even if strict inter-annotator agreement is low.
Contextual Analysis: The study empirically proves that discourse context (turn boundaries and speaker identity) significantly influences the distribution and ambiguity of discourse relations.
Methodological Framework: It provides a pipeline for adapting written-language DR schemas to spoken data, including handling EDU segmentation in noisy speech and multi-label annotation strategies.

5. Significance and Limitations

Significance: The findings challenge the assumption that DRs function identically in spoken and written language. The results suggest that single-turn speech is inherently more ambiguous regarding discourse structure, likely due to the lack of turn-taking signals that clarify intent in cross-speaker interactions. This has implications for discourse parsing, dialogue systems, and NLP models that rely on coherence.
Limitations:
- Dataset Size: The study is limited to 19 dialogues.
- Label Exhaustiveness: The selected 11 labels are not exhaustive; specific cultural or situational relations may be missing.
- Annotation Quality: While sufficient for distributional analysis, the low agreement suggests that creating a "gold standard" single-label dataset for training is difficult without extensive expert adjudication.

In conclusion, the paper establishes that while spontaneous conversation presents unique challenges for discourse relation annotation, these challenges are quantifiable and context-dependent. The resulting data and models offer a foundation for developing more robust dialogue systems that account for the nuances of spoken interaction.