💬 NLP

Information Dynamics of Language Communication

This paper introduces an information-theoretic framework utilizing large language models to quantify and decompose the directed flow of semantic content in dialogue, demonstrating its utility in analyzing cognitive rigidity, persuasion, psychotherapy quality, and argumentative structure.

Original authors: Leonardo S. Goodall, Andrea I. Luppi, Pedro A. M. Mediano

Published 2026-06-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Leonardo S. Goodall, Andrea I. Luppi, Pedro A. M. Mediano

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Measuring the "Flow" of Meaning

Imagine two people talking. Usually, we just listen to what they say. But this paper asks a deeper question: How much does Person A's thinking actually change Person B's next thought?

The authors, Leonardo Goodall, Andrea Luppi, and Pedro Mediano, created a new "ruler" to measure the invisible flow of meaning between people. They call this an Information-Theoretic Framework.

Think of a conversation like a river. Sometimes, the water flows smoothly from one bank to the other. Sometimes, the banks are so rigid that the water doesn't mix at all. Sometimes, two streams merge to create a powerful new current that neither could create alone.

To measure this, they used Large Language Models (LLMs)—the same kind of AI you might chat with—as a "super-listener." These AIs are trained on massive amounts of text, so they are incredibly good at guessing what word comes next. The researchers used this guessing ability to calculate exactly how much one person's words help predict the other person's next words.

They introduced two main tools to do this:

Tool 1: Semantic Transfer Entropy (STE)

The "Who is Leading?" Meter

Imagine you are watching a dance.

High STE: If the dancer on the left moves, the dancer on the right immediately follows with a perfect step. The left dancer is "leading" the flow of information.
Low STE: If the dancer on the left moves, the dancer on the right does something completely unrelated. The flow is broken.

How it works: The researchers ask the AI, "If I hide what Person A just said, how hard is it for you to guess what Person B will say next?"

If the AI struggles a lot without Person A's words, it means Person A is heavily influencing Person B.
If the AI guesses just as well without Person A's words, it means Person B is thinking independently.

What they found:

Rigid vs. Flexible Minds: In computer-generated conversations, people who were programmed to be "cognitively rigid" (stubborn, stuck in their ways) had much lower information flow. They didn't pick up on their partner's cues. Flexible people flowed better.
The Persuader: In charity donation chats, the person asking for money (the persuader) heavily influenced the other person. The flow was one-way: Persuader → Donor. Interestingly, when the persuader told a personal story or used a "foot-in-the-door" tactic, the influence actually dropped slightly. It seems these self-contained stories didn't help predict the donor's next move as much as a direct question did.
Therapy Quality: In therapy sessions, they found a surprising pattern. In high-quality therapy, the therapist didn't dominate the flow. The client led the conversation, and the therapist followed. In low-quality sessions, the therapist dominated, making the client's responses very predictable (and less autonomous). The best therapy looks like a balanced dance, not a lecture.

Tool 2: Semantic Partial Information Decomposition (SPID)

The "Teamwork" Analyzer

Imagine you are trying to guess the ending of a mystery story.

Redundancy: Two friends both tell you, "The butler did it." You hear the same thing twice. It's helpful, but not new.
Unique: One friend says, "The butler did it," and another says, "The victim was poisoned." Each adds a completely different, necessary piece of the puzzle.
Synergy: One friend says, "The butler was in the kitchen," and another says, "The poison was in the tea." Neither clue alone tells you who did it. But together, they reveal the killer. The whole is greater than the sum of the parts.

How it works: The researchers looked at essays where students made a claim (e.g., "Dancing is important") supported by two premises (reasons). They used the AI to see how much those two reasons worked together to support the claim.

What they found:

Synergy is Real: When two reasons supported a claim, they often worked synergistically. The combination of the two reasons made the claim much stronger than just adding the two reasons together.
Too Many Cooks: However, as they added more reasons (3 or 4), the "synergy" disappeared. The extra reasons started to repeat the same ideas (redundancy). It's like adding more people to a team who just say the same thing; you get more noise, not more insight.

Why This Matters (According to the Paper)

Before this paper, scientists measured conversations by counting words, looking at grammar, or checking if people used similar vocabulary. It was like measuring a movie by counting the number of red pixels on the screen.

This new framework measures the actual meaning flowing between people. It doesn't just look at the surface; it looks at the "predictive power" of the conversation.

The Takeaway:
By using AI as a mathematical lens, the authors showed that:

Flexibility creates better information flow.
Good therapy is about the client leading, not the therapist.
Strong arguments rely on premises that work together (synergy), not just piling on more reasons.

The paper proves that we can now measure the "electricity" of a conversation—the invisible spark that happens when one mind truly influences another.

Technical Summary: Information Dynamics of Language Communication

Problem Statement

Quantifying how meaning propagates through communicative exchanges remains underdeveloped in computational linguistics. While the field has advanced from surface-level lexical measures to contextual neural representations, no existing approach provides a principled, unified framework to quantify:

The amount of semantic information flowing between interlocutors.
The directionality of this flow.
How multiple sources jointly contribute to a target's language (distinguishing between redundant, unique, and synergistic contributions).

Current methods rely on ad hoc measures, surface features, or correlational summaries that address directionality or multi-source structure separately, but not both in a unified manner. Furthermore, applying information-theoretic measures like Transfer Entropy (TE) and Partial Information Decomposition (PID) directly to natural language is challenging because these measures require well-defined probability distributions, which are not readily yielded by high-dimensional neural embedding spaces.

Methodology

The authors introduce a framework leveraging Large Language Models (LLMs) as probabilistic estimators of natural language to compute two novel information-theoretic measures: Semantic Transfer Entropy (STE) and Semantic Partial Information Decomposition (SPID).

Core Measures

Semantic Transfer Entropy (STE):
- Definition: STE quantifies the directed predictive influence of a source ( $Y$ ) on a target ( $X$ ). It measures the reduction in uncertainty about $X$ 's future utterance given $Y$ 's past content, beyond what is afforded by $X$ 's own history and the rest of the context ( $Z$ ).
- Formula: $STE_{Y \to X} = I(Y; X | X^-, Z^-) = E[\log p(X|X^-, Y^-, Z^-) - \log p(X|X^-, Z^-)]$ .
- Implementation: Implemented via attention masking in LLMs. The model predicts a target's next token with and without access to the source's prior utterances (by masking attention from the target to the source). The difference in log-likelihoods yields the STE in bits. This preserves the conversational context while isolating the direct information flow.
Semantic Partial Information Decomposition (SPID):
- Definition: SPID extends the analysis to multiple sources, decomposing how they jointly contribute to a target's language into Redundant (shared), Unique (exclusive to one source), and Synergistic (only available when sources are considered jointly) components.
- Implementation: Unlike STE, SPID applies to configurations where sources are independent texts (e.g., premises supporting a claim). It uses physical omission of a source from the input sequence to obtain true marginal distributions, avoiding cascading representational changes.
- Redundancy Functional: The authors adopt the Minimum Mutual Information (MMI) functional to identify redundancy.
- Scalability: For $n > 2$ sources, a full decomposition is unwieldy. The authors utilize the Redundancy-Synergy Index (RSI), defined as $I(Y; X_1, \dots, X_n) - \sum I(Y; X_i)$ , to provide a scalar summary of the redundancy-synergy balance.

Experimental Design

The framework was validated across four distinct experiments:

Experiment 1 (Synthetic): 3,000 dyadic conversations generated by an LLM with manipulated cognitive styles (rigid vs. flexible) to test sensitivity to communicative style.
Experiment 2 (Naturalistic Persuasion): Analysis of the PERSUASIONFORGOOD corpus (300 dyads) to examine directional asymmetry between persuaders and persuadees and the impact of rhetorical strategies.
Experiment 3 (Clinical): Analysis of the AnnoMI corpus (133 motivational interviewing sessions) to distinguish high-quality (client-led) from low-quality (therapist-dominated) therapy.
Experiment 4 (Argumentation): Analysis of the Argument Annotated Essays (AAE-v2) corpus to decompose the informational structure of claims supported by multiple premises.

Key Results

1. Sensitivity to Cognitive Rigidity (Exp 1)

STE successfully detected theoretically predicted differences in information dynamics. Flexible sources generated significantly higher directed information transfer (STE) than rigid sources, regardless of the target's style. This confirms STE's discriminative capacity under controlled conditions.

2. Directional Asymmetry in Persuasion (Exp 2)

STE revealed a striking directional asymmetry in persuasion dialogues: information transfer from persuader to persuadee (mean = 1.70 bits/token) vastly exceeded the reverse direction (0.05 bits/token).

Strategy Effects: Specific strategies (e.g., personal stories, credibility appeals) were associated with reduced STE. The authors interpret this not as ineffectiveness, but as the introduction of self-contained content that adds little incremental signal for predicting the persuadee's next turn beyond their own history.

3. Therapeutic Alliance Quality (Exp 3)

STE served as an objective marker for therapeutic quality in Motivational Interviewing (MI).

Low-quality sessions (characterized by therapist confrontation/directing) showed higher therapist-to-client STE.
High-quality sessions (characterized by client-led change talk) showed lower therapist-to-client STE.
A weak but significant negative correlation was found between therapist-to-client STE and the proportion of client change talk, suggesting that therapist informational dominance correlates with reduced client autonomy.

4. Synergistic Premises in Argumentation (Exp 4)

SPID analysis of argumentative essays revealed:

Positive Synergy: Two-premise arguments exhibited positive synergy on average, meaning premises jointly provided more predictive information about a claim than the sum of their individual contributions.
Redundancy vs. Synergy: As the number of premises increased (from 2 to 4), the Redundancy-Synergy Index (RSI) became increasingly negative, indicating a shift toward redundancy dominance.
Diminishing Returns: Per-premise information yield decreased as more premises were added, suggesting that additional premises often provide overlapping rather than complementary evidence.

Robustness

All key findings replicated across three different language models (LLaMA 3.2-3B, Phi-3-mini-4k, Mistral-7B-v0.3) spanning two architectures and two scales, supporting the framework's robustness independent of the specific estimation model.

Significance and Claims

The paper claims to introduce a general-purpose, principled framework for studying information dynamics in any system where agents exchange meaningful sequential text.

Theoretical Contribution: It unites Shannon's mathematical theory of communication with the semantic representational capacity of modern LLMs. The authors clarify that while STE and SPID are formally structural quantities (measuring uncertainty reduction in bits), they operate over contextualized semantic representations rather than raw symbol frequencies.
Methodological Advance: It moves beyond ad hoc measures (e.g., embedding distances, Granger causality on extracted features) to quantify semantic influence directly on the full semantic content of utterances.
Applicability: The framework is presented as portable to diverse domains including clinical dialogues (tracking therapeutic alliance), political/media communication (tracing semantic framing propagation), education (decomposing instructional sources), and human-AI interaction.

The authors maintain a modest stance regarding causality, noting that STE measures "directed predictive dependence" and that causal interpretations rely on standard Granger causality assumptions (stationarity, no hidden confounders, correct model specification). They also acknowledge limitations, such as the dependence on the specific LLM used for estimation and the computational cost of running forward passes for conditioning sets.