Technical Summary: Beneath the Surface: Investigating LLMs' Capabilities for Communicating with Subtext

1. Problem Statement

Human communication frequently relies on subtext—implied meaning that transcends literal content, requiring shared context, theory of mind, and pragmatic reasoning. While Large Language Models (LLMs) excel at literal text generation, their ability to navigate the nuances of subtext (e.g., allegory, irony, and selective communication) remains under-explored. Prior work suggests LLMs tend to be overly explicit and literal, but lacks systematic, quantitative evaluation across diverse communicative scenarios. This paper addresses the gap by asking: Can current frontier LLMs effectively generate and interpret subtext in multi-agent and storytelling settings?

2. Methodology

The authors introduce four novel evaluation environments designed to test subtext capabilities across two categories: Social Communication Games and Storytelling/Allegory.

A. Social Communication Games (Multi-Agent)

These environments simulate board games where agents must communicate to a specific subset of players while confusing others, relying on "common ground" (shared knowledge).

1. Visual Allusions (Inspired by Dixit):

   • Setup: Multi-player game where a "storyteller" selects a card and provides a natural language clue. Other players guess the card.
   • Objective: The storyteller scores only if some but not all other players guess correctly.
   • Subtext Challenge: The clue must be subtle enough to be ambiguous to the general group but specific enough for a "partner" (if they share common ground) to decode.
   • Variables: Tested with and without Common Ground (shared stories provided to specific agents) and with/without memory of previous rounds.
   • Metrics: Total score, win rate, and a "Spark Coefficient" measuring the exclusivity of communication between partners.

2. Attuned (Inspired by Wavelength):

   • Setup: Two teams compete. A sender picks a target location on a spectrum (e.g., "Ephemeral" to "Permanent") and provides a clue.
   • Objective: The sender's team must guess closer to the target than the opposing team.
   • Subtext Challenge: The sender must craft a clue that aligns their team's interpretation (using shared stories as common ground) while misleading the opponent.
   • Metric: MindRead, measuring the fraction of times the sender successfully guides their own team closer to the target than the opposition.

B. Storytelling and Allegory (Text Generation & Interpretation)

These environments test the generation and interpretation of long-form subtext.

3. Historical Allegories:

   • Setup: Synthetic stories are generated that allegorically represent historical events (e.g., Animal Farm style) without explicit mentions.
   • Objective: Evaluate how well LLMs decode the hidden historical event.
   • Variables: Tested under different Persona Conditions (Critic, Deep Reader, Historian) and Information Conditions (Default, Author Name, Reader Name) to study the impact of paratextual factors.

4. The Aesopian Author:

   • Setup: A multi-agent game involving an Author, an Inquisitor (censor), and a Critic.
   • Objective: The Author must write stories about a banned topic (e.g., democracy) that the Critic recognizes but the Inquisitor interprets as a celebrated topic (e.g., monarchy).
   • Constraints: The Author has 10 attempts; 3 strikes (Inquisitor detecting the banned topic) result in "death." The Author leverages information asymmetry (Inquisitor knows the Author's profile; Critic knows the Author's literary history).
   • Metric: Number of successful writing attempts (Critic detects banned topic, Inquisitor does not).

3. Key Contributions

• New Evaluation Suites: Introduction of four systematic environments (Visual Allusions, Attuned, Historical Allegories, The Aesopian Author) to quantify subtext capabilities, moving beyond static text analysis to dynamic, multi-agent interaction.
• Quantitative Metrics: Development of specific metrics like Spark Coefficient (exclusivity of communication) and MindRead (selective alignment) to measure subtext performance objectively.
• Systematic Analysis of Common Ground: Rigorous testing of whether LLMs can infer and utilize shared knowledge (common ground) to communicate selectively, a critical component of human pragmatics.
• Paratextual Influence Study: Demonstration of how external factors (author identity, reader persona) significantly shift LLM interpretation of subtext, aligning with literary theories (Genette, Barthes).

4. Key Results

A. Social Communication Games

• Literal Bias: Frontier models exhibit a strong bias toward overly literal communication. In Visual Allusions, even the best model (Gemini-2.5-Pro) generated "obvious" clues (guessable by everyone) 60% of the time, failing the core objective of selective communication.
• Common Ground Utilization:
  • When explicitly told about shared stories, advanced models (Gemini-2.5-Pro, GPT-5, Claude-Sonnet-4.5) reduced obvious clues by 30-50%.
  • However, models struggled to infer the existence of common ground when it was not explicitly stated. Even with shared context, performance remained significantly below theoretical maximums.
  • Memory: Adding memory of previous rounds yielded marginal improvements but did not solve the fundamental issue of over-literalization.
• Attuned Performance: Models showed low MindRead scores (~0.33), indicating difficulty in selectively aligning with teammates while confusing opponents, suggesting limitations in Theory of Mind application.

B. Storytelling and Allegory

• Historical Allegories:
  • Persona Impact: Models performed best when explicitly prompted as a "Historian" (up to 93% accuracy for GPT-5).
  • Paratextual Shift: Providing Author Names or Reader Names significantly improved decoding accuracy (e.g., GPT-5 improved from 45% to 61% in "Deep Reader" mode with author name), demonstrating that LLMs are highly sensitive to contextual framing.
• The Aesopian Author:
  • Limited Success: Models struggled significantly with this complex task. The best model (GPT-5) achieved success in only 22% of attempts (2.2 successes out of 10).
  • Strategy: GPT-5 showed a unique strategy of writing "conforming" stories to build trust with the Inquisitor before attempting subtext, but often failed to maintain the balance, resulting in early strikes.
  • Alignment Sensitivity: Performance dropped drastically when the Author's profile was "misaligned" with the state, highlighting the difficulty of subtext generation under adversarial constraints.

5. Significance and Implications

• Pragmatic Reasoning Gap: The results indicate that while frontier LLMs are improving, they lack the sophisticated pragmatic reasoning and Theory of Mind required for nuanced human communication. They default to explicitness, failing to account for the "common ground" necessary for subtext.
• Dual-Use Risks: The ability to use subtext is a dual-use capability. While essential for creative writing and art, it could be exploited for deception, steganography, or evading safety filters (e.g., communicating banned topics to a specific audience while fooling safety systems).
• Future Directions: The paper argues for training models not just on literal instruction following but on socially grounded creative communication. It suggests that current benchmarks focusing on static outputs are insufficient; dynamic, multi-agent environments are necessary to evaluate true communicative competence.
• Open Source: The authors plan to release code for these environments to foster further research into LLMs' social and creative reasoning capabilities.

In conclusion, while current LLMs show emerging capabilities in handling subtext when explicitly guided by shared context, they fundamentally struggle with the implicit, recursive reasoning required for natural, subtext-driven human interaction.