Strategic Dialogue Assessment: The Crooked Path to Innocence

Technical Summary: Strategic Dialogue Assessment (SDA)

Problem Statement
Current research in pragmatics and Large Language Model (LLM) evaluation predominantly assumes cooperative discourse, grounded in Gricean principles where interlocutors share common goals. However, this assumption fails in high-stakes, adversarial settings (e.g., courtroom cross-examinations) where interlocutors have misaligned goals. In such non-cooperative contexts, speakers strategically manipulate language—often appearing cooperative while pursuing self-interested goals—to manage commitments, maintain credibility, and advance discourse objectives. Existing frameworks, such as Signaling Games and Rational Speech Act (RSA) theory, often rely on cooperative utility functions or abstract constructs (e.g., credibility distributions) that are difficult to estimate empirically. Furthermore, current LLMs exhibit limited ability to recognize the strategic value of language in adversarial settings, often misinterpreting evasion or hedging as detrimental rather than strategically beneficial, or failing to distinguish between surface-level damage control and actual strategic gains.

Methodology
To address these gaps, the authors introduce Strategic Dialogue Assessment (SDA), a framework that operationalizes the "jury function" from Message Exchange (ME) Games to assess strategic language use in real-world data. The methodology involves three core components:

1. The Crooked Path Dataset (CPD): A new annotated dataset comprising 4,452 turns from cross-examinations in three prominent U.S. trials (West Memphis Three, O.J. Simpson, and Enron). The dataset focuses on adversarial interactions where goals are strictly opposed.
2. Theoretical Adaptation: The authors adapt the ME Game jury function by integrating two key modifications to make it empirically estimable:
   • Commitment-Based Taxonomy: Instead of relying solely on abstract coherence relations, the framework classifies discourse moves based on the content of commitments relative to the current Question Under Discussion (QUD). Commitments are categorized as Beneficial, Detrimental, Neutral, or None (lacking meaningful commitment).
   • Gricean Proxies for Credibility: To operationalize the abstract credibility term $P_k(Good_i)$, the framework uses observable violations of Gricean maxims (Relevance, Manner, and Quality) as proxies. Violations modulate the strategic value of a commitment; for instance, a beneficial commitment conveyed through an irrelevant or unclear statement is penalized, while a detrimental commitment conveyed indirectly (e.g., via hedging) may be less damaging.
3. Metric Definition: SDA defines three interpretable metrics to quantify strategic effects:
   • Benefit at Turn ($BAT_i$): Quantifies immediate gains from beneficial or neutral commitments, adjusted for maxim violations.
   • Penalty at Turn ($PAT_i$): Quantifies immediate losses from detrimental or non-committal responses, adjusted for consistency and maxim violations.
   • Normalized Relative Benefit at Turn ($NRBAT_i$): A cumulative, z-scored measure of the net strategic advantage ($\sum BAT - \sum PAT$) over the course of the dialogue.

The framework was validated through human annotation of a subset of CPD (approx. 800 turns) by three linguistically trained annotators, followed by an evaluation of various state-of-the-art LLMs (ranging from 7B to 70B parameters, including reasoning-enhanced variants) using zero-shot prompting.

Key Results

• Human Annotation Validity: The SDA metrics demonstrated high inter-annotator agreement (Spearman's $\rho$ of 0.65–0.83 for $BAT$, $PAT$, and $NRBAT$), significantly outperforming agreement on raw outcome judgments (Fleiss' $\kappa$ of 0.29). This suggests the metrics capture objective, discourse-internal properties better than subjective outcome predictions.
• Predictive Power: Logistic regression models using $BAT$ and $PAT$ as predictors achieved 74.6% accuracy and an AUC of 0.80 in predicting human annotators' outcome judgments. The metrics were particularly effective when annotators' reasoning was based on logical arguments rather than personal emotions.
• LLM Performance:
  • Model Size: Larger models generally aligned more closely with human judgments on strategic metrics than smaller models, particularly in identifying strategic gains.
  • Reasoning Limitations: Counter-intuitively, models equipped with explicit reasoning mechanisms (Chain-of-Thought) often performed worse than their non-reasoning counterparts, particularly in identifying strategic losses ($PAT$). The analysis of reasoning traces revealed that these models tend to overcomplicate the task, misinterpreting surface-level damage control strategies (e.g., hedging) as neutral or beneficial, and failing to integrate broader adversarial goals.
  • Gricean vs. Strategic: While LLMs performed well in identifying violations of Gricean maxims (high agreement on Relevance, Manner, and Quality), they struggled significantly with identifying the strategic type of commitments ($\mu \approx 0.14$) and calculating the resulting strategic metrics.

Significance and Claims
The paper claims that SDA provides a theoretically grounded yet empirically operationalizable framework for evaluating strategic communication in non-cooperative settings. Its primary significance lies in:

1. Bridging Theory and Data: It successfully adapts the abstract ME Game jury function to real-world, high-stakes discourse by replacing unobservable constructs with estimable proxies (Gricean violations) and a commitment-based taxonomy.
2. Revealing LLM Limitations: The study demonstrates that current LLMs, even large and reasoning-enhanced ones, lack a robust understanding of strategic language in adversarial contexts. They often fail to distinguish between cooperative flouting and strategic evasion, and their reasoning capabilities can introduce confusion rather than clarity in these settings.
3. Asymmetry of Discourse: The framework highlights that cooperative and non-cooperative discourses are asymmetric; non-cooperative discourse involves a higher frequency of detrimental commitments and maxim violations, yet these are often strategic tools for loss minimization rather than mere failures of communication.

The authors maintain a modest stance, noting that the framework does not perfectly predict conversational outcomes (which depend on factors beyond discourse, such as bias) and that the current annotation focuses on local, turn-level effects rather than long-horizon strategic planning. They position SDA as a necessary step toward better alignment and safety monitoring for AI agents operating in adversarial environments.