NewsInterview: a Dataset and a Playground to Evaluate LLMs' Ground Gap via Informational Interviews

1. Problem Statement

Large Language Models (LLMs) have demonstrated proficiency in generating coherent text but struggle significantly with strategic dialogue and grounding. Grounding refers to the use of language (e.g., acknowledgments, affirmations) to establish mutual understanding, trust, and rapport.

• The Gap: In real-world scenarios like journalism, therapy, or education, effective dialogue requires long-horizon strategic planning and emotional intelligence. Current LLMs fail to mimic human interviewers who use grounding to coax information from reluctant sources.
• Data Scarcity: Existing datasets for studying these phenomena are either small-scale (due to privacy concerns in therapy/education) or crowdsourced (lacking naturalistic complexity). There is a lack of large-scale, naturalistic data to study how LLMs handle multi-turn strategic planning and persuasion.

2. Methodology

The authors propose a three-pronged approach: Dataset Curation, Counterfactual Analysis, and a Simulated Game Environment.

A. Dataset Curation (NewsInterview Dataset)

• Source: Aggregated and cleaned transcripts from NPR and CNN.
• Filtering: Started with ~487k transcripts. Filtered for dyadic (two-person), informational interviews using Llama-3.1-70b to classify participant roles and content nature.
• Final Size: 45,848 high-quality interviews (approx. 40k unique dyadic interactions).
• Characteristics: Average 7.5 turns per interview. Sources speak significantly more (avg. 551 words) than interviewers (avg. 270 words). Topics span politics, academia, international affairs, and more.

B. Counterfactual Analysis (Offline Evaluation)

To assess LLM behavior without running full simulations, the authors performed a counterfactual simulation:

• Setup: Given the first $t-1$ turns of a human interview, the LLM is prompted to generate the $t$-th question.
• Prompting Strategies:
  1. Baseline: Direct generation.
  2. Chain-of-Thought (CoT): Reasoning about info gaps before generating.
  3. Outline: Providing interview goals/outlines to the LLM.
  4. Outline + CoT: Combining both.
• Evaluation Metrics:
  • Consistency: Exact match, informational, motivational, stylistic, contextual, and discourse consistency (evaluated via GPT-4o).
  • Discourse Analysis: Categorizing questions into roles (e.g., Acknowledgement, Follow-up, Outline-level, Broadening, Challenge).

C. The "NewsInterview" Game Environment

To address the lack of long-horizon reward signals, the authors built a simulated game where an Interviewer Agent attempts to extract information from a Source Agent.

• Source Personas: Eight distinct personas designed based on journalism literature (e.g., Anxious, Avoidant, Adversarial, Defensive, Straightforward, Poor Explainer, Dominating, Clueless).
• Persuasion Mechanism:
  • The source has a hidden set of "Information Items."
  • The source evaluates the interviewer's Persuasion Level (1–5 scale) based on conversation history and persona.
  • Stochastic Reward: The number of items revealed is determined by a Beta distribution parameterized by the persuasion level. Higher persuasion = higher probability of revealing more items.
• Objective: The Interviewer maximizes the total unique information items extracted over $K$ turns.

3. Key Findings & Results

A. Discourse Analysis Results

• Absence of Grounding: Human journalists use Acknowledgement Statements (e.g., "I see," "That sounds scary") in ~9% of turns to build rapport. LLMs generate close to 0% of these statements across all prompting strategies.
• Lack of Strategic Planning:
  • LLMs are 50% less likely to make acknowledgments and 30% less likely to pivot to higher-level (outline) questions compared to humans.
  • LLMs tend to get "stuck" in Follow-up Questions or drift into Opinion/Broadening Questions rather than steering the interview toward specific goals.
  • Even with Chain-of-Thought or Outlines, LLMs fail to mimic human strategic multi-turn planning.

B. Game Simulation Results

• Source-LLM Validity: Source-LLMs (acting as interviewees) successfully mimic human behavior in information sharing and persuasion sensitivity (correlation with human ratings $r = 0.43$, $p < 0.0001$).
• Interviewer-LLM Failure:
  • Performance: Even top-tier models (GPT-4o) only extract ~50% of available information in the "Full Game" (with persuasion and withholding).
  • Persuasion Deficit: Removing the persuasion mechanic (making sources always responsive) only marginally improved performance (e.g., Llama-3.1-70b went from 42.6% to 45.5%). This suggests the core issue is not just source withholding, but the interviewer's inability to strategically persuade or recognize when to pivot.
  • Persona Sensitivity: Models struggle most with Adversarial and Defensive personas. Interestingly, Adversarial sources are easier to persuade (once engaged) but harder to extract info from initially.
• Reward Dynamics: Reward per turn declines over time, but total reward increases linearly, indicating LLMs continue to extract info but at a diminishing rate due to poor strategic pivoting.

4. Key Contributions

1. High-Quality Dataset: Release of a 40k+ dyadic informational interview dataset from NPR/CNN, addressing the scarcity of large-scale, naturalistic dialogue data.
2. Discourse Analysis: Comprehensive evidence that LLMs lack grounding language (acknowledgments) and strategic multi-turn planning compared to human journalists.
3. NewsInterview Simulation: A novel game environment incorporating source personas and persuasion dynamics to serve as a benchmark for long-horizon strategic dialogue.
4. Benchmarking: Demonstration that current LLMs, even large ones, are suboptimal at persuasive, strategic information extraction, highlighting a fundamental gap in current training objectives.

5. Significance and Future Work

• Beyond Chatbots: The paper argues that grounding is not just about "politeness" but is a strategic tool for long-term goals (e.g., in therapy, education, or negotiation).
• Training Implications: Current LLMs are trained on next-token prediction, which fails to capture long-horizon rewards. The authors suggest that NewsInterview provides a necessary playground for developing agents with delayed reward signals and long-horizon planning.
• Applications: The framework can be used to train agents for sensitive domains like mental health support, conflict resolution, and educational tutoring, where building trust and strategic questioning are critical.
• Limitations: The simulation is a simplification of real-world interviews; potential dual-use risks (e.g., using these techniques for social engineering) are acknowledged.

In summary, NewsInterview exposes a critical "grounding gap" in LLMs: while they can generate contextually relevant text, they lack the strategic, empathetic, and persuasive capabilities required for effective multi-turn information extraction. The proposed dataset and game environment offer a path forward for training more capable, human-like dialogue agents.