YIELD: A Large-Scale Dataset and Evaluation Framework for Information Elicitation Agents

1. Problem Statement

Current conversational agents (CAs) are predominantly designed for user-driven interactions, where the agent's goal is to satisfy user needs or answer specific queries (e.g., travel booking, coding assistance). However, many critical real-world scenarios—such as academic interviewing, judicial proceedings, and journalistic investigations—require Information Elicitation Agents (IEAs).

In these settings, the agent acts on behalf of an external institutional objective. The agent must proactively guide the conversation to extract specific information from a user (the respondent). Unlike traditional CAs, IEAs face unique challenges:

• High-dimensional action space: Every utterance can drastically shift the dialogue trajectory.
• Subjectivity: Defining "informative" is context-dependent; there is no single optimal question.
• Long-horizon complexity: Agents must maintain coherence over hundreds of turns while adapting to unpredictable or adversarial user replies.
• Data Scarcity: No existing large-scale datasets exist to train or evaluate agents specifically for this proactive information-extraction paradigm.

2. Methodology

A. The YIELD Dataset

The authors introduce YIELD (Information-Elicitation Learning Dialogues), the first large-scale, ethically sourced dataset for IEAs.

• Scale: 2.28 million tokens across 2,281 human-to-human dialogues.
• Domains: Four distinct domains: Academic Interviews, Journalistic Investigations, Judicial Proceedings, and Oral History.
• Characteristics: Unlike standard task-oriented datasets (e.g., MultiWOZ, SGD) which average ~14–20 turns, YIELD dialogues average 171 turns, capturing the complexity of long-form elicitation.
• Construction: Data was acquired from public domain and Creative Commons sources (e.g., Oyez, NARA, JFK archives). The authors performed extensive manual annotation to standardize speaker roles (Elicitor vs. Respondent) and clean heterogeneous formats (PDF, DOCX, HTML).

B. Formalization (POMDP)

The authors formalize information elicitation as a finite-horizon Partially Observable Markov Decision Process (POMDP):

• State ($X_t$): The respondent's hidden information (knowledge, opinions, preferences).
• Action ($A_t$): The elicitor's natural language utterance.
• Observation ($O_{t+1}$): The respondent's reply.
• Representation: Since the hidden state is unbounded, the agent's "belief state" is represented by the token-level hidden representations ($S_t$) of a causal language model processing the conversation history.

C. Training Framework (Offline Reinforcement Learning)

Instead of standard Supervised Fine-Tuning (SFT), the authors employ Offline Reinforcement Learning (ORL) using Advantage-Weighted Regression (AWR).

• Reward Signal: A minimal proxy reward based on factual novelty. The system uses Named Entity Recognition (NER) to reward the elicitor for introducing new entities/facts that the respondent subsequently reveals.
• Optimization: The model is fine-tuned using Low-Rank Adaptation (LoRA). The objective maximizes the likelihood of high-reward utterances weighted by their estimated advantage (return-to-go), encouraging long-horizon strategic behavior rather than just next-token prediction.

D. Evaluation Metrics

The paper proposes three novel metrics tailored to IEAs, moving beyond standard BLEU or task success:

1. Conformity: Measures distributional alignment with human elicitors via Perplexity (on elicitor tokens) and Response Length (token count).
2. Progression ($P$): Quantifies the semantic drift of the conversation. It calculates the average decayed cosine distance between utterance embeddings to ensure the dialogue moves forward rather than looping.
3. Turn-Length Ratio (TLR): Compares the verbosity of the respondent vs. the elicitor. Effective elicitation requires $TLR > 1$ (respondents speak more than the agent).

3. Key Contributions

1. YIELD Dataset: A 26M-token, ethically sourced benchmark covering four domains, significantly larger and longer than existing task-oriented dialogue datasets.
2. IEA Formalization: A POMDP framework for information elicitation that treats the agent as an active information seeker rather than a reactive assistant.
3. Novel Metrics: Introduction of Conformity, Progression, and TLR to evaluate the specific dynamics of elicitation (forward momentum, role asymmetry, and stylistic alignment).
4. ORL Validation: Demonstration that offline reinforcement learning with a minimal novelty reward can compete with or surpass SFT in aligning model behavior with human elicitation strategies.

4. Results

Pilot experiments were conducted on multiple foundation LLMs (Llama-3.1-8B, DeepSeek-R1-Distill, Llama-3.2-3B) comparing Prompt-only, SFT, and ORL approaches.

• Conformity:
  • Perplexity: Fine-tuned models (SFT and ORL) achieved significantly lower perplexity (closer to human data) compared to prompt-only models. For example, Llama-3.1-8B ORL achieved a micro-perplexity of ~12.5 vs. 46.9 for the base prompted model in Academic Interviews.
  • Length: Prompted models generated excessively long responses (e.g., 39.5 tokens vs. human average of 16.9). Fine-tuned models successfully learned to be concise (11–12 tokens), matching human elicitor behavior.
• Progression:
  • Fine-tuned models (SFT/ORL) achieved progression scores close to real human dialogues (0.73–0.75), whereas prompted models lagged (0.63–0.66), indicating they tended to loop on topics rather than advancing the conversation.
• Turn-Length Ratio:
  • Real human dialogues showed a TLR > 1 (respondents speak more). Fine-tuned models successfully reproduced this asymmetry (TLR ~1.8–3.6). Prompted models failed to learn this dynamic, often resulting in TLRs closer to 1 or lower (especially DeepSeek-R1 prompted).
• Human Evaluation:
  • Human raters (via MTurk) found that while prompted models generated "polished" and verbose questions, fine-tuned models aligned more closely with the distribution of real human elicitors, despite receiving slightly lower absolute scores (likely because humans preferred the verbosity of prompted models over the realistic brevity of the fine-tuned ones).

5. Significance

• Paradigm Shift: This work establishes a distinct research track for Information Elicitation Agents, separating them from the dominant "user-service" conversational agent paradigm.
• Data Availability: By releasing YIELD, the authors provide the necessary infrastructure for the community to train agents capable of handling complex, long-horizon institutional tasks (e.g., legal discovery, investigative journalism).
• Training Efficiency: The results suggest that Offline RL with minimal proxy rewards is a viable and effective alternative to SFT for learning strategic dialogue behaviors, potentially reducing the need for massive, high-quality human preference datasets.
• Ethical Framework: The paper explicitly addresses the ethical risks of IEAs (bias, leading questions, power imbalances) and proposes mitigation strategies, including strict oversight and transparency, acknowledging the dual-use nature of agents designed to extract information.

In conclusion, YIELD provides the first comprehensive resource and evaluation framework to advance the development of agents that can effectively and ethically elicit information for institutional goals.