💬 NLP

Whom to Query for What: Adaptive Group Elicitation via Multi-Turn LLM Interactions

This paper proposes a theoretically grounded, adaptive framework that combines LLM-based question scoring with heterogeneous graph neural network propagation to dynamically select both questions and respondents, thereby improving population-level response prediction under constrained budgets and missing data.

Original authors: Ruomeng Ding, Tianwei Gao, Thomas P. Zollo, Eitan Bachmat, Richard Zemel, Zhun Deng

Published 2026-06-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ruomeng Ding, Tianwei Gao, Thomas P. Zollo, Eitan Bachmat, Richard Zemel, Zhun Deng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to figure out the "vibe" of a huge city. You want to know what the average person thinks about new laws, but you can't talk to everyone. You only have a tiny budget for time and money.

In the past, detectives (or survey researchers) would pick a few random people to ask a few random questions. If they missed someone important, they had to guess what that person thought based on the few people they did talk to. This often led to wrong guesses.

This paper introduces a smarter way to do detective work using two powerful tools: a Super-Brain AI (Large Language Model) and a Social Map (Graph Neural Network).

Here is how their new method works, broken down into simple steps:

1. The Problem: The "Guessing Game" is Expensive

Usually, when you want to know what a group thinks, you have to ask many people many questions. But asking people costs money and time. If you only ask a few people, you end up with a lot of missing information.

Old way: Pick a random person, ask a random question, guess the rest.
New way: Be strategic. Ask the right person the right question so you can figure out what everyone else thinks without asking them.

2. The Two Super-Tools

The authors built a system that uses two specific "brains" working together:

The Super-Brain (LLM): This is like a very smart interviewer. It looks at the conversation history and says, "If I ask this specific question to this specific person, I will learn the most new information." It helps decide what to ask.
The Social Map (Heterogeneous GNN): This is like a giant web connecting everyone. It knows that people with similar backgrounds (like age, job, or location) often think similarly. If you ask one person, this map helps you "fill in the blanks" for their neighbors on the map. It helps decide who to ask and how to guess the answers for the people you didn't ask.

3. How It Works: The "Smart Detective" Loop

The system runs in a loop, round after round:

Pick the Best Target: The system looks at the "Social Map." It finds the people who are most likely to give it new, surprising information (the "hard-to-figure-out" people).
Pick the Best Question: The "Super-Brain" chooses the single question that will teach it the most about the group's overall opinion.
Ask and Learn: It asks that question to that specific person.
Spread the News: Once it gets the answer, it doesn't just write it down. It uses the "Social Map" to update its understanding of everyone else. If Person A (who you asked) thinks "Yes," and Person B is very similar to Person A, the system now has a strong guess that Person B also thinks "Yes," even though it never spoke to Person B.
Repeat: It does this again and again, getting smarter with every single question asked.

4. The Results: Doing More with Less

The authors tested this on real-world data about politics and public opinion (like asking if people support higher taxes or better schools).

The Claim: Their method is much better at guessing the whole group's opinion than old methods, even when they are only allowed to talk to a tiny fraction of people (as low as 10%).
The Proof: In one test, they got over 12% better accuracy than the next best method while only interviewing 1 out of every 10 people.
The Secret Sauce: The paper found that the biggest gains came from asking the "hard" people (those whose opinions are hard to guess). By focusing their limited budget on these specific people, the "Social Map" could accurately fill in the rest of the puzzle.

5. What They Didn't Say

It is important to note what this paper is not about:

It is not about predicting the future or changing people's minds.
It is not a medical tool for diagnosing patients.
It does not claim to work perfectly in every single situation forever; it is a method for making surveys more efficient right now.

In a nutshell: Instead of shouting questions into a crowd and hoping someone answers, this method quietly picks the one person who knows the most, asks the one question that matters, and then uses the connections between people to figure out what the rest of the crowd is thinking. It's like solving a puzzle by placing the most critical pieces first.

Technical Summary: Whom to Query for What: Adaptive Group Elicitation via Multi-Turn LLM Interactions

Problem Statement

Surveys and collective assessments aim to infer latent population properties (e.g., political inclination, skill profiles) by querying a subset of individuals. However, real-world constraints—such as respondent burden, interviewer time, and participation incentives—limit the number of questions and completed responses, often resulting in sparse and partially observed data.

While Large Language Models (LLMs) enable adaptive, multi-turn interactions to optimize what to ask, existing elicitation methods typically assume a fixed respondent pool. They fail to adaptively select who to ask or leverage population structure (e.g., demographics, similarity) when responses are incomplete. This creates a bottleneck where efficiency is not maximized because information gathered from a few respondents does not effectively generalize to the broader group under constrained budgets.

Methodology

The authors propose a framework for Adaptive Group Elicitation, a multi-round setting where an agent jointly selects the next question and a subset of respondents to query under explicit query and participation budgets. The framework operates in a closed loop combining two core components:

1. Meta-Training Phase

LLM for Predictive Inference: A language model (e.g., Llama-3.1-8B) is meta-trained on interaction histories to predict individual responses. This model serves as a robust predictor of individual behavior, enabling the quantification of uncertainty about a latent entity $U$ (e.g., a respondent's true preference) based on observed history. The training objective maximizes the likelihood of autoregressive prediction, avoiding the need for parametric priors over complex latent spaces.
Heterogeneous GNN for Group Simulation: To address data sparsity, a Heterogeneous Graph Neural Network (GNN) is employed. The graph contains three node types:
- Group Members: Representing individuals.
- Features: Representing demographic attributes (e.g., age bins, gender).
- Choices: Representing (question, option) pairs.
  The GNN is trained via link prediction to propagate information across the graph. It learns embeddings that capture relational structures, allowing it to impute missing responses for unqueried individuals based on their similarity to observed respondents.

2. Test-Time Adaptive Inference

At each interaction round $t$ , the system performs a two-stage selection process:

Adaptive Query Selection: The system selects a question $X_t$ that maximizes the Expected Information Gain (EIG) regarding the group's latent entity. Using the meta-trained LLM, it calculates the reduction in predictive entropy (uncertainty) for the held-out evaluation set if a specific question were asked.
Group-Relational Respondent Selection: Instead of querying randomly, the system selects a subset of respondents $R_t$ to maximize diversity and representativeness. It clusters member embeddings from the GNN and selects cluster centers. This ensures the selected subset covers the embedding space, maximizing the utility of the limited budget.
Imputation and Belief Updating: After collecting actual responses from $R_t$ , the GNN updates its graph structure with new edges. It performs message passing to refine node embeddings and impute responses for unqueried members. These imputed responses are added to the interaction history, informing the next round of selection.

Key Contributions

Formalization: The paper formalizes adaptive group elicitation as a joint decision problem over questions and respondents under query and participation budgets, addressing the gap in existing methods that only optimize question selection.
Framework Proposal: It introduces a practical instantiation combining an LLM-based expected information gain objective with heterogeneous GNN propagation. This allows for the imputation of missing responses and the selection of representative respondents based on structured similarity.
Theoretical Justification:
- The approach is grounded in de Finetti's predictive perspective, treating reductions in predictive entropy as reductions in uncertainty about the latent entity.
- The authors prove that their greedy selection algorithm (both joint and two-stage) achieves near-optimality under standard submodularity and monotonicity assumptions, providing a constant-factor guarantee relative to the optimal sequence.
Empirical Validation: The method is evaluated on three real-world opinion datasets (CES, OpinionQA, Twin-2k), demonstrating consistent improvements in population-level response prediction under constrained budgets.

Experimental Results

The framework was tested on three datasets with varying respondent budgets (10%, 30%, 50%).

Performance Gains: The proposed method consistently outperformed baselines (Meta-Random, Meta-Greedy, and Meta-Greedy-Imp). Notably, on the Cooperative Election Study (CES) dataset with a 10% respondent budget, the method achieved a >12% relative gain in prediction accuracy compared to the strongest baseline.
Component Analysis:
- Query Selection: Greedy query selection significantly outperformed random selection.
- Imputation: GNN-based imputation provided substantially larger gains than LLM-only imputation, highlighting the necessity of leveraging population-level structure.
- Respondent Selection: The method showed the highest relative recovery on "hard" (high-sensitivity) respondents—those whose ground-truth responses yield the largest performance improvements. This indicates the strategy effectively targets the most informative individuals.
Planning vs. Greedy: Multi-step planning provided only marginal gains over greedy selection while incurring substantial computational costs, suggesting greedy selection is sufficient for this setting.
Robustness: The approach remained effective across different model scales (Llama-3.1-8B vs. Llama-3.2-1B) and demonstrated generalization across geographic regions (e.g., policies learned on the South region transferring to the West and Midwest).

Significance and Claims

The paper claims that effective elicitation depends not only on what questions are asked but critically on who is observed and how partial information is propagated across the population. By jointly optimizing query selection and respondent selection via structured graph propagation, the framework enables the inference of latent group preferences from a small, informative subset of individuals.

The authors position this work as a principled extension of adaptive elicitation to group settings, moving beyond individual-level inference to handle collective latent entities. They emphasize that their method achieves substantial gains in early elicitation rounds, making it particularly valuable for scenarios with tight budget constraints where traditional surveys often suffer from missing data and breakoffs. The work bridges the gap between LLM-based natural language interaction and structured statistical inference, offering a scalable solution for cost-effective population assessment.