Language Model Fine-Tuning on Scaled Survey Data for Predicting Distributions of Public Opinions

1. Problem Statement

Public opinion research relies on surveys to gauge societal views, but these processes are costly, time-consuming, and prone to sampling biases. While Large Language Models (LLMs) have been proposed as tools to simulate human responses, current methods face significant limitations:

• Prompt Engineering Limitations: Existing approaches rely on prompting LLMs with demographic descriptions (e.g., "I am a conservative male"). These methods often fail to accurately predict the distribution of responses, instead producing stereotypical or biased outputs that do not reflect the variance within a subpopulation.
• Distributional Misalignment: Off-the-shelf LLMs tend to mirror the opinions of wealthy, educated demographics and struggle to capture the nuanced distribution of opinions across diverse subpopulations (e.g., different races, income levels, or ideologies).
• Data Scarcity: Prior work lacks large-scale, structured datasets specifically designed to fine-tune LLMs on the distribution of survey responses rather than just individual text completions.

The core problem is how to train LLMs to faithfully predict the probability distribution of survey responses for specific subpopulations, enabling researchers to conduct pilot testing and identify sampling needs before launching full-scale human surveys.

2. Methodology

A. Dataset Construction: SubPOP

To address the lack of training data, the authors curated SubPOP (Subpopulation-level Public Opinion Prediction), a significantly scaled dataset:

• Scale: Contains 3,362 questions and 70,000 subpopulation-response pairs. This is 6.5× larger than previous datasets like OpinionQA.
• Sources:
  • Training (SubPOP-Train): Derived from 53 waves of the Pew Research Center's American Trends Panel (ATP) (Waves 61–132), excluding waves used in OpinionQA.
  • Evaluation (SubPOP-Eval): Derived from the General Social Survey (GSS) by NORC, serving as an out-of-distribution (OOD) benchmark to test generalization across different survey institutions.
• Structure: The data consists of triples: (Survey Question, Subpopulation Description, Observed Human Response Distribution).

B. Fine-Tuning Approach

The authors propose directly fine-tuning LLMs to align their predicted response distributions with human survey data.

• Input Format: The model receives a concatenated prompt containing the survey question and the subpopulation description (e.g., "Answer as a person living in the South").
• Output Target: Instead of generating a single token, the model is trained to output a probability distribution over all possible answer choices (e.g., A, B, C, D, E).
• Loss Function: The training objective minimizes the Forward Kullback-Leibler (KL) Divergence between the model's predicted distribution ($p_\theta$) and the empirical human distribution ($p_H$):
  $$\mathcal{L}(\theta) = \mathbb{E}_{q,g} [ D_{KL}(p_H(A_q | q, g) \parallel p_\theta(A_q | q, g)) ]$$
  • Rationale: Forward KL is sensitive to cases where the human data assigns high probability to an option that the model misses, ensuring the model covers the true distribution rather than just the mode.
• Architecture: The authors use LoRA (Low-Rank Adaptation) for efficient fine-tuning on models like Llama-2 (7B/13B), Mistral-7B, and Llama-3-70B.

C. Evaluation Metrics

• Wasserstein Distance (WD): The primary metric used to measure the "cost" of shifting probability mass between the predicted distribution and the human distribution. Unlike "one-hot accuracy" (which only checks the top prediction), WD accounts for the ordinal nature of survey answers (e.g., predicting "Very Likely" when the truth is "Somewhat Likely" is penalized less than predicting "Very Unlikely").
• Baselines: Compared against Zero-shot prompting (QA, BIO, PORTRAY formats), Few-shot prompting, and Modular Pluralism (a multi-LLM collaboration method).

3. Key Contributions

1. SubPOP Dataset: The release of a massive, curated dataset (70K pairs) specifically for training LLMs on subpopulation-level opinion distributions, enabling scale that was previously impossible.
2. Superior Fine-Tuning Strategy: Demonstrating that direct fine-tuning on response distributions significantly outperforms prompt engineering and existing alignment methods.
3. Generalization Capabilities: Showing that models fine-tuned on one survey family (ATP) generalize effectively to:
   • Unseen Topics: New survey waves and subjects not present in training.
   • Unseen Institutions: Different survey families (e.g., GSS vs. ATP).
   • Unseen Subpopulations: Demographic groups not explicitly present in the training set (e.g., specific age brackets).
4. Scaling Laws: Establishing that model performance continues to improve with dataset size, suggesting that collecting more high-quality survey data is a viable path to further gains.

4. Results

• Performance Gains: Fine-tuned models reduced the Wasserstein Distance between LLM predictions and human responses by 32% to 46% compared to the best baseline (Zero-shot prompting) on the OpinionQA test set.
• Distribution Matching:
  • On OpinionQA, the fine-tuned Llama-2-7B achieved a WD of 0.106, compared to 0.173 for zero-shot prompting.
  • On SubPOP-Eval (GSS data), the improvement was even more pronounced, with WD dropping from 0.206 (baseline) to 0.121 (fine-tuned).
• Generalization:
  • The model successfully generalized to unseen subpopulations (e.g., age groups 18-29 or 65+ which were not in the training data), achieving a relative improvement of ~44% over baselines.
  • The model demonstrated steerability: When prompted with a specific demographic, the model's predictions shifted to match that group's distribution, mirroring the "intergroup disagreement" patterns found in real human data.
• Ablation Studies:
  • Explicit Distribution Modeling: Directly modeling the full distribution (using KL loss) significantly outperformed "one-hot" approaches (predicting only the most likely answer) and data augmentation methods (replicating samples).
  • Model Type: Pre-trained models performed better than instruction-tuned (Chat) models for this specific distribution prediction task.

5. Significance and Implications

• Efficient Survey Design: This method allows researchers to use LLMs as a "pilot" tool to estimate response distributions for new questions or under-sampled demographics before spending resources on human data collection.
• Bias Mitigation: By explicitly training on diverse, calibrated survey data, the approach helps mitigate the inherent biases of pre-trained LLMs, making them more representative of specific subpopulations.
• Scientific Utility: The work bridges the gap between NLP and social science, providing a rigorous framework for using synthetic data to augment, rather than replace, human opinion research.
• Future Directions: The authors highlight that while generalization is strong, challenges remain regarding intersectional identities (e.g., specific combinations of race and religion) and temporal shifts in opinion, suggesting a need for continuous data updates and interdisciplinary collaboration.

In summary, the paper establishes that fine-tuning LLMs on large-scale, structured survey data is a superior method for predicting public opinion distributions compared to prompt engineering, offering a scalable and generalizable solution for modern social science research.