More Parameters Than Populations: A Systematic Literature Review of Large Language Models within Survey Research

Technical Summary: More Parameters Than Populations

Problem Statement
Survey research, traditionally a human-powered field, is currently undergoing a significant shift due to the rapid integration of Large Language Models (LLMs). While survey research has a history of adopting technological tools for data collection and analysis, the unique affordances and risks of LLMs necessitate a structured examination of their current application. The field is transitioning from conceptual ideation to practical implementation, yet there is a need to systematically assess how LLMs are currently being utilized across the survey research pipeline, identify potential pitfalls, and understand the reciprocal relationship where survey research might improve LLM development.

Methodology
The authors conducted a systematic literature review following the PRISMA 2020 guidelines. The process involved:

• Data Retrieval: Keyword searches were performed across three large-scale databases (arXiv, Semantic Scholar, and Web of Science) using 10 LLM-specific and 62 survey-related keywords.
• Selection Criteria: The initial search yielded 1,766 candidate publications. After deduplication, 1,099 unique papers remained. Papers were excluded if they were not in English or fell outside the six-year publication window (January 2019–January 2025), leaving 965 papers for abstract review.
• Final Corpus: Following abstract review by the research team, 161 papers were deemed eligible. An additional 28 papers were identified through citation network analysis (citing or cited papers not in the initial set), resulting in a final corpus of 189 papers.
• Analysis: One author performed a review of these 189 papers, classifying them based on the phase of the survey process (pre-data collection, data collection, post-data collection) and the specific application of the LLM.

Key Contributions and Results
The paper synthesizes findings by mapping LLM applications onto three broad phases of the survey research process:

1. Pre-Data Collection: This phase covers questionnaire writing, item generation, translation, sampling designs, and recruitment.
   • Findings: Empirical applications are concentrated here. For example, Adhikari et al. (2025) demonstrated using LLMs to simulate cognitive interviews for pretesting cross-cultural questionnaires, adapting U.S. items to South African populations.
2. Data Collection: This phase includes AI-assisted interviews and "silicon sampling" (generating data via models rather than human respondents).
   • Findings: Research includes the creation of "doppelgänger" models (Cho et al., 2024) that fine-tune LLMs on individual conversational data to forecast survey responses. However, the authors note cautionary findings from Bisbee et al. (2024) and Mellon et al. (2024) regarding the risk of LLMs hallucinating responses or misclassifying nuanced political opinions, raising concerns about representational accuracy and construct validity in silicon samples.
3. Post-Data Collection: This phase encompasses data processing, weighting, imputation, summarization, report generation, and dataset curation.
   • Findings: LLMs are being deployed for open-text data classification, such as Li et al. (2024) extracting accessibility sentiment from Google Maps reviews to inform urban design.

Emerging Themes and Tools
The review identified a smaller subset of papers focusing on tools and methods under development rather than fully realized applications. These include:

• Fraud Detection: Methods to identify LLM-generated output to assist in detecting fraudulent respondents (Nathanson et al., 2024).
• Synthetic Data: New frameworks for generating synthetic samples and data using LLMs (e.g., Balog et al., 2024).
• Reciprocal Improvement: The review also noted excluded papers suggesting how survey research can improve LLMs, specifically through the curation of training/benchmark data (Xie et al., 2024) and using surveys to understand industry perceptions of LLMs (Gasparini et al., 2024; Johnson et al., 2024).

Significance and Future Outlook
The paper claims that the current state of LLM application in survey research is uneven. While pre-data collection tasks (like item writing and translation) are well-explored, core practices such as live interviewing, recruitment, and cross-lingual adaptation remain under-investigated, despite LLMs' capabilities as multi-modal and multi-lingual processors.

The authors emphasize that few studies systematically assess LLMs across multiple populations, languages, or topics, leaving questions about generalizability. The significance of this work lies in highlighting that survey research, with its long-standing expertise in assessing error sources and ensuring data quality, is uniquely positioned to guide the responsible and effective integration of LLMs into scientific workflows.

Next Steps
The authors outline plans to implement double-coding on the 189-paper corpus, combining manual coding with AI-based approaches. Future coding will classify papers by theoretical vs. empirical status, specific LLMs used, populations/languages covered, substantive domains, and conditions for success. This deeper analysis aims to reveal how LLM training data, model architecture, and research designs pose challenges to augmenting survey research.