Using Zero-Shot LLM-Generated Survey Data for Geographically Explicit Population Synthesis

Technical Summary: Zero-Shot LLM-Generated Survey Data for Geographically Explicit Population Synthesis

Problem Statement
Geographically explicit population synthesis is a critical technique for generating synthetic individual populations by integrating survey data with spatially aggregated marginal constraints (e.g., census tract demographics). These synthetic populations serve as inputs for agent-based models in public health, such as infectious disease transmission and chronic disease dynamics. However, the quality of these synthetic populations is heavily dependent on the availability and quality of input survey data. Many applications face constraints due to limited survey data that are both representative of specific populations and contain the necessary variables for simulation. While Large Language Models (LLMs) have shown promise in generating synthetic human data that mimics real respondents, it remains unclear whether zero-shot LLM-generated survey data can serve as viable inputs for established geographically explicit synthesis workflows, specifically Iterative Proportional Fitting (IPF). The core challenge is not merely the plausibility of individual LLM responses, but whether they preserve the complex multivariate relationships required for downstream synthesis procedures.

Methodology
The study evaluates the feasibility of using zero-shot LLM-generated data as a substitute for real survey data in an IPF-based population synthesis pipeline.

• Data Generation: The authors utilized two state-of-the-art LLMs, GPT-4.1 and Gemini-2.5-Pro, to generate synthetic survey records for two distinct U.S. states: Colorado (CO) and Mississippi (MS). Using a zero-shot prompting strategy (no example records or target distributions provided), the models were instructed to generate responses representative of the 2023 adult populations of these states based on 14 categorical variables from the Behavioral Risk Factor Surveillance System (BRFSS). These variables covered demographics, health insurance, chronic conditions, health behaviors, and BMI.
• Generation Pipeline: To ensure reproducibility and structural integrity, the authors used official APIs with a batch generation approach (75 records per call) and enforced strict JSON schemas. A validation step filtered out records with out-of-range values to prevent bias introduction.
• Population Synthesis: Six synthetic populations were constructed (three per state: one using real BRFSS data as a "ground truth" reference, and two using LLM-generated data). These were fitted to 2019–2023 American Community Survey (ACS) census tract-level demographic constraints (age, race/ethnicity, gender, income, education) using IPF.
• Evaluation: The study employed a multi-stage evaluation:
  1. Distributional Similarity: Jensen–Shannon (JS) divergence was calculated to measure the difference between LLM-generated distributions and BRFSS ground truth.
  2. Synthesis Impact: The JS divergence was recalculated after the IPF step to assess how the synthesis process altered the errors present in the raw generated data.
  3. Spatial Validation: The resulting synthetic populations were aggregated to the census tract level and compared against external benchmarks: ACS estimates for health insurance coverage and CDC PLACES estimates for general health status. Metrics included Pearson's correlation ($r$) and spatial residual analysis.

Key Contributions

1. Geographic Specificity Assessment: Unlike prior work that largely evaluated LLM data at a national level, this study assesses the ability of zero-shot LLMs to approximate state-specific health survey data and capture meaningful geographic contrasts (e.g., between the healthier, wealthier Colorado and the less healthy, lower-income Mississippi).
2. Downstream Workflow Analysis: The paper examines the propagation of errors from the data generation stage through the IPF synthesis process. It specifically investigates whether IPF amplifies or mitigates the biases inherent in LLM-generated data, a question previously unaddressed in the literature.
3. Variable-Dependent Performance: The study highlights that the utility of LLM-generated data is highly variable-dependent, distinguishing between demographics that are easily reproduced (e.g., sex, age) and complex health/economic indicators that are difficult to capture accurately.

Results

• Survey Data Generation: Both LLMs successfully captured major state-level contrasts, reproducing the healthier profile of Colorado and the less healthy profile of Mississippi. However, performance varied significantly by variable. Demographics like sex, age, and exercise were reproduced with near-perfect accuracy (low JS divergence). Conversely, variables such as health insurance, income, heart disease, and smoking status showed higher divergence. Notably, both models struggled with health insurance, often overestimating employer plans and underestimating the uninsured.
• Impact of IPF: The IPF step had mixed effects on data quality. In some cases, synthesis reduced the divergence between LLM data and ground truth (e.g., significant improvements for Mississippi GPT on income and insurance), suggesting IPF can partially regularize upstream errors. In other cases, divergence increased (e.g., BMI for Mississippi GPT), indicating that the fitting process can sometimes amplify existing biases or distort relationships.
• Spatial Validation:
  • General Health: LLM-based populations reproduced spatial patterns of general health reasonably well, particularly in Mississippi, where spatial correlations with external benchmarks were high ($r \approx 0.87$ for Gemini).
  • Health Insurance: LLM-based populations performed poorly in capturing spatial patterns for insurance coverage. They systematically overestimated coverage rates, leading to weak spatial correlations with ACS benchmarks ($r < 0.3$).
  • Comparison to Ground Truth: While BRFSS-based synthetic populations generally provided the most accurate tract-level estimates, LLM-based populations occasionally outperformed the BRFSS-based approach for specific variables (e.g., education and flu vaccination in Mississippi), likely due to the smoothing of noisy real-world survey distributions.

Significance and Claims
The paper concludes that zero-shot LLM-generated survey data shows promise as a supplementary or provisional input for geographically explicit population synthesis but is not yet a viable replacement for high-quality real survey data.

• Contextual Capability: The models demonstrated an ability to generate state-conditioned records that reflect meaningful geographic context rather than generic national averages.
• Limitations: The accuracy of the generated data is strongly variable-dependent. Policy-sensitive, rare, or structurally complex variables (like insurance coverage) remain unreliable under zero-shot conditions.
• Practical Utility: The study suggests that while LLMs can provide a useful starting point when conventional survey data are limited or unavailable, their use requires careful validation. The relationship between input realism and downstream fit is complex; good performance at the survey generation stage does not guarantee accurate synthetic populations, and vice versa. The authors emphasize that future work should explore stronger guidance during generation and validation against less common surveys to fully assess the potential of this approach.