Beyond the Mean: Three-Axis Fidelity for Aligning LLM-Based Survey Simulators from Small Pilot Data

Technical Summary: Beyond the Mean: Three-Axis Fidelity for Aligning LLM-Based Survey Simulators from Small Pilot Data

1. Problem Statement

Large Language Models (LLMs) are increasingly utilized as proxies for human respondents in social science surveys, ranging from "silicon samples" of voters to fine-tuned models predicting experimental outcomes. However, existing audits reveal systematic pathologies in LLM-generated survey responses:

• Marginal Distortion: Outputs often approximate marginal distributions but flatten variance.
• Structural Attenuation: Predictor–outcome relationships are frequently weakened, with effect-size signs flipping in roughly one-third of cases.
• Homogenization: Minority groups are often structurally distorted or homogenized.
• Sensitivity: Outputs are highly sensitive to prompt perturbations and exhibit high analytic flexibility.

The central research question is: Given a small pilot sample of human responses ($D_p$), can an LLM recover the statistical characteristics of the broader held-out population ($D \setminus D_p$)? The authors argue that LLMs should not be treated as drop-in replacements but as conditional generative models requiring rigorous evaluation against ground-truth data.

2. Methodology

Data and Setup
The study utilizes a COVID-19 misinformation survey conducted in South Korea ($N=1,466$). Each respondent provides a profile ($X_i$) including demographics and psychometric scales, and answers 36 belief items (18 false claims, 18 true claims) on a 4-point Likert scale.

• Pilot: A fixed 5% sample ($n=74$) is used for calibration.
• Evaluation: The remaining 1,392 respondents serve as the held-out ground truth.
• Metrics: Three scalar summaries per respondent are derived: Misinformation mean ($m_i$), True information mean ($t_i$), and a Discernment score ($d_i = t_i - m_i$).

Calibration Pipelines
The authors benchmark four families of approaches:

1. Zero-Shot (ZS): Prompting the LLM with respondent profiles ($X_i$) to predict responses ($Y_i$). Variants include batch prompting (all 36 items at once) and per-item prompting.
2. Few-Shot (FS): Injecting five rows from the pilot $D_p$ as in-context examples into the prompt.
3. Parameter-Efficient Fine-Tuning (PEFT): Fine-tuning Qwen3-8B on the pilot $D_p$ using two configurations:
   • LoRA: Autoregressive token generation.
   • LoRA + MLP: LoRA adapters combined with a trained 5-way MLP classification head (4 Likert classes + "Have Not Seen").
4. PPI Rectification: Applying Prediction-Powered Inference (PPI) to correct the per-item population means of prompt-based simulators using the pilot sample.

Three-Axis Fidelity Evaluation
Recovery is decomposed into three distinct axes, each with specific metrics:

• Structural Fidelity: Do predictor–outcome relations match?
  • Metric: Lin's Concordance Correlation Coefficient (CCC) on bivariate correlations and standardized OLS coefficients between 12 predictors and the discernment score ($d_i$).
  • Decomposition: Sign-agreement rate and magnitude ratio ($|x_{sim}|/|x_{GT}|$).
• Marginal Fidelity: Do simulated marginal distributions match human ones?
  • Metric: Cross-respondent Earth Mover's Distance (EMD) on the distributions of $d_i$, $m_i$, and $t_i$.
• Individual Fidelity: Does each simulated respondent track their human counterpart?
  • Metrics: Relative agreement (Pearson $r_d$ between simulated and ground-truth discernment) and absolute agreement (Mean Absolute Error, $MAE_d$).

3. Key Results

Structural Fidelity

• LoRA + MLP achieved the highest structural fidelity (CCC = 0.85 for bivariate correlations, 0.78 for OLS coefficients).
• ZS-perItem (Zero-Shot, per-item) was the strongest prompt-only competitor (CCC = 0.80), outperforming standard Few-Shot prompting.
• Decomposition: High CCC requires both directional agreement and matched magnitude. LoRA alone severely compressed slopes (attenuation), while LoRA + MLP and FS-perItem showed high sign agreement but LoRA + MLP slightly inflated magnitudes.

Marginal Fidelity

• LoRA + MLP achieved the lowest EMD across all three scalar summaries, indicating the closest match to ground-truth population distributions.
• PPI Rectification: PPI effectively corrected the most biased simulators (e.g., Zero-Shot) but was algebraically degenerate for fine-tuned models. When a model perfectly memorizes the pilot (training error $\to$ 0), the PPI correction term collapses, reducing the method to the simulator's raw held-out mean.

Individual Fidelity

• LoRA + MLP led in relative agreement ($r_d = 0.37$), meaning it best preserved the ranking of respondents by discernment.
• ZS-perItem and FS-perItem achieved the lowest absolute error ($MAE_d = 0.58$), meaning their predicted values were closest to ground truth on average, though they ranked respondents less accurately.
• Limitation: Even the best method explained only ~14% of the cross-respondent variance in discernment, suggesting LLMs are currently unsuitable as direct substitutes for individual-level participant measurement.

Subgroup Fidelity (Pluralistic Alignment)

• Fidelity was not uniform across demographic subgroups. For the best model (LoRA + MLP), structural fidelity (CCC) dropped significantly for specific subgroups (e.g., from 0.85 overall to 0.40 for conservatives). This indicates that aggregate high fidelity can mask severe misrepresentation of specific identity groups.

4. Key Contributions

1. Reframing: Defined LLM survey simulation as a recoverability task, decomposing success into Structural, Marginal, and Individual fidelity axes.
2. Evaluation Protocol: Introduced a calibrated protocol mapping each axis to specific metrics (Lin's CCC, EMD, paired $r_d$/MAE) to detect different failure modes.
3. Empirical Benchmark: Conducted a head-to-head comparison of prompting, PPI rectification, and fine-tuning on a real-world misinformation survey, demonstrating that fine-tuning on small pilot samples offers a balanced approach for achieving multiple forms of fidelity.
4. Subgroup Audit: Highlighted that statistical recovery degrades for certain identity subgroups, raising concerns for pluralistic alignment.

5. Significance and Claims

The paper claims that LLM-based survey simulations should not be treated as "drop-in replacements" for human respondents. Instead, they must be evaluated as conditional generative models.

• Methodological Insight: The choice of output head (autoregressive vs. classification) significantly impacts fidelity; a simple architectural change (adding an MLP head) drastically improved performance over standard LoRA.
• Limitation of Rectification: PPI is a conditional tool that benefits biased simulators but fails for fine-tuned models due to training-set memorization.
• Pluralistic Risk: High aggregate fidelity can mask systematic failures in minority subgroups, necessitating subgroup-specific audits.
• Individual Fidelity Gap: A significant gap remains in individual-level prediction accuracy, suggesting LLMs are currently better suited for estimating population-level statistics than individual psychometric profiles.

The authors conclude that while fine-tuning on small pilots is promising, the field must move toward rigorous, multi-axis evaluation to ensure synthetic data does not amplify pre-existing biases or distort minority perspectives in policy or intervention design.