Technical Summary: Silicon Sampling via Cross-Survey Transfer

Problem Statement

Silicon sampling—the use of Large Language Models (LLMs) to simulate human survey respondents—has emerged as a potential solution to declining response rates and rising costs in traditional survey research. However, current evaluation methodologies suffer from a critical limitation: they primarily assess distributional correspondence (comparing aggregate response patterns between LLMs and humans) rather than individual-level prediction accuracy.

This approach risks conflating an LLM's ability to match population-level patterns with its ability to generate coherent, individual-level predictions across different constructs. Existing rigorous tests, such as predicting one survey item based on others, often remain at the distributional level or rely on expensive input modalities (e.g., 2-hour qualitative interviews). Furthermore, most evaluations focus on English-language, Western (WEIRD) contexts, leaving a gap in understanding how silicon sampling generalizes to non-WEIRD, Mandarin-speaking political environments where unique constructs (e.g., cross-strait relations) exist.

Methodology: Cross-Survey Transfer Framework

To address these gaps, the authors propose Cross-Survey Transfer, a rigorous, individual-level evaluation framework designed to test out-of-distribution (OOD) generalization.

Experimental Design

• Dataset: The study utilizes the Taiwan Election and Democratization Study (TEDS) 2024, a nationally representative survey (N=1,206) covering political engagement, institutional trust, democratic values, and cross-strait relations.
• Data Partitioning: Survey items are partitioned into two disjoint sets based on theoretical constructs:
  • Set A (Context): Approximately two-thirds of items per construct, used to build a "persona" for the model. These include demographic profiles and verbatim answers to specific questions.
  • Set B (Prediction Targets): The remaining one-third of items, which the model must predict without prior exposure.
  • Anchoring: Each Set B item is anchored to at least one Set A item measuring a related facet of the same construct, ensuring the model has relevant attitudinal context.
• Prompt Construction: For each respondent, the model receives a prompt containing:
  1. A natural language demographic profile.
  2. Verbatim Set A answers (full question text and labels).
  3. Set B questions with response options.
  4. Population-level response distributions (for base-rate calibration, preventing individual-level data leakage).
• Models Evaluated:
  • Zero-shot LLMs: Three open-weight models (Qwen3.5 27B, gpt-oss 120B, Gemma3 27B) run locally.
  • Abliteration: Safety alignment was removed via directional ablation of the refusal direction to isolate the effects of safety training.
  • Supervised Baselines: Logistic Regression (demographics only) and Random Forest (demographics + Set A) trained via 5-fold cross-validation on the same population. These serve as an upper bound for performance given access to labeled in-domain data.

Metrics

Performance was evaluated at three levels:

1. Item-level: Exact match accuracy, within-±1 accuracy, and Mean Absolute Error (MAE).
2. Construct-level: Accuracy grouped by theoretical constructs (e.g., sovereignty, party competence).
3. Distribution-level: Variance Ratio (VR = $\sigma_{LLM} / \sigma_{Human}$), where values < 1.0 indicate variance collapse.

Key Results

1. Zero-Shot Performance vs. Supervised Baselines (RQ1)

• Accuracy Gap: The Random Forest (RF) baseline, trained on ~960 labeled examples from the target population, achieved 58.3% exact match accuracy.
• LLM Performance: Zero-shot LLMs achieved 52.1% (Qwen3.5), 51.3% (gpt-oss), and 47.6% (Gemma3).
• Significance: Despite operating with zero training examples from the target population, the best LLM closed the gap to within 6 percentage points of the supervised RF. This suggests LLMs possess significant zero-shot generalization capabilities, though they do not yet match supervised models with in-domain data.
• Language Effect: Qwen3.5 (multilingual/Chinese-trained) outperformed the English-centric Gemma3, highlighting the importance of language-specific training data.

2. Construct Predictability Hierarchy (RQ2)

A stable hierarchy of predictability emerged across all models, aligning with Converse's attitude constraint theory:

• High Predictability (>60%): Constructs tightly coupled to partisan identification (Party Competence, Government Performance, Institutional Trust).
• Medium Predictability (40–60%): Constructs requiring nuanced reasoning (Political Participation, Economic Evaluation).
• Low Predictability (<40%): Constructs involving personal judgment or high sensitivity. Notably, the Sovereignty construct (Independence–Unification scale) showed the largest performance gap: RF achieved 56.1% accuracy, while LLMs averaged only 23.3%.

3. Variance Collapse and Safety Alignment (RQ3)

• Variance Collapse: Contrary to the narrative that LLMs uniquely suffer from variance collapse (narrowing of response distributions), the study found that supervised models (RF) also exhibited collapse (VR=0.72). LLMs actually preserved distributional diversity better (VR=0.84–0.85) than the RF baseline, suggesting variance collapse is partly a structural property of the prediction task rather than solely an LLM artifact.
• Alignment Effects: The impact of safety alignment was model-dependent and non-uniform:
  • gpt-oss: Removing alignment decreased accuracy by 6.1 pp and increased variance, suggesting alignment helped calibrate predictions.
  • Gemma3: Removing alignment decreased accuracy by 2.3 pp but significantly reduced variance, suggesting alignment helped maintain diversity.
  • Qwen3.5: Showed negligible change in accuracy or variance.
  • Sovereignty Item: On the sensitive sovereignty item, aligned gpt-oss outperformed its abliterated version, while Gemma3 improved after abliteration. This indicates that alignment effects cannot be generalized across model families.

Key Contributions

1. Novel Evaluation Framework: Introduction of Cross-Survey Transfer, shifting the evaluation standard from distributional correspondence to individual-level, out-of-construct prediction without requiring expensive interview data.
2. Non-WEIRD Context: Empirical validation of silicon sampling in a Mandarin-language, East Asian political context, addressing the over-reliance on Western, English-language datasets.
3. Comparative Baselines: Direct comparison of zero-shot LLMs against supervised machine learning baselines trained on the same population, clarifying the "information asymmetry" between the two approaches.
4. Nuanced Limitation Analysis: Systematic disentanglement of variance collapse and safety alignment effects, demonstrating that these limitations are not inherent to all LLMs but vary by model family and task structure.

Significance and Claims

The paper positions cross-survey transfer as a diagnostic tool to define the boundaries of silicon sampling. The authors claim that:

• Promise: Zero-shot LLMs can achieve meaningful accuracy (approx. 90% of the supervised ceiling) on unseen items, making them viable for exploratory purposes, hypothesis generation, and survey piloting where labeled data is unavailable.
• Boundaries: LLMs are not yet substitutes for statistical models when labeled in-domain data exists. Their performance degrades significantly on constructs involving personal judgment or high political sensitivity (e.g., sovereignty), where attitudinal constraints are weak.
• Methodological Shift: The study challenges the assumption that variance collapse is a unique LLM failure, suggesting it is a broader challenge in predicting individual survey responses. It also argues against universal assumptions about safety alignment, urging researchers to verify alignment effects on a per-model basis.

The authors conclude that while silicon sampling offers unique zero-shot applicability, its current utility for individual-level prediction is modest, and its most promising near-term application lies in hybrid approaches combining LLM-generated priors with smaller human samples.