Technical Summary: Instruction-Tuned Language Models Cannot Sample from Distributions They Can Describe

1. Problem Statement

The paper addresses a fundamental flaw in "silicon sampling," a methodology where Large Language Models (LLMs) are used as proxies for human survey respondents. In standard pipelines (e.g., Argyle et al., 2023), a model is prompted with a specific persona and a question, and each model call is treated as an independent draw from that persona's response distribution. These draws are then aggregated to estimate population-level statistics.

The authors argue that this pipeline relies on a false assumption: that instruction-tuned models actually sample from a conditional distribution. Instead, they demonstrate that these models suffer from a categorical collapse, where the same persona on the same question consistently returns the exact same answer across repeated calls. This failure leads to systematic miscalibration, where the aggregated model output fails to capture the variance and distributional shape of real human data.

2. Methodology

The authors employ a multi-layered experimental approach to characterize this failure, trace its origin, and propose mitigations.

A. Characterizing the Sampling Failure

• Synthetic Targets: The authors tested models on controlled synthetic categorical targets (e.g., uniform random integers 1–100, skewed binary distributions, bimodal distributions).
• Logit Analysis: They examined the model's internal logits at temperature zero to determine if the collapse was due to decoding constraints or a deeper structural issue.
• Chain-of-Thought (CoT) Scaffolding: They tested whether algorithmic scaffolding (e.g., asking the model to simulate a random number generator internally) could fix the sampling issue.

B. Tracing the Origin (Base vs. Instruction-Tuned)

• Controlled Comparison: The study compared instruction-tuned models against their pretrained base counterparts across three distinct model families: Llama-3.1-8B (RLHF-based), Mistral-7B-v0.3 (SFT-based, no RLHF), and Qwen2.5-7B (DPO-based).
• Metric: Total Variation (TV) distance between the empirical distribution of 200 calls and the target distribution was used to quantify failure.

C. The KNOWS/DOES Split

• DOES Pathway: The model is asked to produce a single sample per call (standard sampling).
• KNOWS Pathway: The model is asked to describe the entire response distribution in a single call (e.g., via JSON probabilities or a list of exemplars).
• Comparison: The authors compared the accuracy of these two pathways against human survey data (OpinionQA/Pew Research Center).

D. Proposed Mitigations

• Describe Pathway: Using a single call to output the full distribution directly.
• Prompt-Perturbed Argyle (PPA): A modification for applications requiring per-respondent outputs. PPA randomizes surface features of the prompt (option ordering, question phrasing, and persona placement) across calls to induce variation without changing the semantic content.

3. Key Results

The Categorical Collapse

• High Repetition: On synthetic targets, instruction-tuned models produced a single answer in over 94% of calls, regardless of the target distribution's shape (e.g., uniform, bimodal, skewed).
• Logit Gap: The failure is structural. At temperature zero, the gap between the top two logits exceeded 14 nats on some targets. This gap is too large to be bridged by standard API decoding temperatures (max ~2.0), meaning the model is not "sampling" but deterministically selecting a single token.
• Degenerate Primitive: When asked to generate a random number without a specific target, models consistently favored specific values (e.g., "42" for 1–100 integers, "7" for 1–10). This "degenerate sampling primitive" persists even when the model is asked to simulate a random number generator via CoT, as the internal "random" number generated by the model is itself non-uniform.

The KNOWS/DOES Split

• DOES Failure: All "DOES" interventions (standard sampling, CoT, bias framing) failed to produce the target distribution (TV > 0.20).
• KNOWS Success: When asked to describe the distribution in a single call (KNOWS), the model accurately recovered the target shape (TV < 0.05).
• Mechanism: The model possesses the distributional knowledge but cannot enact it through per-call sampling. The "DOES" pathway fails because the model collapses to a single deterministic output; the "KNOWS" pathway succeeds because it bypasses the sampling primitive, effectively converting the task into a counting or specification operation.

Alignment Training as the Cause

• Base vs. Instruct: Base models exhibited significantly lower TV distances (better sampling) than their instruction-tuned counterparts across all three model families.
• Induction: The failure appears in models trained with SFT alone (Mistral) and those with RLHF/DPO, suggesting the failure is induced by alignment training broadly, not just RLHF.

Application to Silicon Sampling (OpinionQA)

• Standard Aggregation: On 100 OpinionQA items, standard persona aggregation resulted in a TV distance of 0.46 from human Pew data. 57% of (persona, item) pairs returned the exact same answer in 50 repeated calls.
• Demographic Matching: Matching personas to actual Pew demographics did not significantly improve results (TV 0.484), ruling out persona mismatch as the primary cause.
• Describe Pathway: A single-call description of the population distribution reduced the TV distance to 0.22, cutting the error by more than half.
• PPA Performance: For applications requiring individual outputs, PPA reduced the TV distance to 0.36 (a 21% relative reduction) at no added API cost.
• Entropy Correlation: The improvement from PPA was directly correlated with an increase in per-call entropy. Non-semantic perturbations (whitespace, punctuation) that did not increase entropy failed to improve accuracy, confirming that the mechanism is the induction of variation, not prompt variation itself.

4. Significance and Claims

The paper claims to identify a "more basic failure" underlying silicon sampling miscalibration: the per-call output of instruction-tuned models is not a sample from a distribution but a collapse to a single output.

• The KNOWS/DOES Split: The authors define a gap where models can accurately describe a distribution (KNOWS) but cannot sample from it (DOES). This is attributed to a degenerate sampling primitive visible in the logits and induced by alignment training.
• Practical Implications:
  • For population estimation, the "describe" pathway is superior to aggregation, offering higher accuracy with lower cost.
  • For per-respondent simulations, PPA offers a cost-free method to recover some variance by perturbing prompt surface features.
  • The assumption that LLM outputs are independent draws from a target distribution must be verified rather than inherited in any aggregate-dependent pipeline (e.g., generative agents, synthetic data generation, LLM-as-judge).
• Limitations: The authors note that the "describe" pathway accuracy serves as a same-session ceiling and may degrade on populations outside the model's training distribution. They also acknowledge that their base-vs-instruct comparison is limited to 8B parameter models due to data availability, though qualitative trends replicate at frontier scales.

In summary, the paper argues that instruction-tuned models have lost the ability to function as stochastic samplers due to alignment training, necessitating a shift in methodology from "sampling and aggregating" to "describing and specifying" or "perturbing to induce entropy."