💬 NLP

Large Language Models Are Bad Dice Players: LLMs Struggle to Generate Random Numbers from Statistical Distributions

This paper demonstrates that current large language models fundamentally lack the ability to generate random numbers from specified probability distributions, exhibiting severe statistical failures in both batch and independent sampling modes that propagate systematic biases into downstream applications.

Original authors: Minda Zhao, Yilun Du, Mengyu Wang

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Minda Zhao, Yilun Du, Mengyu Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Magic Dice" That Isn't Magic

Imagine you ask a super-smart robot to roll a die 1,000 times and write down the results. You expect the numbers 1 through 6 to appear roughly the same amount of time (about 166 times each). This is what statisticians call a uniform distribution.

This paper asks a simple but shocking question: Can modern AI models (LLMs) actually do this on their own?

The answer is a resounding no. The researchers found that while these models are amazing at writing stories, coding, and chatting, they are terrible at being random. They don't have a "randomness engine" inside them. Instead, they are just guessing based on patterns they've seen before, which leads to predictable, biased results.

The Two Ways to Ask for Randomness

The researchers tested the AI in two different ways, like asking a friend to roll dice in two different scenarios:

The "Batch" Request (The One-Shot):
- The Prompt: "Please roll a die 1,000 times and list all the numbers in one big list."
- The Result: The AI does okay. It gets about 7% of the distributions right.
- Why? When the AI writes a long list, it can "look back" at what it just wrote. If it rolled too many "6s" in the first 100 numbers, it might subconsciously try to roll more "1s" later to balance the list out. It's like a human trying to "fix" their mistake as they go.
The "Independent" Request (The One-by-One):
- The Prompt: "Roll a die once. Now, roll it again (in a new chat). Now again..." (Repeating this 1,000 times).
- The Result: Total failure. 10 out of 11 models failed completely.
- Why? Without the ability to look back at previous rolls, the AI's internal bias takes over. It has a "favorite number" (like always picking 7 or always picking the middle option) and it can't stop itself from picking it.

The Analogy: Think of the AI as a student taking a test.

In the Batch mode, the student can see their previous answers and say, "Oh, I picked 'C' too many times, I'll pick 'A' now to balance it out."
In the Independent mode, the student is in a different room for every question. They can't see their past mistakes, so they just keep picking their favorite answer, "C," over and over again.

The "Complexity" Problem

The researchers tested the AI with different types of "dice":

Simple Dice: Just picking a number between 1 and 10 (Uniform).
Tricky Dice: Numbers that follow a bell curve (Gaussian) or have weird shapes (Cauchy).

The Finding: The more complex the "dice" is, the worse the AI gets.

Analogy: Imagine the AI is a chef who has cooked millions of simple omelets. If you ask for a simple omelet, they are great. But if you ask for a complex, multi-layered soufflé with a specific chemical structure, the chef panics and just serves you a burnt omelet. The AI can mimic simple randomness because it's seen it in its training data, but it doesn't understand the math behind complex randomness.

The "Longer is Worse" Paradox

Usually, in math, if you take more samples, your results get more accurate.

Normal Expectation: "If I roll the die 10,000 times, the average will be perfect."
AI Reality: "If I ask the AI to roll the die 10,000 times, the results get worse."

The Analogy: Imagine a drunk person trying to walk in a straight line.

If they take 10 steps, they might look okay.
If they take 1,000 steps, they will eventually wander miles off course.
The AI accumulates its tiny biases over time. The longer the list of numbers it has to generate, the more it drifts away from true randomness.

Why Should You Care? (The Real-World Mess)

You might think, "So the AI is bad at math. Who cares?" But this is dangerous because we are starting to use AI to make real-world decisions that need to be random.

1. The Biased Test Maker

Scenario: Schools use AI to generate multiple-choice questions.
The Rule: The correct answer (A, B, C, or D) should be random so students can't cheat by guessing "C" is always right.
The AI Failure: The study found that even when told to be random, the AI heavily favored certain letters (like B or C).
Result: Students could figure out the pattern and pass the test without knowing the material. The test is no longer fair.

2. The Unfair Image Generator

Scenario: A company asks an AI to create 1,000 images of people for a diversity study. They want the images to match real-world demographics (e.g., 50% men, 50% women; specific percentages for different ethnicities).
The AI Failure: The AI ignored the instructions. It generated 97% women, or 0% Hispanic people, or almost everyone wearing a green coat (even though they asked for all colors).
Result: The data is biased. If a doctor or a policy maker uses this data, they will make decisions based on a fake reality that doesn't represent real people.

The Bottom Line

The paper concludes that LLMs do not have a functional "randomness switch" inside them. They are excellent at describing randomness, but terrible at doing it.

The Takeaway:
If you need true randomness for anything important (like cryptography, fair testing, or scientific simulations), do not ask the AI to do it. You must use a dedicated random number generator (like a computer chip or a physical dice roll) and let the AI just write the text around it. The AI is a great writer, but it is a terrible dice player.

1. Problem Statement

As Large Language Models (LLMs) transition from conversational agents to core components of stochastic pipelines (e.g., synthetic data generation, agent simulations, and automated assessment), the ability to faithfully sample from specified probability distributions has become a critical functional requirement.

The paper addresses a fundamental gap: while LLMs can describe randomness linguistically, it is unknown whether they possess an internal functional sampler capable of generating statistically valid random numbers without relying on external code (e.g., numpy.random). Previous studies were limited by small sample sizes ( $N=100$ ), simple distributions, and experimental designs that could not distinguish between context-dependent self-correction and genuine independent sampling.

2. Methodology

The authors conducted the first large-scale, statistically powered audit of native probabilistic sampling in frontier LLMs.

Experimental Design

Models: 11 state-of-the-art models (including GPT-4o, GPT-5.2, Gemini-3, DeepSeek-V3.2, Llama-3.3/4, Qwen3, etc.).
Distributions: 15 probability distributions categorized into three complexity tiers:
- Tier I (Basic): Uniform, Gaussian, Bernoulli.
- Tier II (Bounded/Discrete): Beta, Binomial, Poisson, Exponential.
- Tier III (Complex/Heavy-tailed): Cauchy, Student's t, Chi-Square, F-Distribution, Gamma, Weibull, Laplace, Logistic.
Dual-Protocol Approach: To disentangle failure modes, two protocols were used:
1. Batch Generation (Protocol A): The model generates $N=1000$ samples in a single response. This tests the model's ability to maintain distributional fidelity over a long context window (autoregressive conditioning).
2. Independent Requests (Protocol B): The model makes $N=1000$ stateless calls, generating one sample per call. This isolates the model's intrinsic priors without contextual scaffolding.
Metrics:
- Statistical Validity: Kolmogorov-Smirnov (KS) test for continuous distributions and Chi-square Goodness-of-Fit ( $\chi^2$ ) for discrete distributions (significance level $\alpha = 0.01$ ).
- Fidelity: Wasserstein-1 distance ( $W_1$ ) and KL Divergence to measure geometric distance from the target distribution.
Downstream Tasks:
- MCQ Generation: Testing if models can enforce uniform answer-position constraints (A, B, C, D).
- Text-to-Image Prompt Synthesis: Testing if models can generate prompts with controlled demographic and attribute distributions (Gender, Race, Height, Coat Color).

3. Key Contributions

Systematic Audit: Provided the first large-scale benchmark ( $N=1000$ per distribution) across 15 distributions and 11 models, revealing that current LLMs lack a functional internal sampler.
Protocol Asymmetry Discovery: Demonstrated a sharp asymmetry where batch generation achieves modest validity (via context-based self-correction), while independent requests result in near-total failure.
Complexity and Scaling Analysis: Revealed that sampling fidelity degrades monotonically with distributional complexity and, counter-intuitively, degrades as the sampling horizon ( $N$ ) increases (inverse scaling).
Downstream Bias Propagation: Showed that native sampling failures lead to systematic biases in real-world applications, specifically in MCQ answer positioning and demographic representation in synthetic data.

4. Key Results

A. Protocol Asymmetry

Batch Generation: Models achieved a median pass rate of only 7% across distributions. The best model (GPT-4o) passed 40% of distributions. This suggests that "sampling" in batch mode is an emergent property of autoregressive conditioning (self-correction) rather than true stochasticity.
Independent Requests: The failure was catastrophic. 10 out of 11 models passed 0% of the distributions. Even the simplest Uniform distribution failed, with Wasserstein distances jumping from $\approx 0.01$ (batch) to $\approx 0.15$ (independent).

B. Complexity Stratification

Sampling fidelity is inversely correlated with distribution complexity.
Tier I: Some models showed marginal success.
Tier III (Heavy-tailed/Complex): Performance collapsed completely. No model passed statistical tests for Cauchy or Chi-Square distributions.
Inverse Scaling: As the sample size $N$ increased (from 50 to 2000), the Wasserstein distance ( $W_1$ ) and KL divergence increased rather than converging to zero. Larger sample sizes exposed statistically significant discrepancies that were latent in smaller samples.

C. Downstream Application Failures

MCQ Generation: Despite explicit instructions to distribute correct answers uniformly (25% each), all models exhibited severe positional bias. For example, GPT-OSS-120B placed 54.6% of answers in position C and only 4.5% in A.
Attribute-Constrained Generation: In text-to-image prompt synthesis:
- Demographics: Models systematically violated target distributions (e.g., GPT-4o generated 75% Male vs. 49.5% target; Llama-4 generated 97.2% Female).
- Continuous Attributes: Models failed to capture variance. For height ( $N(169, 10)$ ), models collapsed the standard deviation to $\sigma \approx 1\text{--}6$ cm (target 10 cm).
- Categorical Attributes: Models exhibited "mode collapse," generating 96% of coats as Green (Llama-3.3) or 54% Red (GPT-OSS) despite a uniform 7-color target.

5. Significance and Implications

Lack of Functional Competence: The findings suggest that LLMs have learned to produce linguistic descriptions of randomness but lack the functional competence to simulate stochastic processes internally. They mimic the surface statistics of training data without understanding the underlying mathematical structure.
Necessity of External Tools: For applications requiring statistical guarantees (e.g., scientific simulation, fair AI, randomized controlled trials), relying on native LLM sampling is unsafe. External tools (code interpreters, random number generators) remain mandatory.
Bias in Synthetic Data: The propagation of sampling errors into downstream tasks means that synthetic datasets generated by LLMs may contain systematic, uncorrected biases, potentially reinforcing stereotypes or invalidating experimental results in education and social science.
Architectural Limitation: The "Context-Fidelity Dilemma" implies that increasing context length does not solve the problem; in fact, long-horizon autoregression leads to drift, while stateless calls reveal intrinsic biases.

Conclusion: Current frontier LLMs are fundamentally incapable of acting as reliable random number generators. They require external probabilistic tools to ensure statistical validity in any application involving stochastic sampling.