💬 NLP

The Illusion of Stochasticity in LLMs

This paper demonstrates that while Large Language Models can utilize provided random seeds to generate target distributions, they fundamentally fail to reliably sample directly from specific distributions based on their internal probability estimates, representing a critical limitation for their use as agentic systems.

Original authors: Xiangming Gu, Soham De, Michalis Titsias, Larisa Markeeva, Petar Veličković, Razvan Pascanu

Published 2026-04-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xiangming Gu, Soham De, Michalis Titsias, Larisa Markeeva, Petar Veličković, Razvan Pascanu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Fake Random" Problem

Imagine you hire a brilliant chef (the AI) to run a casino. The casino needs to roll dice, spin a roulette wheel, and shuffle cards. These actions must be truly random so that no one can predict the next move.

The paper argues that while Large Language Models (LLMs) are incredibly smart chefs who can talk about probability, they are terrible at actually doing random things. When asked to pick a number or a direction at random, they don't roll a dice; they guess based on what they've seen before. They create an "Illusion of Stochasticity"—it looks random on the surface, but it's actually full of hidden patterns and biases.

1. The "Knowing-Doing" Gap

The Analogy: Imagine a math genius who can perfectly explain the rules of a dice game but, when asked to roll the dice, always lands on a "6" because they think "6" is the most important number.

The Reality: The paper shows that LLMs understand what a "uniform distribution" (a perfectly fair random choice) is. They can explain it in their "thinking" process. However, when they actually generate the output, they fail.

Example: If you ask an AI to pick a letter from A to Z, it might pick "C" way too often.
Example: If you ask it to pick a number between 1 and 100, it loves the number 42 (a famous number from pop culture) or 7. It's like a person who claims to be random but always chooses the same favorite snack.

2. The "Positional" Bias

The Analogy: Imagine a waiter asking you to pick a dish from a menu. If the menu lists "Burger" first, you pick it. If the waiter moves "Burger" to the bottom, you pick the "Salad" instead. You aren't picking randomly; you're picking based on where the item sits.

The Reality: The AI's "randomness" is often just a reaction to the order of words in the prompt.

If you list options as [Left, Right, Up, Down], the AI picks Up most of the time.
If you shuffle the list to [Up, Down, Left, Right], the AI suddenly starts picking Left or Right.
It's not rolling a dice; it's just following a hidden rule based on the list's order.

3. Why "Thinking Harder" Doesn't Help

The Analogy: Imagine asking a person to guess a random number. You tell them, "Think about it for a long time, write down your reasoning, and then pick a number." They write a beautiful essay about probability, but when they finally say the number, it's still 42.

The Reality: The researchers tested if making the AI "think" (Chain-of-Thought) or changing its "temperature" (how creative vs. strict it is) would fix this. It didn't. Whether the AI is small or huge, smart or simple, it still fails to generate true randomness. The "thinking" process is just a performance; the final output is still biased.

4. The "Tool" Solution (And Its Flaws)

The Analogy: Since the chef can't roll dice, you give them a real, mechanical dice-rolling machine (a computer tool).

Success: If the chef uses the machine correctly, the results are random.
Failure: The chef often forgets to reset the machine between rolls. If they don't reset it, the machine gets stuck on a specific number, and the "randomness" breaks again.

The Reality: When LLMs are allowed to write and run code (using a tool) to generate random numbers, they do much better. However, they often forget to set a "seed" (a starting point for the randomness). If they don't set a new seed every time, the computer generates the exact same "random" number every single time. Only the very smartest models remember to do this correctly.

5. The "Magic Trick" That Actually Works

The Analogy: Imagine you have a magic box that gives you a random number between 0 and 1. You ask the chef to turn that number into a random color.

The Trick: The chef doesn't have to guess the color. They just have to follow a strict recipe: "If the number is between 0.0 and 0.2, pick Red. If 0.2 to 0.4, pick Blue."
The Result: Because the input (0.0 to 1.0) is already random (provided by a computer), and the recipe is a strict math rule, the output (the color) becomes perfectly random.

The Reality: The paper found that LLMs are great at converting a random number they are given into a specific distribution. They are terrible at generating the random number themselves.

Bad: "Pick a random number." (Fails)
Good: "Here is a random number: 0.63. Now, turn this into a random color." (Succeeds)

Why Does This Matter?

If you are building an AI agent that needs to explore the world (like a robot learning to walk or a trading bot), it needs to make random choices to avoid getting stuck in a loop.

If the AI thinks it's being random but is actually predictable, an enemy could trick it.
If the AI is supposed to simulate a fair game, it will cheat by always picking the same "lucky" numbers.

The Bottom Line

LLMs are like actors playing the role of a gambler. They can talk the talk, explain the odds, and even follow a script to convert a random number into a decision. But if you ask them to be the source of randomness, they are just guessing based on their training data, not rolling the dice.

The takeaway: Don't trust an AI to generate randomness on its own. Give it a random number from a real computer, and let the AI do the math to convert it. That's the only way to get a truly fair roll.

1. Problem Statement

As Large Language Models (LLMs) are increasingly deployed as autonomous agents, they are required to act stochastically according to inferred policies (e.g., in multi-armed bandit problems, exploration tasks, or adversarial games). While standard Reinforcement Learning (RL) agents rely on external, mathematically rigorous sampling mechanisms to generate actions from a probability distribution, LLMs must internally emulate this process.

The core problem identified is that LLMs fail to reliably sample from specific probability distributions, even when they explicitly understand the distribution's definition. This creates a "knowing-doing gap": the model may correctly infer a policy requiring stochastic exploration but fails to execute the necessary random sampling, leading to predictable, biased behaviors that compromise agent reliability and security.

2. Methodology

The authors conducted a rigorous empirical analysis across multiple model families (Gemini 2.5/3.0, Qwen3, OLMO-3), varying sizes (from 1.7B to 32B+ parameters), and different sampling strategies.

Experimental Setup:

Independent Sampling: Models were prompted to generate $N=1024$ independent samples from target distributions (Uniform discrete, Uniform continuous, Gaussian) and the empirical distributions were compared to theoretical targets using Goodness-of-Fit (GoF) tests (Chi-Square, Kolmogorov-Smirnov).
Decoding Parameter Ablation: The study tested various temperatures ( $T \in \{0.0, \dots, 2.0\}$ ), top-p, and top-k settings to see if decoding choices could mitigate bias.
Sequential & Batched Sampling:
- Sequential: Models were given previous samples as context (all history vs. last history) to simulate stateful PRNGs.
- Batched: Models were asked to generate a large list of numbers in a single prompt.
Tool-Use & PRNG Simulation:
- Tool-Use: Models were allowed to generate and execute Python code using random libraries.
- Simulation: Models were provided with PRNG algorithm code (e.g., Linear Congruential Generator) and a seed, asked to simulate the algorithm via Chain-of-Thought (CoT) without execution tools.
Distribution Conversion: Models were given a pre-sampled uniform random number $u \in [0,1]$ and asked to deterministically convert it to a target distribution (e.g., Gaussian) using inverse transform sampling.

3. Key Findings & Results

A. Fundamental Failure in Direct Sampling

Bias in Independent Sampling: LLMs consistently fail to sample uniformly or normally. They exhibit strong positional biases (preferring specific values like 7, 42, or the middle of a range) and distributional biases (e.g., placing correct answers in position "C" for multiple-choice questions).
Statistical Significance: Goodness-of-fit tests yielded p-values near zero ( $< 10^{-40}$ ) across all models and distributions, rejecting the hypothesis that LLM outputs follow the target distribution.
Ineffectiveness of Tuning: Adjusting decoding parameters (temperature, top-p) or disabling Chain-of-Thought reasoning did not resolve the bias; in some cases, it amplified specific failures (e.g., Qwen3-8B exclusively outputting "42").

B. Sequential and Batched Limitations

Temporal Bias: Sequential sampling (using history) introduces strong auto-correlation. While "all-history" sampling approximates uniformity better than "last-history," it exhibits a "repulsive" bias (avoiding the previous state). Batched sampling often results in periodic patterns (e.g., repeating sequences every 10 samples) and fails to count the correct number of outputs.
Statefulness Issues: LLMs struggle to maintain the state required for pseudo-random number generation (PRNG) across independent inference calls.

C. The Role of Tools and PRNGs

Tool-Use Dependency: When allowed to run Python code, LLMs can sample correctly only if the random seed is explicitly managed (e.g., random.seed(time.time())). Without this, execution environments often default to fixed seeds, causing deterministic failure.
PRNG Simulation Limits:
- Uniform Distributions: Larger models (e.g., Qwen3-4B+) can simulate PRNG algorithms with >90% accuracy when provided with code and a seed.
- Gaussian Distributions: Simulation fails significantly for Gaussian distributions. The Box-Muller transform requires generating two large random numbers and performing complex multiplications. LLMs fail at the arithmetic of large numbers (e.g., multiplying two 10-digit integers), leading to incorrect Gaussian samples.

D. Distribution Conversion (The "Emergent" Solution)

Deterministic Conversion Works: When provided with a pre-generated random seed (a number from $U[0,1]$ ) and asked to convert it to a target distribution (Uniform or Gaussian) via a deterministic algorithm, LLMs succeed.
Emergent Property: This capability scales with model size. Models >4B parameters can reliably convert $U[0,1]$ to Gaussian distributions, and models >1.7B can handle Uniform discrete conversions.
Mechanism: The models successfully perform inverse transform sampling or bucketization logic in their reasoning traces, proving they understand the math of sampling but cannot generate the randomness internally.

4. Key Contributions

Identification of the "Illusion of Stochasticity": The paper demonstrates that the apparent randomness in LLM outputs is often an artifact of training data biases or logit entropy, not a controlled sampling from a target distribution.
Comprehensive Empirical Benchmark: A large-scale evaluation across model families, sizes, and sampling paradigms (independent, sequential, batched, tool-based).
Distinction between Generation and Conversion: The paper establishes a critical boundary: LLMs are fundamentally flawed at generating random numbers from scratch but are highly capable of converting provided random seeds into target distributions via deterministic reasoning.
Analysis of Failure Modes: Detailed categorization of biases (positional, temporal, arithmetic) and the specific computational bottlenecks (large number multiplication) that prevent PRNG simulation.

5. Significance and Implications

Agentic Reliability: For LLMs acting as agents in complex environments (e.g., game playing, security testing, exploration), the inability to sample stochastically renders policies predictable and exploitable. An agent that cannot explore effectively due to sampling bias may fail to find optimal solutions.
Architectural Recommendations: The authors argue that relying on LLMs to emulate PRNGs is computationally prohibitive and unreliable. Instead, agentic systems should be equipped with stateful external samplers or tools that manage the random state, while the LLM focuses on the deterministic logic of policy execution and distribution conversion.
Future Directions: The work suggests that current Transformer architectures may lack the inductive bias for true stochasticity, potentially requiring specific training objectives (e.g., learning randomized algorithms) or architectural changes to achieve reliable stochastic behavior.

In conclusion, the paper posits that reliable stochastic sampling is a fundamental yet unfulfilled requirement for LLMs, and current frontier models rely on an "illusion" of randomness that breaks down under rigorous statistical scrutiny.