💬 NLP

UnpredictaBench: A Benchmark for Evaluating Distributional Randomness in LLMs

The paper introduces UnpredictaBench, a new benchmark and metric (KS@N) designed to evaluate the ability of large language models to accurately sample from true underlying distributions, revealing that current models struggle significantly with distributional randomness despite their growing use in simulations.

Original authors: Amirhossein Abaskohi, Amirhossein Dabiriaghdam, Liang Luo, Ellie Dingqiao Wen, Lele Wang, Giuseppe Carenini, Peter West

Published 2026-06-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Amirhossein Abaskohi, Amirhossein Dabiriaghdam, Liang Luo, Ellie Dingqiao Wen, Lele Wang, Giuseppe Carenini, Peter West

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a chef to cook a meal for a party. You don't just want one delicious dish; you want a buffet that perfectly represents the entire menu. If the menu says "50% pasta, 30% salad, 20% soup," you expect the buffet to have roughly those proportions.

UNPREDICTABENCH is a test to see if Large Language Models (LLMs) can act like that chef. The researchers wanted to know: Can these AI models generate random outcomes that truly match the "recipe" (the statistical distribution) they are given?

Here is the breakdown of the paper using simple analogies:

1. The Problem: The "Safe" Chef

The authors noticed that while AI models are great at reasoning, they are terrible at being truly random.

The Issue: When asked to simulate a random event (like rolling a die or simulating a stock market crash), the AI tends to get "bored" or "safe." Instead of spreading its answers out to match the real-world chaos, it collapses into a single, predictable answer.
The Analogy: Imagine asking an AI to simulate 100 coin flips. A real coin would give you roughly 50 heads and 50 tails, scattered randomly. The AI, however, might give you 50 heads in a row, or just say "Heads" every time because it thinks that's the "most logical" answer. It fails to capture the unpredictability of the real world.

2. The Test: UNPREDICTABENCH

To fix this, the researchers built a giant test called UNPREDICTABENCH.

The Setup: They created 448 different challenges. Some were simple math problems (like "Give me a number from a Bell Curve"), some were code puzzles, and some were real-world scenarios (like "What happens if two cars arrive at a traffic light at the same time?").
The Goal: They asked the AI to generate 100 samples for each problem.
The Scorecard (KS@100): They used a statistical ruler called the Kolmogorov-Smirnov test (think of it as a "Shape Matcher").
- If the AI's 100 answers looked like the perfect, real-world shape, it got a high score.
- If the AI's answers were clumped together or looked like a flat line, it got a low score.
- The Result: The highest score any model got was only about 32%. This means even the smartest AI is failing to mimic true randomness more than two-thirds of the time.

3. The Findings: Who Failed and Why?

The researchers tested many different models, from tiny open-source ones to massive, expensive corporate models.

The "Collapse": Most models suffered from "mode collapse." Imagine a DJ who is supposed to play a mix of rock, jazz, and pop. Instead, they just play the same rock song on repeat because they think it's the "best" song. The AI does this with numbers; it picks one "plausible" number and sticks to it.
Size Doesn't Matter: Bigger models weren't necessarily better. Some massive models performed worse than tiny ones.
Reasoning Didn't Help: The researchers tried telling the AI to "think harder" before answering. It helped a little, but it didn't fix the core problem. The AI still couldn't generate a truly random spread of numbers.
Instruction Tuning Made It Worse: Models that were fine-tuned to be "helpful" and "safe" actually got worse at being random. Being "helpful" seems to make the AI want to give a single, confident answer rather than a messy, random one.

4. The Metaphor: The Broken Dice

Think of an LLM as a magical dice roller.

What we want: A fair die that lands on 1, 2, 3, 4, 5, and 6 with equal frequency over time.
What the AI does: It rolls the die, sees a "3," and decides, "3 is a very reasonable number. I will roll 3 again." And again. And again.
The Consequence: If you use this AI to simulate a disease outbreak or an economic crash, your predictions will be wrong because the AI isn't simulating the chaos of the event; it's simulating its own confidence in a single outcome.

5. The Conclusion

The paper concludes that current AI models are not ready to replace humans in simulations that require true randomness (like economic modeling or scientific experiments). They are too "deterministic" (predictable).

The researchers created this benchmark to show us exactly where the AI is failing. Until we can teach these models to be truly unpredictable and spread their answers out like a real random process, we cannot trust them to simulate complex, chaotic systems.

In short: The AI is a great storyteller, but a terrible gambler. It knows the rules of the game, but it can't actually play the game randomly.

Technical Summary: UNPREDICTABENCH

Problem Statement

Large Language Models (LLMs) are increasingly deployed as proxies for human behavior and stochastic systems in applications such as economic simulations, epidemiological modeling, and multi-agent environments. While LLMs demonstrate the ability to reason about probability distributions and estimate probabilities, there is a critical gap in their ability to generate samples that faithfully reflect the underlying target distributions. Current models often exhibit a tendency to "collapse" toward a single plausible answer or a narrow range of outputs, failing to capture the true unpredictability and variance inherent in real-world stochastic processes. Existing work on improving output diversity is insufficient for these simulation tasks, which require samples calibrated to a specific target distribution rather than merely varied outputs. There is a lack of systematic evaluation to determine whether LLMs can produce uncertain outcomes that are statistically consistent with ground-truth distributions.

Methodology

To address this gap, the authors introduce UNPREDICTABENCH, a benchmark designed to evaluate the distributional randomness of LLMs.

Benchmark Construction

The benchmark consists of 448 test instances derived from 40 target probability distributions (ranging from unimodal to multimodal, discrete to continuous). The tasks are categorized into seven types to probe distributional understanding across varied representations:

Text Explicit: The distribution and parameters are explicitly named; the model must generate a sample.
Text Implicit: A real-world scenario is described without naming the distribution; the model must infer the stochastic process and sample.
Code Explicit: The model predicts the output of a Python program that directly samples from a distribution (e.g., via NumPy).
Code Implicit: The model predicts the output of a program where the target distribution is induced through transformations (e.g., ratios of gamma variables) without explicit naming.
Multimodal: Sampling from mixtures of two or more component distributions.
Shuffling: Generating a uniform random permutation of a list (encoded via Lehmer codes for evaluation).
Real-World Scenarios: Manually curated tasks involving inherently uncertain environments (e.g., race conditions, MCMC dynamics, network simulations).

Evaluation Metric: KS@N

The primary evaluation metric is KS@N, which quantifies distributional fidelity using the Kolmogorov-Smirnov (KS) statistical test.

Procedure: For a given problem, the model generates $N$ independent samples (typically $N=100$ ). These are compared against a reference set of $M=10,000$ samples drawn from the ground-truth distribution.
Calculation: A two-sample KS test is performed to obtain a p-value. If the p-value exceeds a strict threshold ( $p_{threshold} = 0.0001$ ), the model's samples are not rejected as inconsistent with the ground truth.
Score: KS@N is defined as the fraction of problems where the model's samples are not rejected. A higher score indicates better distributional calibration.
Complementary Metrics: The study also reports Debiased Wasserstein-1 Distance (WDZ) to capture tail behavior and location shifts, and Jensen–Shannon Divergence (JSD) to measure density-level shape mismatches.

Key Results

The authors evaluated a diverse set of open-weight and proprietary models (including GPT-4o/5.4, Claude Sonnet, Llama 3, Qwen, Nemotron, and DeepSeek).

General Performance: No model achieved a KS@100 score above 40%. The best-performing model, Nemotron-3 Super 120B, achieved 32.64%, while many frontier models (e.g., GPT-5.4, Claude Sonnet 4.6) scored below 15%. Most models clustered between 0% and 20%.
Failure Modes: Models fail in two distinct ways: (1) lacking an internal representation of the target distribution, or (2) understanding the rough shape but collapsing to a narrow output range (mode collapse).
Task Difficulty:
- Shuffling and Code tasks were the most challenging, with no model exceeding 40% and several strong performers collapsing to 0%.
- Real-World tasks yielded the highest individual scores for some models (e.g., Llama-3.2-1B reached 59.09%), likely due to narrower effective output ranges in those specific scenarios.
- Multimodal distributions proved significantly harder for models with strong deterministic tendencies, often causing total collapse.
Impact of Interventions:
- Reasoning: Enabling reasoning tokens improved final output performance for some models but revealed that others "know more than they say," as their reasoning tokens contained a broader set of candidates than their final outputs.
- Instruction Tuning: Instruction tuning provided only marginal benefits and, in some cases, actively reduced output diversity compared to base models.
- Temperature: Higher temperatures (e.g., $T=1.2$ ) generally improved KS@100 by increasing diversity, though weaker models sometimes suffered from increased tail deviations.
- List Prompting: Asking models to generate lists of samples (e.g., 10 at a time) improved global coverage for capable models but hurt performance for others (e.g., Llama-3.2-1B).

Significance and Claims

The paper positions UNPREDICTABENCH as a necessary first step toward using LLMs as reliable stand-ins for complex stochastic systems.

Statistical Grounding: Unlike "LLM-as-a-judge" evaluations, UNPREDICTABENCH offers a statistically grounded metric (KS@N) that directly measures distributional fidelity against known ground truths.
Correlation with Utility: The benchmark correlates strongly with utility metrics from creativity benchmarks (NoveltyBench, CREATE), suggesting that distributional fidelity captures a genuine aspect of model quality relevant to creative and diverse generation.
Current Limitations: The authors emphasize that even simple distributional simulation remains a significant challenge for current models. The large gap between current performance and the theoretical ceiling (Random Machine) indicates that LLMs are not yet ready for high-stakes simulation tasks requiring faithful randomness.
Future Roadmap: The benchmark provides a template for future work, offering a scalable difficulty curve (via increasing $N$ ) and a unified testbed for investigating randomness, uncertainty calibration, and the effects of training objectives (e.g., RLHF) on stochastic behavior.

The paper concludes that while LLMs can reason about randomness, their ability to produce it remains fundamentally flawed, necessitating further research into model architectures and training strategies that preserve output diversity without sacrificing calibration.