💻 computer science

The Dice Roll Method: A StandardizedProtocol for Measuring Stochastic Bias in Large Language Model Output

This paper introduces the "Dice Roll Method," a statistically principled protocol that replaces flawed parametric assumptions with a negative-binomial GLMM framework and simulation-based power analysis to establish standardized iteration counts and reliability thresholds for measuring stochastic bias in large language model outputs.

Original authors: Dmitrij Żatuchin

Published 2026-06-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dmitrij Żatuchin

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out if a specific six-sided die is "fair." You roll it once and get a 4. Is the die broken? Maybe. You roll it again and get a 2. Is it broken? Maybe. To really know if the die is fair, you can't just roll it a few times; you have to roll it many, many times and look at the pattern.

This paper is about doing the exact same thing, but instead of a physical die, we are testing Large Language Models (LLMs) like the ones that power chatbots.

Here is the simple breakdown of what the author, Dmitrij Żatuchin, discovered and proposed.

1. The Problem: The "Digital Die" is Wobbly

When you ask an AI the exact same question twice, it often gives you two slightly different answers. This isn't a bug; it's a feature called stochasticity (randomness). The AI is like a digital die that rolls a new answer every time.

Researchers have been trying to measure if these AIs are biased (e.g., do they recommend different brands to men vs. women?). But they were rolling the "die" too few times. Some researchers rolled it 5 times, others 40 times. They didn't have a rulebook for how many rolls were actually enough to trust the results.

2. The Solution: The "Dice Roll Method"

The author formalized a standard rulebook called the Dice Roll Method. Think of it as a standardized recipe for testing AI fairness. The paper argues that you can't just treat AI answers like simple math problems (like averaging test scores). AI answers are messy, connected, and follow weird patterns.

To fix this, the author built a new "statistical kitchen" with specific tools:

The Right Scale: Instead of using a ruler that assumes everything is a straight line (Gaussian), they use a flexible tape measure (Negative Binomial GLMM) that handles the messy, "bumpy" nature of AI text.
The Right Ruler: Instead of measuring "how far apart" two answers are using old-school math (Cohen's d), they use a new ruler (Cliff's δ) that works better when the data is skewed or has zero answers.
The Right Simulation: Instead of guessing how many times to roll the die, they use a computer simulation (Monte Carlo) to play out thousands of scenarios to find the "sweet spot."

3. The Golden Rules: How Many Times to Roll?

The most practical takeaway is a three-tiered guide on how many times you should ask the AI the same question to get a reliable answer. The author calls these tiers based on how serious your investigation is:

Tier 1: The "Curious Explorer" (5–7 rolls)
- Goal: Just to get a rough idea or describe what's happening.
- Reliability: Low. Good for spotting huge, obvious differences, but you might miss smaller, subtle biases.
Tier 2: The "Fact-Checker" (10–12 rolls)
- Goal: This is the recommended standard for most research.
- Reliability: High. If you want to confidently say, "Yes, this AI is biased," you should aim for this. It catches most real-world biases.
Tier 3: The "Courtroom Lawyer" (15–20 rolls)
- Goal: For rigorous, high-stakes studies where you need to prove even small, subtle biases.
- Reliability: Very High. This is for when you need to be absolutely certain.

4. The Cost of Truth

The paper also looks at the price tag. It turns out, getting a "good" answer isn't that expensive.

Asking an AI 10 times instead of 5 times only costs a few extra dollars in API fees.
The author argues that the small extra cost is worth it because it stops researchers from making false claims based on fluke results.

5. New Tools for the Job

The paper introduces a few new ways to measure "stability" (how consistent the AI is):

The "Fairness Score" (PASOR): Imagine a brand wants to know if they are being treated fairly across different questions. This tool measures if a brand is getting a fair share of attention, regardless of how the question was phrased.
The "Meaning Check" (Embedding Ensemble): Instead of just checking if the words are the same, the paper suggests checking if the meaning is the same using three different "translation" tools (embedding models) to ensure the AI isn't just saying the same thing in different words.
The "Drift Detector": AI models can change slightly over time, even if the version number stays the same. The paper suggests splitting your data into "first half" and "second half" to make sure the AI didn't change its personality while you were testing it.

Summary

This paper is a rulebook for testing AI fairness. It tells researchers: "Stop guessing how many times to ask the AI a question. Use our new, more accurate math tools, and aim for at least 10 rolls to get a trustworthy answer."

It replaces old, rigid math with flexible, realistic tools that understand that AI is a bit like a rolling die: unpredictable, but predictable enough if you roll it enough times.

Technical Summary: The Dice Roll Method for Measuring Stochastic Bias in LLMs

Problem Statement
Researchers increasingly rely on repeated identical prompts to measure stochastic bias in Large Language Model (LLM) outputs. However, the field lacks a standardized protocol for determining adequate iteration counts, selecting appropriate stability metrics, or establishing reliability thresholds. Existing studies often rely on ad-hoc iteration counts (ranging from 5 to 40) and statistical assumptions that violate the nature of autoregressive text generation. Specifically, standard approaches often treat repeated queries as independent, identically distributed (i.i.d.) Gaussian draws, ignoring the conditional dependencies of token generation, the over-dispersed and zero-inflated nature of count data (e.g., brand mentions), and potential temporal drift in model snapshots. This paper addresses these gaps by formalizing the "Dice Roll Method" as a rigorous, standardized audit protocol grounded in the generative mechanics of temperature-scaled nucleus sampling.

Methodology
The paper proposes a revised analytical stack that replaces parametric, normal-theory assumptions with a conditional, non-parametric, mixed-model framework. Key methodological components include:

Generative Model & Variance Decomposition: The protocol is anchored in an explicit model of temperature-scaled nucleus sampling. Total response variance is decomposed into four components: token-level sampling ( $\sigma^2_{tok}$ ), prompt phrasing ( $\sigma^2_P$ ), model differences ( $\sigma^2_M$ ), and model-version/temporal drift ( $\sigma^2_{V,T}$ ).
Inferential Framework:
- Modeling: Replaces independent-samples t-tests with a Negative-Binomial Generalized Linear Mixed Model (GLMM). This accounts for over-dispersion in count data and treats iterations as repeated measures nested within prompts and models.
- Effect Sizes: Replaces Cohen's $d$ with Cliff's $\delta$ (and rank-biserial correlation) as the primary distribution-free effect size, robust to skewness and heavy tails.
- Resampling: Replaces ordinary bootstrap with moving-block bootstrap (to preserve conditional dependence) and parametric bootstrap (for coefficient uncertainty in GLMMs).
- Power Analysis: Replaces classical power tables with simulation-based power analysis using the simr procedure, which fits the specific GLMM structure to the data.
Reliability & Drift:
- Generalizability Theory (G-theory): Uses G-studies and D-studies to decompose variance across prompt, iteration, and model facets, projecting generalizability coefficients ( $G$ ) for different iteration counts.
- Stationarity: Implements explicit drift diagnostics using Kolmogorov–Smirnov (KS) tests on continuous metrics and Population Stability Index (PSI) on discrete counts across disjoint time windows.
Metric Battery: Organizes stability metrics into four families (volume, set composition, distributional shape, semantic consistency) and introduces PASOR (Prompt-Adjusted Share of Recommendation) for fairness-adjusted visibility. Semantic consistency is measured via an ensemble of three embedding models to propagate variance.

Key Results
The authors reanalyzed raw data from five empirical studies (approx. 190,000 observations across three to five LLMs, 270+ brands, and 6 languages) under this revised stack:

Iteration Guidance: Three tiers of iteration counts are established based on Generalizability Theory D-studies:
- Exploratory ( $n=5$ ): $G \approx 0.58$ . Adequate for descriptive reporting of very large effects.
- Confirmatory ( $n=10$ ): $G \approx 0.74$ . Recommended default for most auditing; supports inference on large effects.
- Rigorous ( $n=15$ ): $G \approx 0.81$ . Crosses the 0.80 group-level decision threshold; supports detection of medium effects.
Convergence: Block-bootstrap analysis shows a diminishing-returns pattern. Approximately 77.5% of asymptotic precision is reached at $n=7$ , and 94.4% at $n=8-9$ . The trajectory aligns with a power-law ( $n^{-0.5}$ ) rather than a purely logarithmic decay.
Metric Orthogonality: On the tested "Valentine's Day" subset, structural count-based metrics (CV, Gini, Shannon) collapsed into a single dominant dimension ( $|r| > 0.97$ ). However, the paper argues for a parsimonious three-metric core: one structural metric (Gini or CV), semantic consistency (cosine similarity via embedding ensemble), and fairness-adjusted visibility (PASOR-Gini).
Effect Size Validation: Re-analysis of gender bias effects revealed that while some effects (e.g., Grok, $\delta \approx 0.26$ ) were detectable, others (e.g., GPT, $\delta \approx 0.18$ ) required higher iteration counts for confirmatory power. The use of Cliff's $\delta$ with BCa bootstrap intervals provided more honest uncertainty estimates than previous Cohen's $d$ calculations.
Drift & Cost: Drift diagnostics indicated that stationarity holds for short collection windows (e.g., 2 weeks) but requires monitoring in longer studies. Cost-efficiency analysis suggests that increasing iterations from 5 to 10 yields substantial power gains for a marginal cost increase (e.g., ~$18.75 for a 250-query study).

Significance and Claims
The paper claims to provide the first statistically principled protocol for auditing stochastic LLM behavior that respects the conditional dependencies and non-Gaussian distributions inherent in autoregressive generation. Its primary contributions are:

Formalization: Grounding the Dice Roll Method in an explicit generative model of nucleus sampling.
Statistical Rigor: Replacing flawed normal-theory assumptions with a GLMM-based stack that handles over-dispersion and dependence.
Empirical Guidance: Providing data-driven iteration thresholds (5, 10, 15) tied to generalizability coefficients rather than arbitrary budget constraints.
Reproducibility: Offering a complete, open-source Python implementation (including glmmTMB, simr, and drift diagnostics) to standardize future LLM auditing.

The authors emphasize that this protocol does not invalidate prior findings but refines their interpretation, particularly regarding the statistical power of studies using low iteration counts ( $n=5$ ) and the necessity of reporting distribution-free effect sizes and drift diagnostics. The protocol is presented as a tool for researchers, auditors, and regulators to ensure evidentiary defensibility in AI bias research.