📊 statistics

From Stochasticity to Signal: A Bayesian Latent State Model for Reliable Measurement with LLMs

This paper proposes a Bayesian latent state model that treats LLM classifications as noisy measurements of a true latent state to rigorously quantify uncertainty, recover population-level metrics, and estimate causal impacts, thereby overcoming the limitations of naive single-round or majority-voting approaches in both supervised and unsupervised settings.

Original authors: Yichi Zhang, Ignacio Martinez

Published 2026-04-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yichi Zhang, Ignacio Martinez

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a business owner trying to figure out if your customers are happy with your service. You have thousands of chat logs and phone call transcripts, but reading them all yourself is impossible. So, you hire a super-smart robot (a Large Language Model, or LLM) to read them and tell you: "Happy" or "Unhappy."

Here's the problem: The robot is a bit jittery.

If you ask the robot to read the same chat log five times, it might say "Unhappy" three times and "Happy" two times. It's not lying; it's just that the robot's brain works on probability, like rolling dice. If you just take the first answer, or the "majority vote," you might get a slightly wrong picture of reality. You might think 20% of customers are unhappy when it's actually 30%, or vice versa. This is called measurement error.

This paper, written by researchers at Google, introduces a new way to fix this. They call it a "Bayesian Latent State Model." That's a fancy term for a statistical "truth detector."

Here is how it works, using simple analogies:

1. The "Hidden Truth" vs. The "Noisy Witness"

Imagine a crime scene where the true outcome (Did the customer get their problem solved?) is a hidden treasure buried underground. You can't see the treasure directly.

Instead, you have a team of noisy witnesses (the LLM). You ask the same witness to look at the spot 10 times.

Sometimes they say, "I see a treasure!" (Positive rating).
Sometimes they say, "I see nothing!" (Negative rating).

If you just listen to them once, you might be wrong. If you just take a vote, you might still be slightly off because you don't know how unreliable the witness is.

The Google Model doesn't just count the votes. It acts like a detective who figures out two things at the same time:

How often does this witness lie? (The error rate).
Where is the treasure actually buried? (The true customer satisfaction rate).

By running the robot 10 times and analyzing the pattern of its "lies" and "truths," the model mathematically calculates the real probability of the customer being happy, stripping away the robot's jitteriness.

2. The "Difficulty Score" (The Foggy Window)

Sometimes, the chat logs are easy to read (like a clear sunny day), and sometimes they are messy and confusing (like a thick fog).

The paper suggests asking the robot to also rate how hard the task was for it.

Easy case: The robot is 99% sure.
Hard case: The robot is confused and might flip a coin.

The model uses this "difficulty score" to adjust its detective work. If the robot says "Unhappy" on a super-hard, confusing chat log, the model knows, "Ah, this robot is likely to make a mistake here," and weighs that answer differently than a clear-cut case.

3. The "Human Anchor" (When the Robot is Broken)

What if the robot is so confused on hard cases that it gets the answer wrong more than 50% of the time? It's like a compass that points South when it should point North. Just asking it more times won't help; it will just confidently point the wrong way.

In this scenario, the paper suggests a Semi-Supervised approach.

You take a tiny, random sample of chats (say, 10%) and have a real human read them to get the "Gold Standard" truth.
You use these human answers as anchors (like a lighthouse in a storm).
The model uses the human anchors to "re-calibrate" the robot's compass, correcting its bias so it can accurately guess the truth for the other 90% of chats where no human looked.

Why Does This Matter?

Businesses make big decisions based on data. If you think your customer service is great because the robot said so, but the robot was actually jittery and biased, you might stop spending money on training your staff, and your customers will leave.

This paper provides a mathematical safety net. It turns the robot's "jittery guesses" into reliable, scientific insights. It tells you:

"We are 95% sure that 22% of customers are unhappy."
"The robot makes mistakes 10% of the time on difficult calls."
"Changing our support script actually improved satisfaction by 5%."

The Bottom Line

Think of this paper as a filter for noise.

Old Way: Ask the robot once, take the answer, and hope for the best. (Like listening to a rumor).
New Way: Ask the robot 10 times, check how confused it gets, maybe have a human check a few samples, and use a smart math formula to find the signal hidden inside the noise.

It turns a chaotic, probabilistic robot into a reliable tool for making real-world business decisions.

1. Problem Statement

Large Language Models (LLMs) are increasingly used for automated text classification in business (e.g., customer satisfaction analysis). However, LLMs are inherently stochastic; running the same prompt multiple times can yield different outputs.

The Core Issue: When a deterministic outcome is required (e.g., "Was the customer satisfied?"), this stochasticity introduces measurement error.
Limitations of Current Practices:
- Single-round output: Treats one LLM generation as ground truth, ignoring uncertainty.
- Naive aggregation (Majority Vote/Average): Fails to quantify uncertainty, cannot estimate the LLM's intrinsic error rates, and produces biased estimates of population-level metrics (e.g., prevalence of dissatisfaction) and causal effects.
- Lack of Ground Truth: In many business scenarios, obtaining high-quality human labels for the entire dataset is prohibitively expensive, leaving researchers with only noisy LLM outputs.

2. Methodology: Bayesian Latent State Model

The authors propose a formal statistical framework that treats the true classification as a latent variable and multiple LLM ratings as noisy measurements.

A. Base Model (Unsupervised)

Designed for scenarios where no ground truth labels are available, assuming the LLM is generally better than random guessing (error rates < 50%).

Latent Variable: $D_c \in \{0, 1\}$ represents the true state (e.g., Dissatisfied) for call $c$ .
Observations: $N_c$ independent LLM ratings $R_{c,i}$ for each call.
Parameters Estimated:
1. Base Rate ( $\theta_c$ ): Probability of the outcome (modeled via logit link with covariates and interventions).
2. False Positive Rate ( $\epsilon_0$ ): $P(R=1 | D=0)$ .
3. False Negative Rate ( $\epsilon_1$ ): $P(R=0 | D=1)$ .
4. Causal Impact ( $\eta$ ): The effect of business interventions on the outcome.
Likelihood: A mixture model combining the probability of the latent state and the binomial distribution of errors.
Identifiability: Theorem 1 proves that if $N_c \ge 3$ ratings are collected and $\epsilon_0 + \epsilon_1 < 1$ , the parameters are strictly identifiable.

B. Extension 1: Heterogeneous Error Model

Addresses cases where classification difficulty varies significantly across calls.

Mechanism: Introduces a "difficulty" covariate $H_c$ $H_{c}$ (e.g., LLM self-reported difficulty or transcript length) to model error rates as functions of difficulty:
- $\text{logit}(2\epsilon_{0,c}) = \alpha_0 + \gamma_0 H_c$
- $\text{logit}(2\epsilon_{1,c}) = \alpha_1 + \gamma_1 H_c$
Benefit: Captures correlation between ratings from the same call and handles varying difficulty levels, preventing the "label switching" issue by assuming error rates remain $< 50\%$ .

C. Extension 2: Semi-Supervised Model (With Ground Truth Anchors)

Designed for "hard" cases where the LLM may be directionally biased (error rates $> 50\%$ for difficult subsets), making the unsupervised model unidentifiable.

Approach: Utilizes a small, randomly sampled subset of human-labeled data (anchors) to correct bias.
Likelihood Split:
- Unlabeled Data: Uses the marginalized mixture likelihood.
- Labeled Data: Uses exact conditional likelihood.
Tempering: Applies a weight $W = \lfloor N_U / N_L \rfloor$ to the labeled likelihood to balance the influence of the massive unlabeled dataset against the small labeled set.
Identifiability: Theorem 3 proves that integrating a labeled subset satisfying the "Common Support" assumption guarantees strict global identifiability even when error rates exceed 50%.

3. Key Contributions

Formal Statistical Framework: Moves beyond ad-hoc voting to a rigorous Bayesian Latent Variable Model that jointly estimates population rates, individual probabilities, error rates, and causal effects.
Theoretical Guarantees: Provides formal proofs for the strict identifiability of model parameters under both unsupervised and semi-supervised settings.
Model-Agnostic & Flexible: Applicable to any LLM (open or proprietary) and adaptable to varying task difficulties and data availability (fully unsupervised to semi-supervised).
Practical Guidelines: Proposes a decision framework for practitioners:
- Easy Cases (Error < 50%): Use unsupervised models with multiple inference rounds.
- Complex Cases (Error > 50%): Use semi-supervised models anchored by human labels.

4. Results

Simulation Studies

Study 1 (Simple Classification): The proposed model significantly outperformed naive methods (Single Round, Majority Vote).
- Correlation with True State: Proposed (0.959) vs. Majority Vote (0.915) vs. Naive (0.575).
- Bias Reduction: Naive methods overestimated False Negative rates by >30% and underestimated treatment effects.
Study 2 (Heterogeneous Errors): The heterogeneous extension consistently outperformed the homogeneous base model, especially as the number of inference rounds increased. It accurately recovered impact parameters where Majority Vote failed.
Study 3 (Directional Bias/Semi-Supervised): In scenarios where LLM error rates exceeded 50% (systematic bias):
- Unsupervised and Majority Vote methods failed (Correlation ~0.3–0.4).
- The Semi-Supervised model (using only 10% human labels) achieved a correlation of 0.880 with true states and accurately corrected population prevalence estimates.

Real-World Case Study

Dataset: 14,108 customer support transcripts.
Task: Classify "True Resolution" vs. "Others."
Findings:
- The model estimated a latent "True Resolution" rate of ~22%.
- Heterogeneous Model Insight: Revealed that while the LLM was generally accurate (FPR ~0.65%), it had a high False Negative Rate (10.4%) specifically on difficult/nuanced cases.
- Business Impact: This granular insight allows organizations to route only high-difficulty, ambiguous transcripts to human auditors, optimizing workflow efficiency.

5. Significance

This paper bridges the gap between Learning with Noisy Labels (LNL) and Latent Class Models (LCM) in the context of LLMs. Its significance lies in:

Reliability: It transforms probabilistic, noisy LLM outputs into reliable, quantifiable scientific and business metrics.
Causal Inference: It enables valid causal evaluation of business interventions (e.g., new support workflows) by correcting for measurement error, which naive methods cannot do.
Cost Efficiency: It demonstrates how to achieve high-accuracy measurements with minimal human labeling (via semi-supervised anchoring) or no labeling (via unsupervised multi-round inference), solving the bottleneck of expensive manual review.
Actionable Uncertainty: Unlike point estimates, the Bayesian approach provides credible intervals, allowing businesses to quantify the risk of decisions based on LLM data.