🤖 AI

Estimating Uncertainty in Classifier Performance with Applications to Large Language Models and Nested Data

This paper evaluates confidence interval methods for classifier performance metrics under conditions typical of social science text classification—such as small sample sizes and nested data—demonstrating that standard approaches often yield inaccurate coverage while recommending superior alternatives like Agresti-Coull, Wilson, and hierarchical bootstrapping to improve transparency and validation in machine learning applications.

Original authors: Kylie Anglin

Published 2026-06-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kylie Anglin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef who has just invented a new recipe for a "Perfect Chocolate Cake." You want to tell the world how good it is. You don't just say, "It's 90% delicious." You need to prove it. So, you invite 20 people to taste it. 18 of them say, "Yes, this is great!"

In the world of computer science and Large Language Models (LLMs), this is exactly what researchers do. They train computers to read text and sort it into categories (like "angry," "happy," or "spam"). To prove their computer is smart, they show it a sample of text that humans have already graded, and they calculate a score, like "Recall" (how many of the "angry" texts did the computer find?).

The Problem: The "Guessing Game" of Confidence
The paper argues that researchers are currently playing a dangerous game of "Guessing Game." They report their score (e.g., "90% accuracy!"), but they rarely report how sure they are about that number.

Think of it like a weather forecast. If a meteorologist says, "There is a 90% chance of rain," that's useful. But if they just say, "It will rain," without telling you if that's based on a satellite image or a guess, you can't trust them.

In this paper, the author, Kylie Anglin, says that when researchers use small groups of people to test their computers, or when the "rain" (the specific text they are looking for) is very rare, the standard ways of calculating "how sure we are" (called Confidence Intervals) are often broken. They are like a ruler that shrinks and stretches unpredictably.

The "Broken Rulers" (Old Methods)
The paper tests several ways to measure this certainty. It finds that the two most common methods used today are like using a ruler made of rubber:

The Wald Interval: This is the "textbook" method. It works fine if you have a huge crowd and average results. But if you have a small group or a very high score (like 95%), this ruler shrinks too much. It tells you, "We are 95% sure the score is between 94% and 96%," when in reality, the true score could be anywhere from 80% to 100%. It gives a false sense of precision.
The Basic Bootstrap: This is a method where you simulate the experiment thousands of times on a computer to see what happens. The paper finds that the standard version of this is also "wobbly" and often underestimates the risk, especially with small groups.

The "Better Rulers" (New Recommendations)
The paper suggests swapping those rubber rulers for sturdier ones that don't stretch as much.

For Independent Texts (The "Solo" Test): If every text comes from a different person and is totally unique, the author recommends using the Agresti-Coull method. Think of this as adding a tiny bit of "safety padding" to your calculation. It's simple, easy to calculate, and it keeps the ruler from shrinking too much.
For Tricky Metrics (Like F1): Some scores are complex combinations of different things (like the F1 score). You can't use the simple formulas for these. The author proposes a new trick called Pseudo-Count Regularization. Imagine you are baking a cake, but you are worried you don't have enough ingredients. You add a "ghost ingredient" (a pretend egg and a pretend cup of flour) to your mix before you start. This stabilizes the recipe so your final result doesn't collapse if you only have a few real ingredients. This method makes the computer simulations much more reliable.

The "Echo Chamber" Problem (Nested Data)
Here is where it gets tricky. In real life, people don't just write one sentence; they write many. A therapist might write 30 notes; a student might write 20 essays. These texts are "nested" inside the person. They aren't independent; they share the same voice, style, and quirks.

The paper explains that if you treat 30 notes from one person as if they were 30 notes from 30 different people, you are cheating. You are tricking yourself into thinking you have more data than you do. It's like asking one friend to taste your cake 30 times and counting that as 30 different opinions. You aren't getting 30 opinions; you're getting one opinion repeated 30 times.

When you ignore this "echo chamber" effect:

Your confidence intervals become too narrow. You think you are 95% sure, but you might only be 65% sure.
The Fix: The paper suggests two main ways to fix this:
1. Adjust the Math: Use a formula that shrinks your "effective sample size" to account for the fact that the texts are related. It's like saying, "We have 30 notes, but because they are all from the same person, they only count as 10 independent opinions."
2. The Hierarchical Bootstrap: Instead of just shuffling individual notes around in a computer simulation, you shuffle the people first, then shuffle the notes inside them. This respects the fact that the notes belong to a specific person.

The Big Takeaway
The main message of the paper is a call for honesty in reporting.

Don't just report the score. If you say your computer is 90% accurate, you must also say, "We are 95% confident the real number is between X and Y."
Use the right tools. Don't use the old, rubber rulers (Wald or basic Bootstrap) for small or high-performing datasets. Use the sturdier ones (Agresti-Coull or the new Pseudo-Count method).
Watch out for clusters. If your data comes from the same people writing multiple times, you need to adjust your math to avoid overconfidence.

By using these better methods, researchers can stop pretending they know more than they do. It helps them realize when they need to collect more data (more people, more texts) to get a truly reliable result, rather than just hoping their small sample size is enough.

Technical Summary: Estimating Uncertainty in Classifier Performance with Applications to Large Language Models and Nested Data

Problem Statement
Researchers increasingly utilize text classification—via supervised models or Large Language Models (LLMs)—to extract constructs from natural language data. While performance metrics such as recall, precision, and F1 are routinely reported to validate these models, they are frequently presented as point estimates without accompanying measures of statistical uncertainty (e.g., confidence intervals or standard errors). When uncertainty is reported, the methods used are often inappropriate for the specific conditions typical of social science text classification: small to moderate sample sizes, infrequent constructs (leading to low counts of positive cases), high performance rates (common with LLMs), and nested data structures (texts clustered within individuals). Default methods, such as the Wald interval and the basic percentile bootstrap, are shown to yield inaccurate coverage, often falling significantly below the nominal 95% level, particularly when sample sizes are small or proportions are extreme. Furthermore, standard interval estimation procedures assume independent and identically distributed (i.i.d.) observations, an assumption violated when texts are nested within individuals, leading to underestimated uncertainty and overly narrow confidence intervals.

Methodology
The paper employs a series of simulation studies to evaluate the coverage and width of various confidence interval methods under conditions typical of text classification.

Independent Data Simulations: The authors simulate binomial proportions ( $p$ ) ranging from 0.5 to 0.95 and sample sizes ( $n$ ) from 10 to 500. They compare analytic intervals (Wald, Wilson, Agresti-Coull, Clopper-Pearson) against bootstrapped intervals (percentile, BCa, and a novel pseudo-count regularized bootstrap).
- Novel Method: A "pseudo-count regularized bootstrap" is proposed. This involves augmenting the original sample with pseudo-observations (e.g., adding one success and one failure for proportions, or adding pseudo-observations to each quadrant of the confusion matrix for F1) before resampling. This mimics the behavior of the Agresti-Coull analytic interval within a bootstrap framework.
- F1 Evaluation: Specific simulations assess F1 score coverage by varying true precision and recall across imbalanced populations (prevalence = 0.10).
Nested Data Simulations: To address non-independence, the authors simulate hierarchical data where texts are nested within individuals.
- Design: A population of 2000 individuals, each producing 60 texts, is generated. Sampling schemes vary between "small corpus" (average ~3 texts per individual) and "moderate corpus" (average ~30 texts per individual).
- Methods Tested: The study compares unadjusted analytic intervals, analytic intervals adjusted for effective sample size ( $n_{eff}$ ) and design degrees of freedom ( $df_{design}$ ), and bootstrapped intervals (cluster bootstrap vs. hierarchical bootstrap).
- Adjustments: For nested data, the study tests adjustments using the design effect (DEFF) based on the intra-class correlation (ICC) and the use of $t$ -quantiles instead of $z$ -quantiles to account for reduced degrees of freedom.

Key Contributions

Systematic Comparison: The paper provides a systematic evaluation of confidence interval coverage specifically tailored to the high-performance, small-sample, and nested-data conditions prevalent in social science text classification.
Identification of Default Failures: It demonstrates that the Wald interval and the basic percentile bootstrap are the least accurate methods, often producing coverage rates as low as 0.40–0.66 in scenarios with small sample sizes and high performance (e.g., $p=0.95$ ).
Novel Regularization: The authors introduce and validate a pseudo-count regularized bootstrap. This method improves coverage for both proportions and non-proportion metrics (like F1) in small samples and high-performance regimes, addressing the erratic behavior of standard bootstrapping.
Nested Data Guidance: The study clarifies that for nested data, simply increasing the number of texts is insufficient; adjustments for effective sample size and design degrees of freedom are necessary. It compares cluster vs. hierarchical bootstrapping, finding that hierarchical bootstrapping is generally more accurate but can be overly conservative when individuals produce very few texts.
Sample Size Planning: The paper provides tables and visualizations illustrating the relationship between sample size, performance rate, and confidence interval width, highlighting the high cost of achieving narrow intervals when performance is high or sample sizes are small.

Results

Independent Data: The Agresti-Coull, Wilson, and Clopper-Pearson analytic intervals consistently achieve coverage near or above the nominal 0.95 level, even with small samples. The pseudo-count regularized bootstrap performs comparably well for proportions and is the only robust option for F1 scores in small samples. The Wald and basic percentile bootstrap consistently under-cover.
Nested Data: Unadjusted intervals (treating texts as independent) severely under-cover (e.g., ~65% coverage when 95% is expected) when texts are nested.
- Adjustments: Intervals adjusted for $n_{eff}$ and $df_{design}$ (specifically using the Agresti-Coull or Wilson formulas with these adjustments) provide accurate coverage.
- Bootstrap Performance: The pseudo-count regularized hierarchical bootstrap provides accurate coverage across conditions. The cluster bootstrap is more precise (narrower intervals) but risks under-coverage, particularly when the number of texts per individual is large (~30) and the number of individuals is small.
Sample Size Sensitivity: The width of confidence intervals is highly sensitive to the number of individuals in nested designs rather than just the total number of texts. Increasing the number of individuals yields greater statistical precision than increasing the number of texts per individual.

Significance and Claims
The paper argues that the magnitude of a performance metric is insufficient without a measure of its precision. The absence of accurate confidence intervals threatens the transparency of machine learning applications in the social sciences, potentially leading to overconfidence in model-based inferences.

The authors claim that their work offers evidence-based recommendations to improve reporting standards:

For independent data, the Agresti-Coull interval is recommended as a default due to its simplicity and accuracy, with the pseudo-count regularized bootstrap as the preferred alternative for non-proportion metrics like F1.
For nested data, analysts must adjust for effective sample size and degrees of freedom. The pseudo-count regularized hierarchical bootstrap is recommended when analytic adjustments are infeasible.
The paper emphasizes the need for researchers to consider sample size requirements at the design stage, particularly noting that rare constructs and high performance levels require careful planning to ensure sufficient "analytic sample sizes" (e.g., the number of positive cases, not just total texts) to support reliable inference.

The authors modestly note limitations, including that their nested simulations assume a single ICC structure and that the paper focuses on uncertainty estimation rather than the unbiased estimation of the metrics themselves. They call for future work to establish benchmarks for common ICC values and performance ranges in psychology to further aid study design.

Technical Summary: Estimating Uncertainty in Classifier Performance with Applications to Large Language Models and Nested Data

More like this