📊 statistics

Asymptotic Standard Errors for Reliability Coefficients in Item Response Theory

This paper proposes a general strategy for deriving asymptotic standard errors that account for both item parameter estimation and sample moment substitution variability, applying this framework to calculate reliable sampling variability estimates for CTT and PRMSE reliability coefficients under the graded response model in item response theory.

Original authors: Youjin Sung, Yang Liu

Published 2026-03-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Youjin Sung, Yang Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to perfect a new recipe. You want to know two things:

How consistent is your cooking? (If you make the dish 100 times, does it taste the same every time?)
How well does the taste reflect the quality of your ingredients? (If the ingredients are high quality, does the dish prove it?)

In the world of psychology and education, this "dish" is a test (like a math exam or a personality survey), the "ingredients" are the questions, and the "taste" is the score a person gets.

This paper is about a new, more precise way to measure the reliability (consistency) of these test scores, especially when the tests are long and complex.

Here is the breakdown of what the authors did, using simple analogies:

1. The Problem: The "Recipe" vs. The "Taste Test"

In the past, statisticians had a good way to measure reliability, but it only worked if they knew the "perfect recipe" (the exact difficulty of every question) beforehand. They assumed the only error came from estimating the recipe itself.

But in the real world, we don't know the perfect recipe. We have to guess the recipe based on a sample of people taking the test.

The Old Way: Only counted the error from guessing the recipe.
The Reality: There are two sources of error:
1. Guessing the recipe (estimating item parameters).
2. The fact that we only tasted a small batch of the soup (using a sample of people instead of the whole world).

The authors realized that previous methods ignored the second source of error, which becomes huge when you have a long test with many possible answer combinations. It's like trying to judge a whole ocean's temperature by dipping a thermometer in one spot, but only accounting for the thermometer's inaccuracy, not the fact that the ocean is vast and variable.

2. The Solution: A "Double-Check" Formula

The authors (Youjin Sung and Yang Liu) invented a new mathematical "recipe" (a formula) that accounts for both sources of error simultaneously.

Think of it like a security system with two locks:

Lock 1: Is the lock mechanism itself sturdy? (Item parameter estimation).
Lock 2: Did we check enough doors to be sure? (Sample moments).

Their new formula calculates the "Standard Error" (a measure of uncertainty) by checking both locks at once. This gives researchers a much more honest "confidence interval"—a range of numbers where the true reliability likely sits.

3. The Two Types of Reliability (The "Direction" of the Question)

The paper focuses on two specific ways to look at reliability, which they call CTT and PRMSE. Here is the difference using a Map vs. Territory analogy:

CTT Reliability (The Map Check):
- Question: "If I look at the Map (the test score), how well does it represent the Territory (the person's true ability)?"
- Analogy: You have a map of a city. You want to know: "If I follow this map, how close will I get to the actual destination?"
- Use Case: You are a teacher. You gave a test (the map) and want to know how well the scores reflect the students' actual knowledge (the territory).
PRMSE (The Territory Check):
- Question: "If I look at the Territory (the person's true ability), how well can I predict it using the Map (the test score)?"
- Analogy: You are standing in the city (the territory). You want to know: "If I use this map to describe where I am, how much of my actual location does it explain?"
- Use Case: You are a researcher studying human potential. You want to know how much of a person's true intelligence can be "captured" or "predicted" by the test questions.

4. The Simulation: Testing the New Recipe

To prove their new formula works, the authors ran a massive computer simulation.

They created thousands of fake tests with different lengths (short, medium, long) and different numbers of students (small class, large university).
They compared their new formula against the "real truth" (which they knew because they made the data up).
The Result: Their new formula was incredibly accurate. It correctly predicted how much the reliability scores would wiggle around due to sampling error, even with relatively small groups of students.

5. The Real-World Test: The SAT Example

They applied their method to real data from a science test (similar to the SAT).

They calculated the reliability of the test scores.
The Old Way: Would just give a single number (e.g., "Reliability is 0.92").
The New Way: Gives a number plus a margin of error (e.g., "Reliability is 0.92, give or take 0.03").
This tells researchers: "We are 95% confident the true reliability is between 0.89 and 0.95." This is crucial for making fair decisions about students or policies.

Why Does This Matter?

Imagine you are a judge deciding if a student gets a scholarship based on a test score.

If you don't know the uncertainty of the test, you might think a score of 90 is perfect.
But with this new method, you realize, "Ah, the test is actually a bit shaky. The true score could be anywhere between 85 and 95."

This prevents you from making unfair decisions based on a number that looks precise but is actually fuzzy.

Summary

This paper is about honesty in measurement.
The authors built a better calculator that admits, "We don't know the exact truth, and our sample isn't perfect." By accounting for both the uncertainty in the test design and the uncertainty in the group of people taking the test, they provide a clearer, more reliable picture of how good a test really is.

In short: They gave researchers a better ruler that comes with a built-in "wiggle room" indicator, so we stop pretending our measurements are perfect when they aren't.

1. Problem Statement

Reliability is a fundamental index of measurement precision in psychometrics. While recent frameworks (Liu et al., 2025b) have unified Classical Test Theory (CTT) reliability and Proportional Reduction in Mean Squared Error (PRMSE) under a regression-based definition, a critical gap remains in quantifying the sampling variability of these coefficients under Item Response Theory (IRT) models.

Existing methods for calculating standard errors (SEs) for reliability coefficients (e.g., marginal reliability, test reliability) rely on the Delta method and assume that sampling variability arises only from the estimation of item parameters. However, in practice, especially for long tests:

Calculating exact population moments (variances) over all possible response patterns is computationally infeasible due to exponential growth in response patterns.
Researchers must substitute population moments with sample moments (empirical reliability).
This substitution introduces a second source of sampling variability (from the sample moments themselves) that existing SE formulas fail to capture, leading to potentially invalid uncertainty quantification.

2. Methodology

The authors propose a general asymptotic framework to derive SEs that simultaneously account for both sources of variability: item parameter estimation and the use of sample moments.

A. General Theoretical Framework

The authors define a family of parameterized sample statistics based on a vector of functions $H(Y_i; \nu)$ , where $\nu$ represents the item parameters.

Estimation: They define the population expectation $\eta(\nu)$ and its sample estimator $\hat{\eta}(\hat{\nu})$ , where $\hat{\nu}$ is the Maximum Likelihood (ML) estimator of item parameters.
Transformation: Reliability coefficients are expressed as a differentiable transformation function $\phi$ applied to these moments (e.g., $\phi(\eta(\nu))$ ).
Asymptotic Normality: Using the Delta method, they derive the asymptotic distribution of the estimated reliability $\phi(\hat{\eta}(\hat{\nu}))$ . The asymptotic covariance matrix $\Sigma(\nu)$ incorporates the Fisher information matrix (from parameter estimation) and the variability of the sample moments.
SE Formula: The asymptotic SE is derived as:
$SE[\phi(\hat{\eta}(\hat{\nu}))] = \sqrt{\frac{1}{n} \nabla\phi(\hat{\eta}(\hat{\nu}))^\top \hat{\Sigma}(\hat{\nu}) \nabla\phi(\hat{\eta}(\hat{\nu}))}$

B. Specific Applications

The theory is applied to two specific estimators under the Graded Response Model (GRM) (which includes the 2-Parameter Logistic model as a special case):

PRMSE for the Latent Variable (LV):
- Defined as the coefficient of determination when regressing the latent score $\Theta_i$ onto observed responses.
- Requires estimating the variance of the Expected A Posteriori (EAP) score and the conditional variance of the LV.
- The authors define a 3-dimensional vector $H$ involving the EAP score, its square, and the conditional variance.
CTT Reliability for the EAP Score:
- Defined as the coefficient of determination when regressing the observed EAP score onto the latent variable.
- Technical Challenge: The numerator requires the variance of the "true score" (conditional expectation), which involves an intractable integral over the continuous latent variable.
- Solution: The authors approximate this integral using numerical quadrature. They re-express the conditional expectation as a marginal expectation (using Bayes' rule) to make it estimable via sample means. This expands the dimension of the $H$ vector to include quadrature weights and terms.

3. Key Contributions

Dual-Source Variability: The paper is the first to provide a general analytical derivation of SEs that accounts for both item parameter estimation error and the error introduced by substituting population moments with sample moments.
Generalizability: The framework is applicable to any fully specified parametric IRT model and any observed or latent score, though the paper demonstrates it specifically for EAP scores under the GRM.
Computational Feasibility: By utilizing numerical quadrature and sample moments, the method allows for the calculation of exact-form reliability coefficients and their SEs for long tests where population moment calculations are impossible.
Practical Implementation: The authors provide the full mathematical derivations and R code (via the mirt package context) for implementing these SEs.

4. Simulation Results

A simulation study was conducted using the 2PL and GRM models with varying sample sizes ( $n=250, 500, 1000$ ) and test lengths ( $m=4$ to $32$).

Accuracy of SEs: The derived asymptotic SEs closely matched the empirical standard deviations of the reliability estimates across all conditions. The discrepancy was negligible (often $<0.002$ ), even for moderate sample sizes ( $n=500$ ).
Point Estimate Bias:
- Small Samples ( $n=250$ ): Both CTT reliability and PRMSE exhibited slight positive bias (overestimation).
- Large Samples: Bias diminished as $n$ increased, aligning with asymptotic theory.
Confidence Interval (CI) Coverage:
- Small Samples: 95% CIs showed undercoverage (e.g., ~92% instead of 95%), particularly for PRMSE. This was attributed to the non-normality (skewness) of the sampling distribution and bias in point estimates, rather than inaccurate SEs.
- Moderate/Large Samples ( $n \ge 500$ ): Coverage rates converged to the nominal 95% level, indicating the SEs and normal approximation are valid for standard research sample sizes.
CTT Reliability Specifics: In extreme conditions (small $n$ , long test), CTT estimates occasionally exceeded 1.0 due to sampling variability, highlighting the need for potential bias-correction or transformation in very small samples.

5. Significance and Implications

Valid Uncertainty Quantification: This study provides researchers with a rigorous method to report confidence intervals for reliability coefficients in IRT, moving beyond point estimates. This is crucial for high-stakes testing and meta-analyses.
Handling Long Tests: The method solves the computational bottleneck of calculating reliability for long tests (e.g., $m > 20$ ) where exact population moments are intractable.
Future Directions: The authors suggest extending this framework to multidimensional IRT models, exploring bias-correction methods (e.g., bootstrap or Fisher z-transformation) for small samples, and applying the theory to other measurement precision indices.

In conclusion, Sung and Liu provide a robust, theoretically grounded solution for estimating the uncertainty of IRT reliability coefficients, bridging the gap between theoretical definitions and practical application in modern psychometric assessment.