📊 statistics

Stability and Interrelationships of Classical Test Theory Indicators Under Varying Difficulty Conditions: A Monte Carlo Simulation Study

This Monte Carlo simulation study demonstrates that under structural independence, Classical Test Theory item-level indicators exhibit mathematically determined relationships—such as the perfect functional equivalence of binary item statistics and the stability of discrimination-total correlations across difficulty levels—while test-level reliability remains low due to the absence of inter-item covariance.

Original authors: Ammar Yasser

Published 2026-07-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ammar Yasser

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to taste-test a new recipe. You have a list of tools to measure how good the dish is: a taste meter, a texture gauge, a color scanner, and a "crowd-pleaser" score. In the world of educational testing, these tools are called Classical Test Theory (CTT) indicators. They are the math formulas researchers use to decide if a test question is too hard, too easy, or just right.

This study is like a simulation kitchen. Instead of cooking with real ingredients (real students taking a real test), the researcher, Ammar, used a computer to "cook" 30 test questions for 200 imaginary students. He did this three times, changing the "difficulty" of the ingredients:

The Easy Meal: Questions were very easy (like serving cake).
The Medium Meal: Questions were moderate (like a standard dinner).
The Hard Meal: Questions were very tough (like a spicy challenge).

Crucially, the researcher made sure the "dishes" had no secret connection to each other. In real life, if a student is smart, they might get all the questions right. But in this simulation, getting one question right had nothing to do with getting another right. This allowed the researcher to see how the math formulas themselves behave, stripped of any real-world "smartness" factors.

Here is what the study found, translated into everyday language:

1. The "Twin" Tools (Stable Relationships)

The study found that two of the tools—Item Discrimination (how well a question separates high scorers from low scorers) and Item-Total Correlation (how well a question matches the overall test score)—are like identical twins.

No matter if the questions were easy, medium, or hard, these two tools always moved in perfect sync. If one went up, the other went up.

The Analogy: Imagine you have a thermometer and a heat sensor. They are built using the same internal wiring. If the room gets hot, both numbers go up. They aren't "independently" measuring the heat; they are mathematically locked together. The study found that these tools are "locked together" by their math formulas, not just by chance.

2. The "Chameleon" Tool (Unstable Relationships)

However, the tool that measures Item Difficulty (how many people got it right) acted like a chameleon.

In the Easy group, it had a strong negative relationship with the other tools (like a shadow that gets shorter as the sun gets higher).
In the Medium group, it barely had any relationship at all (like a ghost that disappeared).
In the Hard group, it flipped and had a positive relationship (like a shadow that suddenly appeared on the other side).

The Catch: The researcher admits this "flip-flopping" might be partly because the "Easy" group had a wide range of difficulties, while the "Hard" group had a very narrow range. It's like trying to compare the speed of a race car on a long highway versus a tiny parking lot; the results look different just because the space is different, not because the car changed.

3. The "Mathematical Mirror" (Functional Equivalence)

The study discovered something surprising about four specific measurements: Difficulty, Standard Deviation, Skewness, and Kurtosis.

The Finding: These four are not four different things. They are four different names for the exact same object.
The Analogy: Imagine you have a glass of water. You can measure it by its volume, its weight, the height of the water level, or the pressure at the bottom. If you know the volume, you automatically know the weight, the height, and the pressure. They are mathematically identical.
The Lesson: If you put all four of these into a computer model at the same time, the computer will crash (or give nonsense results) because you are asking it to solve the same puzzle four times. The researcher suggests you just pick Difficulty and ignore the other three, as they are just "mathematical mirrors" of the same data.

4. The "Silent" Moderator

The researcher asked: "Does the difficulty level change how the 'Twin' tools (Discrimination and Correlation) relate to each other?"

The Answer: No. The relationship stayed the same.
The Reason: This isn't because the tools are magically robust in the real world. It's because the math formulas are algebraically determined. It's like asking, "Does the color of the car change the fact that 2 + 2 = 4?" The answer is no, because the math is fixed. The study confirms that these relationships are built into the formulas, not the data.

5. The "Broken" Reliability Score

Finally, the study looked at Reliability (how consistent the test is).

The Result: The scores were very low (around 0.37), which usually means a test is bad.
The Twist: This wasn't a bad test; it was a perfectly designed simulation. Because the researcher made sure the questions had no connection to each other (no "common trait" like general intelligence), the math had to produce a low reliability score.
The Analogy: If you ask a group of people to guess the weather in three different cities, and you make sure their guesses are totally random and unrelated, your "group consistency" score will be zero. That doesn't mean the people are bad guessers; it means you set up the game so they couldn't be consistent. This result simply proved the simulation worked as intended.

The Big Takeaway

This paper is a mathematical reality check. It tells us that many of the numbers we see in test analysis aren't always independent "discoveries" about how students think. Sometimes, they are just mathematical echoes—numbers that move together because the formulas that create them are built from the same ingredients.

The researcher warns us: Don't treat these math formulas as if they are always revealing deep psychological truths. Sometimes, they are just showing us how the calculator works.

Technical Summary: Stability and Interrelationships of Classical Test Theory Indicators Under Varying Difficulty Conditions

Problem Statement
Classical Test Theory (CTT) provides a standard set of quantitative indicators for evaluating test quality, including item difficulty, item discrimination, item-total correlation, and reliability indices. While widely applied, the systematic interrelationships among these indicators and their stability across varying levels of item difficulty remain insufficiently researched. A specific concern is the potential mathematical redundancy among indicators derived from binary response data, where statistics such as standard deviation, skewness, and kurtosis are closed-form functions of item difficulty ( $p$ ). Furthermore, it is unclear whether the relationships between indicators (e.g., discrimination and item-total correlation) are driven by genuine psychometric covariation or by the algebraic structure of the formulas themselves. This study addresses these gaps by isolating the mathematical properties of CTT formulas from the confounding influence of latent trait structures.

Methodology
The study employed a Monte Carlo simulation approach using R to examine CTT indicator formulas under strictly controlled conditions.

Design: Data were generated for 200 examinees responding to a 30-item binary (0/1) test. Crucially, item responses were generated under complete inter-item independence (structural independence) without a latent trait structure. This design choice was deliberate to isolate the deterministic mathematical properties of CTT formulas from psychometric covariation.
Conditions: Three difficulty conditions were simulated:
- Easy: $p = 0.60–0.90$
- Medium: $p = 0.30–0.50$
- Hard: $p = 0.10–0.20$
- Note: The ranges were of unequal widths, a factor the authors acknowledge may introduce range restriction artifacts.
Indicators: Item-level indicators included item difficulty ( $p$ ), item discrimination ( $D$ ), item-total correlation ( $r_{it}$ ), item-total correlation without the item ( $r_{it(woi)}$ ), item reliability ($IR$), and item reliability without the item ( $IR_{woi}$ ). Test-level reliability (Cronbach's alpha and McDonald's omega) was also computed.
Analysis: Spearman rank correlations were computed within each condition with Bonferroni correction ( $\alpha = .0033$ ). Stability was defined descriptively as a maximum absolute difference in correlation ( $|\Delta\rho|$ ) less than 0.20 across conditions. Functional equivalence was tested by correlating difficulty with descriptive statistics. Moderation effects of difficulty level on discrimination-based relationships were tested using hierarchical OLS regression models.

Key Results

Stable Discrimination-Based Relationships: Correlations between item discrimination and item-total correlation indices (including those without the item and reliability indices) were stable across all three difficulty conditions ( $\rho = 0.69–0.88$ ). The authors attribute this stability to the shared mathematical dependence of these indicators on item variance and total score composition, rather than empirical psychometric robustness.
Unstable Difficulty-Based Relationships: Relationships involving item difficulty as a predictor varied substantially. The correlation between difficulty and discrimination shifted from negative in the easy condition ( $\rho = -0.58$ ) to negligible in the medium condition ( $\rho = -0.06$ ) and positive in the hard condition ( $\rho = +0.36$ ). The authors note this variation is partially confounded by the unequal width of the difficulty ranges.
Functional Equivalence: Item difficulty, standard deviation, skewness, and kurtosis demonstrated complete functional equivalence ( $\rho = \pm 1.00$ ) across all conditions. This confirms that in binary data, these are mathematically interchangeable expressions of the same variable ( $p$ ).
No Moderating Effect: Difficulty level did not exert a statistically significant moderating effect on the relationship between item discrimination and other indicators. The authors argue this is algebraically predictable under structural independence, as the functional form of the relationships remains identical across conditions.
Low Reliability Indices: Test-level reliability coefficients (Cronbach's alpha and McDonald's omega) were uniformly low ( $\alpha \approx 0.37–0.41$ ). This is a direct consequence of the simulation design (zero inter-item covariance) and serves only to confirm the null structure of the data, not to evaluate real-test reliability.

Significance and Claims
The paper claims that its primary contribution is the clarification of the algebraic structure of CTT indicator relationships under structural independence.

Mathematical vs. Empirical: The study concludes that the high, stable correlations between discrimination and item-total correlation are "structurally guaranteed" by their definitional interdependence within CTT formulas. They represent a confirmation of known mathematical properties rather than novel empirical discoveries of psychometric robustness.
Methodological Guidance: The finding of complete functional equivalence among difficulty, standard deviation, skewness, and kurtosis provides actionable guidance: researchers should not include these indicators simultaneously in analytical models to avoid perfect multicollinearity. Item difficulty is recommended as the preferred representative.
Interpretation of Stability: The lack of a significant moderating effect by difficulty level is interpreted not as evidence of structural robustness in applied testing, but as an evaluation of an algebraic identity.
Limitations: The authors explicitly state that findings are bounded to the algebraic structure under structural independence. Generalization to applied contexts with latent traits requires replication using latent-trait data-generating models (e.g., IRT). Additionally, the study acknowledges limitations regarding single-replication per condition, unequal range widths, and insufficient sample size for powered correlation analysis after Bonferroni correction.

In summary, the paper serves as a methodological cautionary note, urging researchers to distinguish between relationships that arise from the mathematical necessity of CTT formulas and those that reflect genuine psychometric phenomena.