💬 NLP

Dimensionality and Measurement Precision in HLE's Multiple-Choice Subset

This paper uses psychometric analysis to demonstrate that Humanity's Last Exam (HLE) primarily measures a single general reasoning factor rather than distinct domain-specific capabilities, and that its measurement precision is insufficient to effectively discriminate among frontier language models.

Original authors: Mayank Sharma, Savira Nadela, Tyler Matteson

Published 2026-07-31

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mayank Sharma, Savira Nadela, Tyler Matteson

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher trying to figure out which student in your class is the smartest. You give them a giant, 428-question test that covers everything from ancient history to advanced physics. If a student gets a high score, you might assume they are a genius at everything. But what if you want to know more? What if you want to know if they are a math whiz but terrible at poetry, or a science nerd who can't write a sentence? To answer that, you might look at their "sub-scores" for each subject. This is exactly how we evaluate Artificial Intelligence (AI) today. We use massive tests called "benchmarks" to see how smart our AI models are. But here's the tricky part: just because a test has sections labeled "Math" and "Chemistry" doesn't mean the AI actually has separate "Math brains" and "Chemistry brains." Maybe it just has one giant "Reasoning Brain" that helps it do well on everything. This paper is like a detective story for test designers. It asks: Are these subject labels real, distinct skills, or are they just a fancy way of slicing up a single, general ability? And, even more importantly, is the test actually good at telling the difference between the very best AI models, or does it just get fuzzy when things get really hard?

The paper in question, titled "Dimensionality and Measurement Precision in HLE's Multiple-Choice Subset," dives deep into a famous new AI test called "Humanity's Last Exam" (HLE). The researchers, a team from Stanford, took 29 different AI models and ran them through the text-only, multiple-choice version of this exam, which contains 428 tricky questions. They didn't just count the right answers; they used a special kind of math called Item Response Theory (IRT). Think of this like a super-precise ruler that doesn't just measure height, but also tells you how sharp the ruler is at different heights. They wanted to see two things: first, does the test actually measure eight different skills (like math, biology, history), or is it really just measuring one big "general reasoning" skill? Second, where on the "smartness scale" is this test actually good at making distinctions?

The results were a bit of a reality check for the AI community. The team found overwhelming evidence that HLE is essentially measuring just one thing: a general reasoning ability. They calculated a statistic called McDonald's $\omega_h$ , which came out to a staggering 0.998. To put that in perspective, if 1.0 means "perfectly one single skill," this test is almost perfectly one-dimensional. The labels we put on the questions—like "Math" or "Engineering"—explained only 3.5% of the differences in how models answered. It's like if you took a group of people and tried to sort them by "running speed" and "swimming speed," but you found out that everyone who was good at running was also good at swimming, and the "swimming" label didn't tell you anything new that "running" didn't already tell you. The researchers also found that the ability estimates for specific domains were nearly identical to the total score (correlations of r ≥ 0.81), meaning the sub-scores are basically redundant.

Furthermore, the paper discovered a problem with where the test is most accurate. The test is very good at telling the difference between average AI models and slightly better ones. However, the "precision" of the test drops off a cliff for the very smartest models. The test is most precise around an ability level of $\theta = -0.35$ , but the top-tier "frontier" models sit at $\theta > 0$ . In this high-ability zone, the test becomes blurry. It's like a thermometer that is great at telling the difference between a cold day and a mild day, but when it gets to "boiling hot," the needle just sticks and can't tell you if it's 212°F or 250°F. Interestingly, the "Engineering" section of the test was the only one that kept some of its sharpness for these super-smart models (concentrating 69.1% of its information at $\theta > 0$ ), but since Engineering is only 4% of the whole test, it's not enough to save the day.

So, what does this mean? The authors suggest that we shouldn't treat the different subject scores on HLE as proof that an AI has distinct, separate skills in math or chemistry. Instead, we should probably view the test as a measure of general reasoning. More importantly, as AI models keep getting smarter, this test might stop being useful for telling the very best models apart, because it wasn't designed with enough "hard" questions to distinguish between the absolute giants. The paper doesn't say the test is useless, but it does warn us that we need to be careful not to over-interpret the sub-scores and that we might need new, even harder tests to keep up with the rapid progress of AI.

Technical Summary: Dimensionality and Measurement Precision in HLE's Multiple-Choice Subset

Problem Statement

Humanity's Last Exam (HLE) has rapidly become a prominent benchmark for evaluating frontier large language models (LLMs), organizing its questions into eight subject-domain categories (e.g., mathematics, chemistry, humanities). These domain-specific subscores are frequently interpreted as evidence of distinct, separable latent capabilities. However, this interpretation relies on the untested assumption that the benchmark's categorical labels correspond to empirically distinct latent factors rather than facets of a single general reasoning ability. Furthermore, there is a lack of systematic evaluation regarding the benchmark's measurement precision, specifically whether it can effectively differentiate between models of similar high ability (frontier models) or if score differences merely reflect measurement noise.

Methodology

The study evaluates 29 contemporary LLMs on the text-only multiple-choice subset of HLE, comprising $J = 428$ items (filtered from an initial 513 to exclude zero-variance items and models with poor coverage). The authors apply psychometric methods, specifically Item Response Theory (IRT), to address two primary questions: (1) whether the eight-domain structure reflects distinct latent constructs, and (2) where along the ability continuum the benchmark concentrates measurement precision.

Data Collection and Preprocessing:

Models: 29 models were selected from five families (OpenAI, Anthropic, Google, DeepSeek, and open-weight models) after filtering out models with low response coverage (<95%).
Scoring: Responses were parsed using rule-based extraction and scored as binary (correct/incorrect). Missing data was minimal (0.79%) and coded as incorrect.
IRT Modeling: A two-parameter logistic (2PL) IRT model was fitted to estimate item discrimination ( $a_j$ ) and difficulty ( $b_j$ ) parameters, as well as latent model ability ( $\theta_i$ ).

Analytic Framework:
To assess dimensionality without relying on Confirmatory Factor Analysis (which was precluded by rank-deficient correlation matrices due to the small sample size $N=29$ ), the authors employed four N-robust alternatives:

McDonald's $\omega_h$ : Calculated from 2PL discrimination parameters to quantify the proportion of variance attributable to a general factor.
Principal Component Analysis (PCA): Applied to item response profiles to determine if domain labels explain variance in the principal components.
Residual Correlation Analysis: Compared within-domain and between-domain residual correlations after removing the general factor.
Domain-Level Ability Correlations: Computed correlations between domain-specific ability estimates ( $\hat{\theta}_{domain}$ ) and total ability estimates ( $\hat{\theta}$ ).

Measurement precision was analyzed using the Test Information Function (TIF), decomposed by subject domain, to identify which ability levels ( $\theta$ ) and domains provide the most discrimination.

Key Results

1. Unidimensionality of HLE
The analysis provides convergent evidence that HLE measures a single general reasoning factor rather than distinct domain-specific capabilities:

McDonald's $\omega_h$ : The hierarchical omega was 0.998 (95% CI [0.998, 0.999]), indicating that 99.8% of common item variance is attributable to a single general factor. Domain-specific $\omega_h$ values ranged from 0.936 to 0.994.
PCA: Domain labels explained only 3.5% of the variance in the first three principal components, suggesting no discernible structure beyond the general factor.
Residual Correlations: After removing the general factor, within-domain and between-domain residual correlations were nearly identical (Cohen's $d = 0.016$ ), indicating domain membership explains no additional structure.
Ability Redundancy: Domain-specific ability estimates were highly correlated with total ability estimates across all well-powered domains ( $r \ge 0.81$ , $\rho \ge 0.82$ ), rendering domain subscores near-redundant with the total score.

2. Measurement Precision and Discrimination

Precision Distribution: The Test Information Function (TIF) peaks at $\theta = -0.35$ , coinciding with the ability range of most models in the sample (10th–90th percentile: $[-1.14, -0.26]$ ).
Frontier Limitations: Measurement precision drops sharply for $\theta > 0$ , the region where frontier models are located. This suggests the benchmark is less effective at discriminating among the strongest models.
Domain Contributions: While Biology/Medicine contributed the most total information, Engineering items concentrated the highest proportion of information at the frontier (69.1% at $\theta > 0$ ), despite representing only 4% of the benchmark. Conversely, the largest domains (Mathematics and Humanities/Social Science) contributed disproportionately little to frontier discrimination.

Key Contributions

Empirical Validation of Dimensionality: This is the first psychometric dimensionality analysis applied to HLE, demonstrating that its eight domain labels do not correspond to empirically distinct latent constructs.
Precision Mapping: The study maps the measurement precision of HLE, revealing a critical limitation: the benchmark loses discriminatory power precisely where it is needed most (among frontier models).
Methodological Adaptation: The paper demonstrates the application of N-robust psychometric alternatives (e.g., calculating $\omega_h$ directly from 2PL parameters) in settings where the number of subjects (models) is small relative to the number of items, a scenario inverted from traditional psychometrics.

Significance and Claims

The authors claim that the current practice of interpreting HLE domain subscores as evidence of distinct capabilities is not warranted by the instrument's psychometric properties. Instead, HLE should be treated as a measure of general reasoning.

Furthermore, the findings highlight a "precision gap" for frontier evaluation. As models improve, the benchmark's ability to differentiate between them diminishes because the items are calibrated for moderate ability levels. The paper concludes that while HLE is a challenging benchmark, its utility as a differentiating instrument for the strongest models is limited and will become increasingly constrained as model capabilities advance. The authors emphasize that these conclusions are specific to the text-only multiple-choice subset and call for future work on the full benchmark with larger model samples to enable more rigorous testing (e.g., Confirmatory Factor Analysis).