📊 statistics

Human- vs. AI-generated tests: dimensionality and information accuracy in latent trait evaluation

This study compares AI-generated and human-developed versions of the Body Awareness Questionnaire, finding that while surface wording is similar, AI adaptations exhibit significant differences in dimensionality and information accuracy, underscoring the need for rigorous statistical validation of AI-driven measurement tools.

Original authors: Mario Angelelli, Morena Oliva, Serena Arima, Enrico Ciavolino

Published 2026-02-16

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Mario Angelelli, Morena Oliva, Serena Arima, Enrico Ciavolino

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Can an AI Write a Better (or Worse) Test?

Imagine you are a teacher who needs to create a quiz to measure how "aware" students are of their own bodies (like noticing if their heart is racing or if they are tense). You have a perfect, time-tested quiz created by human experts.

Now, you ask a super-smart AI (like ChatGPT) to write a new quiz that looks just like the human one. The AI does its job, and the new quiz looks almost identical on the surface. The words are similar, the questions are about the same topics, and the students seem to answer them just fine.

The paper asks: Is the AI's quiz actually the same as the human one underneath the surface?

The researchers (Mario, Morena, Serena, and Enrico) decided to put both quizzes under a microscope. They didn't just look at the words; they looked at the math behind the answers to see if the AI was secretly measuring something different or doing a sloppy job.

The Analogy: The "Shadow Puppet" vs. The "Real Hand"

Think of the Human-Generated Test as a real hand casting a shadow on a wall. The shape of the shadow perfectly matches the hand because the hand is solid and consistent.

Think of the AI-Generated Test as a puppet made of sticks and cloth. From a distance (or if you just look at the words), the puppet looks exactly like the hand. But if you poke it, the joints move differently. The "bones" (the internal logic) aren't the same.

The researchers found that while the AI's puppet looked like the human hand, its internal structure was wobbly and inconsistent.

What They Found (The Three Big Differences)

1. The "One-Track Mind" vs. The "Confused Mind" (Dimensionality)

The Concept: A good psychological test should measure one main thing (like "Body Awareness").
The Finding: The human test was like a laser beam, focusing on one clear target. The AI test, however, was like a flashlight with a cracked lens. Sometimes it measured body awareness, but sometimes it accidentally measured something else (like general anxiety or confusion).
The Metaphor: If the human test is a straight arrow hitting the bullseye, the AI test is an arrow that wobbles in the air and hits a slightly different spot, even though it was aimed at the same target.

2. The "Difficulty Ladder" (Item Ordering)

The Concept: In a test, some questions are easy, and some are hard. In a good test, the order of difficulty should be consistent. If Question A is harder than Question B for humans, it should be harder for the AI version too.
The Finding: The researchers found the AI scrambled the ladder.
- Human Test: "Question 1 is easy, Question 18 is hard."
- AI Test: "Question 1 is hard, Question 18 is easy."
The Metaphor: Imagine a video game level. The human designers made Level 1 easy and Level 10 hard. The AI designer looked at the list and accidentally made Level 1 a boss fight and Level 10 a tutorial. The names of the levels were the same, but the experience was completely flipped.

3. The "Flashlight Beam" (Information Accuracy)

The Concept: A test is most useful when it gives you the most accurate information about a person. Some tests are great at spotting people who are very aware, but bad at spotting people who are not aware.
The Finding: The human test was like a spotlight. It was very sharp and precise in the middle range (where most people fall). It told the researchers exactly where a person stood.
The AI test was like a floodlight. It spread its light everywhere. It wasn't wrong, but it was "blurry." It gave information over a wide range, but it wasn't as sharp or precise as the human version.
The Metaphor: If you are trying to measure the temperature of a cup of coffee, the human test is a precise thermometer that tells you it's exactly 60°C. The AI test is a guess that says "It's somewhere between 40°C and 80°C." Both are technically "right," but the human one is much more useful.

The "Prompt" Problem

The researchers tried two different ways of asking the AI to write the test:

The Strict Prompt: "Here is the exact list of questions. Rewrite them but keep the meaning." (This made a better AI test, but it still had the "wobbly" internal structure).
The Vague Prompt: "Write a test about body awareness with 18 questions." (This made a much worse test that was confused and inconsistent).

The Lesson: Even if you tell the AI exactly what to do, it doesn't "understand" the concept the way a human expert does. It's just predicting the next word based on patterns it saw in its training data.

Why Should You Care?

This paper is a warning label for the future of research and psychology.

Don't trust the surface: Just because an AI-generated survey looks like a real one doesn't mean it measures the same thing.
The "Black Box" risk: We can't always see why the AI made a question hard or easy. It might be picking up on hidden biases or stereotypes in its training data.
Human Supervision is Key: We can use AI to help write tests, but we must have human experts check the "math" (the internal logic) to make sure the AI isn't hallucinating a fake structure.

The Bottom Line

AI is a powerful tool, like a very fast, very creative intern. But if you ask that intern to build a bridge (a psychological test), you can't just look at the paint job. You have to check the steel beams underneath. In this study, the AI's beams were a bit shaky, even if the paint job looked perfect.

In short: AI can mimic the words of a human test, but it struggles to mimic the soul (the statistical structure) of one. We need to keep humans in the loop to ensure the tests we use are actually telling the truth.

1. Problem Statement

The rapid adoption of Large Language Models (LLMs) like ChatGPT in social and psychological research has introduced the capability to generate measurement instruments (e.g., questionnaires) automatically. While LLMs offer efficiency in translation and adaptation, there is a critical lack of understanding regarding their psychometric validity compared to human-validated instruments.

The core problem addressed is whether AI-generated questionnaires, even when prompted to mimic human-designed ones, accurately capture the underlying latent traits (constructs) they intend to measure. Specifically, the authors investigate:

Do AI-generated items maintain the same dimensionality (unidimensionality vs. multidimensionality) as human-designed tests?
Do AI items possess the same discrimination power and difficulty parameters?
How does the information content (accuracy) of AI-generated tests compare to human tests across the spectrum of latent traits?

2. Methodology

2.1 Experimental Design

The study utilized the Body Awareness Questionnaire (BAQ), a validated 18-item instrument measuring awareness of bodily reactions, as the human-generated benchmark.

Human Version (BAQ): The original validated questionnaire.
AI Version 1 (GPT.v1): Generated using a highly specific prompt that requested a one-to-one rephrasing of the original 18 items.
AI Version 2 (GPT.v2): Generated using a generic prompt requesting a new 18-item psychological test on the same construct without referencing specific original items.

Data Collection:

Sample: $N=178$ participants randomly assigned to the three versions ( $n_{BAQ}=56$ , $n_{v1}=57$ , $n_{v2}=65$ ).
Scale: 7-point Likert scale (1 = "Not at all true" to 7 = "Very true").

2.2 Statistical Framework

The authors employed Item Response Theory (IRT), specifically the Graded Response Model (GRM), within a Bayesian hierarchical framework. This approach was chosen over Classical Test Theory (CTT) to separate item characteristics from respondent traits and to analyze information functions.

Model: Bayesian GRM estimating:
- $\gamma_j$ : Item discrimination parameters.
- $\beta_j$ : Item difficulty parameters.
- $\delta_h$ : Threshold parameters for response categories.
Estimation: Markov Chain Monte Carlo (MCMC) with 20,000 iterations and 3 chains.
Key Metrics Analyzed:
1. Reliability: Cronbach's $\alpha$ , ordinal $\alpha$ , $\omega$ coefficients, and composite reliability ( $\rho_C$ ).
2. Dimensionality: Exploratory Factor Analysis (EFA) using polychoric correlations and the empirical Kaiser criterion; Confirmatory analysis of Essential Unidimensionality using the polyDETECT method (indices: DETECT, ASSI, RATIO).
3. Co-monotonicity: Checking if the ranking of items by difficulty ( $\beta$ ) and discrimination ( $\gamma$ ) is preserved between human and AI versions.
4. Information Functions: Calculation of Item Information Functions (IIF) and Test Information Functions (TIF).
  - Overlap Index ( $\cap$ ): Measures similarity between information curves.
  - Dominance Index ( $D_m$ ): Measures the proportion of the latent trait domain where one test provides more accurate information than the other.

3. Key Contributions

Methodological Framework for AI Validation: The paper proposes a rigorous statistical protocol using Bayesian IRT to validate AI-generated psychometric instruments, moving beyond simple content similarity to structural and informational equivalence.
Quantification of "Epistemic Uncertainty": It introduces metrics (Overlap and Dominance indices) to quantify the divergence in information accuracy between human and AI tests, identifying "hidden" sources of uncertainty in black-box AI generation.
Prompt Sensitivity Analysis: The study demonstrates that the specificity of the prompt significantly alters the psychometric properties of the generated test, affecting dimensionality and item ordering.

4. Results

4.1 Reliability and Dimensionality

Reliability: All three versions showed comparable internal consistency (Cronbach's $\alpha$ and other reliability indices were non-significantly different).
Dimensionality (EFA):
- BAQ & GPT.v1: Suggested a unidimensional structure (consistent with the original construct).
- GPT.v2: Suggested a multidimensional structure (two factors retained), indicating that the generic prompt introduced structural noise.
Essential Unidimensionality (polyDETECT):
- When using Bayesian GRM estimates for latent traits, all versions satisfied essential unidimensionality criteria.
- However, when using naive scores (median of responses), GPT.v1 failed the unidimensionality test, suggesting that simple scoring methods may not capture the latent structure of AI-generated items effectively.

4.2 Parameter Ordering (Co-monotonicity)

Lack of Co-monotonicity: Despite GPT.v1 being a direct rephrasing of BAQ, the ranking of items by difficulty ( $\beta$ $β$ ) and discrimination ( $\gamma$ $γ$ ) was not preserved.
- The most difficult items in the BAQ were not the most difficult in GPT.v1.
- This indicates that while surface wording is similar, the underlying psychometric properties have shifted.

4.3 Information Accuracy

Threshold Shifts: GPT.v1 showed lower threshold values ( $\delta$ ) compared to BAQ, suggesting a shift in how respondents access higher response categories. GPT.v2 showed even higher thresholds, indicating increased ambiguity.
Information Distribution:
- BAQ: Information is concentrated and highly accurate in the central range of the latent trait.
- GPT.v1: Information is dispersed over a wider range of the latent trait. It is slightly more informative at extreme values but less precise in the center.
Overlap vs. Dominance:
- Overlap: High similarity (72.8% – 93.5%) between IIFs, suggesting superficial similarity.
- Dominance: Significant divergence. The BAQ dominates in the central trait range (63.7% of information), while GPT.v1 dominates only in specific extreme ranges.
- Correlation: A strong positive correlation ( $\rho_S = 0.763$ ) was found between the discrimination parameter ( $\gamma$ ) of the BAQ and its dominance over the AI version, confirming that high-discrimination items in human tests are crucial for central trait accuracy.

5. Significance and Conclusion

Significance:
The study reveals that reliability alone is insufficient to validate AI-generated tests. Two tests can have identical reliability coefficients but fundamentally different dimensional structures and information accuracy.

Construct Validity Risk: AI-generated tests may introduce epistemic uncertainty, where the test measures the construct differently (e.g., shifting focus from the center to the extremes of the trait distribution) or introduces hidden multidimensionality.
Prompt Engineering Impact: The "black box" nature of LLMs means that even with specific prompts, the resulting instruments may not align with the theoretical grounding of the original human-designed scale.

Conclusion:
The authors conclude that while LLMs are powerful tools for generating prototypes, they require human supervision and rigorous psychometric validation (specifically using IRT and information functions) before deployment. The study advocates for the use of statistical measures of accuracy (like dominance and overlap indices) to ensure that AI-driven tools do not compromise the validity of social and psychological indicators. Future work should explore differential item functioning (DIF) and larger sample sizes to generalize these findings.