Automated item evaluation: Predicting item acceptance and rejection using LLM-generated critiques

Imagine you are a teacher who has to grade thousands of homework assignments every week. Some are brilliant, some are confusing, and some are just plain wrong. Usually, you have to read every single one to decide which ones to keep and which to throw away. This is slow, expensive, and tiring. Now, imagine a super-smart robot assistant that can read those assignments and tell you, "Hey, this one looks risky, maybe don't use it," before you even spend your time grading it. This is the world of Automated Item Evaluation (AIE). It's a branch of educational technology that tries to use computers to judge the quality of test questions (called "items") without needing a human expert to read every single one or test it on real students first. The big question researchers are asking is: Can we teach a computer to spot a "bad" test question just by reading the text, saving us from the headache of finding out later that the question was broken?

This paper is about building that super-smart robot assistant. The researchers, Hotaka Maeda and Yikai Lu, wanted to create a system that could look at a test question and predict whether it would be "accepted" (good to go) or "rejected" (trash it) based on a massive database of 52,759 real questions from a large standardized testing program. They didn't just feed the raw questions to a computer; they tried a clever trick. They asked a large language model (a type of AI that can write and think) to first write a short "critique" or review of the question, and then they fed both the original question and that AI review into their prediction model. Think of it like hiring a student to write a quick summary of a book before you decide if you want to read the whole thing yourself.

The results were pretty exciting, but with some important caveats. The "fusion model" (the one that looked at both the question and the AI critique) was the best performer. It got the right answer about 75% of the time overall. However, it was a bit of a superhero for math questions (getting it right 73% of the time) but struggled a bit more with English Language Arts questions (only 51% accuracy). The researchers found that if they made the robot a little more paranoid—telling it to flag anything that might be bad rather than waiting for it to be definitely bad—it could catch 90% of the bad questions, though it would also accidentally flag some good ones.

Here is the most important part: the robot was great at spotting questions that were too hard, had confusing wording, or didn't match the curriculum. But, it was terrible at spotting questions that were biased, unfair, or sensitive to certain groups of people. The paper suggests that while computers are getting really good at checking the "mechanics" of a test question, they still need human teachers to check the "heart" and fairness of the questions. So, the future isn't about replacing humans, but giving them a powerful tool to filter out the obvious junk so they can focus on the tricky, important stuff.

The Story of the "Bad Question" Detector

The Problem: The Mountain of Homework
Creating a standardized test is like building a giant Lego castle. You need thousands of individual bricks (questions) to make it stand up. But not every brick is good. Some are cracked, some are the wrong color, and some just don't fit. Traditionally, experts have to pick up every single brick, inspect it, and sometimes even test it on a group of kids to see if it works. This takes forever and costs a fortune. With the rise of AI that can generate questions automatically, the mountain of bricks is getting bigger every day. We need a way to quickly sort the good bricks from the bad ones without checking every single one by hand.

The Experiment: The Two-Brain Approach
The researchers decided to train a computer brain (specifically a model called DeBERTaV3-large) to do this sorting. They had a huge dataset of 52,759 questions. About 34% of these were "rejected" (permanently removed because they were bad), and the rest were "accepted."

They tried three different ways to teach the computer:

1. The Raw Reader: They fed the computer just the text of the question.
2. The Critic: They asked a different AI (Qwen3) to write a two-sentence review of the question first, then fed that review to the computer.
3. The Fusion Model: They combined both! They fed the computer the original question and the AI's review.

The Results: A Smart, but Flawed, Assistant
The "Fusion Model" was the clear winner. It achieved an accuracy of 0.75 (meaning it was right 75% of the time) and an F1 score of 0.64.

• Math vs. English: The model was a math whiz, with an F1 score of 0.73 for math questions. But for English Language Arts (ELA), it stumbled with an F1 score of only 0.51. The authors suggest this might be because English questions often depend on long reading passages, which the researchers had to leave out of the data because they were too long.
• The "Paranoid" Setting: The researchers noticed that if they lowered the "alarm threshold" from 0.5 to 0.25, the model became much better at catching bad questions. It caught 90% of the rejected items (sensitivity of 0.90 for math and 0.88 for ELA). The trade-off? It started flagging too many good questions as bad (specificity dropped). The authors suggest this "paranoid" setting might be perfect for automated item generation, where it's cheap to make a question but expensive to review it. If you catch 90% of the bad ones early, you save a lot of time.

What the Model Missed: The Human Touch
Despite its success, the model had a blind spot. It was very good at spotting questions that were too difficult or had "psychometric" issues (statistical problems). However, it struggled significantly to detect questions related to bias, sensitivity, fairness, or accessibility. For English questions, the model only caught 27% of these issues. The authors explain that these are deeply human problems. A computer can see if a word is hard, but it might not realize that a specific story is offensive to a certain culture or group. This suggests that while AI can do the heavy lifting of filtering, human experts are still absolutely necessary to check for fairness.

The Takeaway
This paper suggests that we can build a near-comprehensive tool to automatically evaluate test questions. By combining the raw text of a question with an AI-generated critique, we can get a much better prediction of whether a question will pass or fail. It's not a magic wand that solves everything—especially when it comes to fairness and bias—but it is a powerful new tool that could save educators and testing companies a massive amount of time and money. The future of testing might involve a robot doing the first pass of sorting, leaving the human experts to focus on the questions that really need a human heart to judge.