💻 computer science

Classification Fidelity of Large Language Models on Endodontic Items from the Turkish Dental Specialty Examination

This study evaluates the classification fidelity of four large language models in categorizing endodontic examination questions across three educational taxonomies, revealing that while models like Gemini and DeepSeek achieve substantial agreement for Bloom's and SOLO taxonomies, all models struggle with Miller's Pyramid, demonstrating that LLM performance is taxonomy-dependent and distinct from answer accuracy.

Original authors: Bilge Hakan Şen, Aslıhan Yekeler, Duygu Kürklü Arpaçay, Jülide Ocak, Oğuzhan Akkoçan

Published 2026-07-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bilge Hakan Şen, Aslıhan Yekeler, Duygu Kürklü Arpaçay, Jülide Ocak, Oğuzhan Akkoçan

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant library of dental exam questions (specifically about root canals) from Turkey, written between 2012 and 2025. You want to know if four famous AI chatbots (ChatGPT, Gemini, Kimi, and DeepSeek) can do more than just answer these questions correctly. You want to know if they can understand the "difficulty level" of the question itself.

Think of it like a music teacher. A student might play a song perfectly (getting the answer right), but can they tell you if the song is a simple nursery rhyme, a complex jazz piece, or a full symphony? That's what this study tested.

Here is the breakdown of the study using simple analogies:

1. The Three "Difficulty Rulers"

The researchers used three different "rulers" to measure how hard a question is. The AI had to sort every question into the right category on each ruler:

Bloom's Taxonomy (The "Thinking Ladder"): This measures how much brainpower is needed.
- Bottom rungs: Just remembering facts (like "What is a tooth?").
- Top rungs: Complex thinking like analyzing a problem or creating a solution.
SOLO Taxonomy (The "Building Block" Tower): This measures how many pieces of information you need to connect to answer.
- Small tower: You only need one fact.
- Tall tower: You need to connect many facts and see how they relate.
Miller's Pyramid (The "Doctor's Ladder"): This measures if the knowledge is just in your head or if you can actually do it.
- Bottom: "I know the theory."
- Top: "I can do it in real life on a patient."

2. The Human "Gold Standard"

Before asking the AI, the researchers hired human experts (specialist dentists and education professors) to sort the questions first. They did this very carefully, training the experts, auditing their work, and having them debate until they agreed. This created the "Answer Key" for what the questions actually were.

3. The AI Showdown

The researchers fed the same 200 questions to the four AI models and asked them to sort the questions using the three rulers above. They then compared the AI's sorting to the Human "Answer Key."

The Results: Who Got It Right?

The Good News (Bloom & SOLO):

Gemini and DeepSeek were the star students. They were very good at sorting questions by how "thinking-heavy" they were (Bloom) and how many "building blocks" were needed (SOLO). They agreed with the human experts most of the time.
ChatGPT and Kimi were okay, but they made more mistakes. They often thought simple questions were too hard, or complex questions were too easy.

The Bad News (Miller's Pyramid):

Everyone struggled here. No AI did a great job sorting questions by "Can this be done in real life?" (Miller).
The Twist: The rankings flipped completely!
- Gemini and DeepSeek were the best at the "Thinking" rulers but the worst at the "Real Life" ruler.
- ChatGPT was the worst at the "Thinking" rulers but the best (though still only "fair") at the "Real Life" ruler.

Key Takeaways from the Paper

Answering vs. Understanding are different skills: Just because an AI can get the right answer to a dental question doesn't mean it understands why the question is hard or what kind of thinking it requires. It's like a calculator that can solve a math problem but doesn't understand the concept of algebra.
One size does not fit all: You can't use one AI for every job. If you want to check if questions are getting harder (Bloom/SOLO), use Gemini or DeepSeek. If you are trying to guess if a question relates to real-life practice (Miller), ChatGPT might actually be your best bet, even though it's worse at the other tasks.
The questions are getting harder: The study found that the dental exam questions from 2025 are significantly more complex and require more "thinking" than the questions from 2012.
AI gets confused by the top of the ladder: All the AIs found it much harder to correctly identify the very hardest questions (the ones requiring analysis or evaluation) compared to the easy ones (just remembering facts).

What the Paper Does Not Say

The paper does not say that these AIs should be used to grade students, replace teachers, or diagnose patients. It strictly says that while AIs are getting better at sorting questions by difficulty, they are not perfect, and their performance changes depending on which difficulty ruler you use. It also notes that because the exam questions are getting harder over time, older studies on AI might have overestimated how smart the AI really is.

Technical Summary: Classification Fidelity of Large Language Models on Endodontic Items from the Turkish Dental Specialty Examination

Problem Statement
While Large Language Models (LLMs) are increasingly utilized in dental education and assessment, existing research has focused almost exclusively on answer accuracy (i.e., whether the model selects the correct option). A critical gap remains regarding whether LLMs can reliably identify the cognitive demand of examination items. This capability is distinct from answering correctly and is essential for assessment quality assurance and curriculum mapping. Furthermore, it is unknown if LLMs can consistently classify cognitive complexity across multiple, distinct educational taxonomies (Revised Bloom's, SOLO, and Miller's Pyramid) or if their performance varies significantly between models and frameworks.

Methodology
The study employed a cross-sectional observational design to evaluate the classification fidelity of four publicly accessible LLMs: ChatGPT (GPT-5.2 Instant), Gemini (3.1 Pro), Kimi (K2.5), and DeepSeek (V3.2).

Dataset: 200 endodontic multiple-choice questions (MCQs) from the Turkish Dentistry Specialty Examination (DUS) administered between 2012 and 2025. The items were presented in their original Turkish language.
Gold Standard: A three-layer expert consensus procedure established the ground truth. Two experts (an endodontist and a prosthodontist with educational expertise) independently classified items, followed by a calibration training session, a procedural audit by an external advisor, and structured joint deliberation to resolve disagreements.
Taxonomies: Each item was classified according to:
1. Revised Bloom's Taxonomy: Six levels (Remember to Create).
2. SOLO Taxonomy: Five levels (Pre-structural to Extended Abstract).
3. Miller's Pyramid: Four levels (Knows to Does).
Experimental Controls: LLMs received a standardized prompt in dedicated incognito browser sessions to eliminate cross-session memory and personalization effects. Generation parameters (e.g., temperature) were platform-managed and not modifiable.
Statistical Analysis: Agreement was measured using weighted Cohen's kappa ( $\kappa_w$ ) with bootstrap 95% confidence intervals. Classification performance was evaluated via accuracy and macro-averaged F1 scores. Inter-model differences were tested using Cochran's Q and pairwise McNemar tests with Holm–Bonferroni correction.

Key Results

Framework-Dependent Performance:
- Bloom's Taxonomy: Gemini ( $\kappa_w = 0.676$ ) and DeepSeek ( $\kappa_w = 0.667$ ) achieved substantial agreement with the gold standard. ChatGPT ( $\kappa_w = 0.365$ ) and Kimi ( $\kappa_w = 0.428$ ) achieved only fair to moderate agreement.
- SOLO Taxonomy: Gemini again led with substantial agreement ( $\kappa_w = 0.622$ ). The other models achieved moderate agreement ( $\kappa_w$ range: 0.433–0.549).
- Miller's Pyramid: Performance was significantly weaker across all models. Agreement did not exceed fair for any model. ChatGPT performed best ( $\kappa_w = 0.372$ ), while Gemini, Kimi, and DeepSeek achieved fair to slight agreement.
Complete Ranking Reversal:
A striking finding was the total reversal of model rankings between frameworks. Models that excelled at Bloom and SOLO (Gemini, DeepSeek) performed poorly on Miller's Pyramid. Conversely, ChatGPT, which ranked lowest on Bloom and SOLO, achieved the highest accuracy and agreement on Miller's Pyramid.
Cognitive Level Trends:
- Classification fidelity declined as cognitive levels increased. All models classified lower-order thinking (Bloom Levels 1–2) with higher fidelity than higher-order thinking (Levels 3–5).
- ChatGPT exhibited a systematic bias, over-classifying Level 2 (Understand) items as Level 4 (Analyze).
- DeepSeek correctly identified all five Level 5 (Evaluate) items but over-assigned Level 5 to other items.
- Gemini demonstrated the most balanced misclassification profile and was the only model to show consistent per-level F1 scores across all three SOLO levels.
Temporal Trends:
A significant positive correlation was found between examination year and gold standard Bloom level ( $\rho = 0.301, p < 0.001$ ), indicating that DUS endodontic questions have become progressively more cognitively demanding from 2012 to 2025.

Key Contributions

Decoupling Accuracy from Classification: The study demonstrates that answer accuracy and taxonomy classification fidelity are separate capabilities. A model can answer questions correctly (as noted in prior studies for ChatGPT) while failing to accurately identify the cognitive complexity of those questions.
Non-Interchangeability of Taxonomies: The results provide empirical evidence that Bloom, SOLO, and Miller's Pyramid assess distinct constructs. An LLM's performance in one framework does not predict its performance in another; therefore, they cannot be used interchangeably as proxies for cognitive demand.
Model Selection Guidance: The study offers specific guidance for selecting LLMs based on the assessment purpose: Gemini and DeepSeek are superior for cognitive complexity mapping (Bloom/SOLO), while ChatGPT showed relative strength in clinical applicability stratification (Miller), though overall Miller performance remained weak.

Significance and Claims
The paper claims that LLMs can classify cognitive complexity with meaningful fidelity for Bloom's and SOLO taxonomies, particularly when using Gemini or DeepSeek, but that this capability is not uniform across all educational frameworks. The authors emphasize that Miller's Pyramid remains a challenging domain for current LLMs, with no model achieving agreement beyond the "fair" level.

The study concludes that for assessment quality assurance and curriculum mapping, LLM selection must be tailored to the specific taxonomy and purpose. Furthermore, the observed increase in cognitive demand over time suggests that studies relying on older item banks may overestimate current LLM classification performance. Finally, the authors assert that taxonomy classification fidelity must be evaluated independently from answer accuracy to ensure the validity of AI-assisted educational tools.