Medical concept understanding in large language models is fragmented

This paper reveals that while large language models excel in medical applications, their understanding of medical concepts is significantly fragmented, with strong performance on identity and hierarchy masking substantial gaps in grasping concept meaning.

The Gist

The Big Question: Do AI Doctors Actually "Get It"?

Imagine you have a brilliant new student, let's call him "AI Alex." Alex has read every medical textbook, watched every surgery video, and memorized millions of patient records. When you ask him, "What's the best treatment for a broken leg?" he answers perfectly. When you ask him to diagnose a rare rash, he gets it right 90% of the time.

You might think, "Wow, Alex is a genius doctor!"

But this paper asks a deeper, more unsettling question: Does Alex actually understand what a "broken leg" is, or is he just really good at guessing the right answer based on patterns?

The researchers wanted to see if Large Language Models (LLMs) like the ones powering modern AI truly understand medical concepts, or if they are just "parrots" that sound smart without knowing the underlying logic.

The Test: The Three-Layer Cake

To find out, the researchers didn't just give Alex a medical exam. Instead, they built a special test based on the Human Phenotype Ontology (HPO). Think of the HPO as the ultimate, perfectly organized library card catalog for human body symptoms.

They broke down "understanding" into three layers, like a three-layer cake:

1. Layer 1: The Name Game (Concept Identity)

   • The Test: Can Alex realize that "Loss of smell" and "Anosmia" are the exact same thing?
   • The Analogy: Imagine someone asks, "Is a 'soda' the same as a 'pop'?" A smart person knows they are synonyms.
   • The Result: Alex is great at this! He got it right about 90% of the time. He knows the different names for the same thing.

2. Layer 2: The Family Tree (Concept Hierarchy)

   • The Test: Can Alex understand that "Anosmia" is a type of "Abnormality of the sense of smell"?
   • The Analogy: If you know "Golden Retriever" is a type of "Dog," and "Dog" is a type of "Animal," you understand the family tree.
   • The Result: Alex is okay, but not perfect. His score dropped to about 84%. He sometimes gets confused about how things fit together in the big picture.

3. Layer 3: The Deep Meaning (Concept Meaning)

   • The Test: Can Alex pick the correct definition of a symptom from a list of 20 very similar-sounding definitions?
   • The Analogy: This is like asking, "What exactly is a 'broken leg'?" and giving you 20 definitions where 19 are slightly wrong (e.g., "a leg that hurts" vs. "a bone that has fractured").
   • The Result: This is where Alex really struggled. His score dropped to 72%. Even worse, if you tricked him with a hint that was slightly wrong, he got confused and his score plummeted.

The Big Discovery: The "Fragmented" Mind

The most shocking finding is that Alex's understanding is fragmented.

Imagine a puzzle. If Alex truly understood a medical concept, he would have the whole puzzle piece: the name, the family tree, and the definition all connected.

But the researchers found that for many concepts, Alex only has parts of the puzzle:

• Sometimes he knows the name but not the definition.
• Sometimes he knows the definition but gets the family tree wrong.
• Sometimes he gets all three right.

The Stats:

• 57.7% of the time, Alex understood everything perfectly (The whole puzzle piece).
• 41.3% of the time, he understood some parts but missed others (A broken puzzle piece).
• 1.1% of the time, he knew nothing about the concept.

The "Thinking" Mode Twist

The researchers also tested what happens when they tell Alex to "think step-by-step" before answering (a feature called "Reasoning Mode").

• Good News: It helped him with some hard questions.
• Bad News: It sometimes made him worse at other questions, and it didn't fix the fundamental gaps in his understanding.

The "Poisoned" Hint Experiment

In the "Meaning" test, the researchers tried a trick. They gave Alex a hint that said, "These two words are not related," even though they actually were.

• Result: Alex believed the lie! His performance dropped significantly.
• What this means: Alex doesn't have a solid, unshakeable internal truth about what a medical term means. He is easily swayed by what he is told in the moment. He is like a student who memorized the answer key but doesn't understand the math; if you tell him the answer key is wrong, he panics.

The Takeaway: Why This Matters

The paper concludes that just because an AI can pass a medical exam doesn't mean it understands medicine.

• The Danger: If an AI is used to help doctors, and it has "fragmented" understanding, it might give a correct answer by luck, but fail catastrophically in a slightly different situation because it doesn't truly grasp the logic.
• The Solution: We can't just rely on AI to "learn" medicine from reading books. We need to build AI systems that are explicitly connected to medical "maps" (ontologies) so they have a solid foundation, not just a good memory.

In short: The AI is a very talented mimic, but it's not yet a true medical thinker. It knows the words, but it's still learning the meaning.

Technical Summary

1. Problem Statement

While Large Language Models (LLMs) demonstrate high performance on downstream medical tasks (e.g., clinical question answering, diagnostic assistance), it remains unclear whether this success stems from a genuine, systematic understanding of medical concepts or merely from pattern matching on surface-level text. Medical intelligence relies on structured knowledge encoded in biomedical ontologies (e.g., synonymy, hierarchical relations, and formal definitions). The authors argue that current evaluations are too task-specific and fail to assess whether LLMs possess coherent, ontology-grounded representations of medical concepts. The core question is: To what extent do LLMs understand the fundamental dimensions of medical concepts (identity, hierarchy, and meaning), and is this understanding consistent or fragmented?

2. Methodology

The authors propose an ontology-grounded, concept-centered evaluation framework using the Human Phenotype Ontology (HPO) as the ground truth.

• Dataset Construction:
  • Started with 11,547 HPO concepts.
  • Filtered for concepts with non-trivial synonyms (lexical dissimilarity) and structural feasibility (available parent/sibling nodes).
  • Final curated set: 6,252 phenotype concepts.
• Evaluation Dimensions: The study decomposes concept understanding into three distinct tasks:
  1. Concept Identity (Synonym Recognition): Can the model recognize that different terms (e.g., "Anosmia" vs. "Loss of smell") refer to the same concept? (4-choice multiple-choice).
  2. Concept Hierarchy (Hypernym Recognition): Can the model identify the correct parent concept (is-a relationship) within the ontology? (4-choice multiple-choice).
  3. Concept Meaning (Definition Selection): Can the model select the correct formal definition for a concept when queried with a non-preferred synonym? (20-choice multiple-choice).
     • Contextual Variations: This task was tested under three conditions: Baseline (no guidance), Augmented (correct ontological guidance provided), and Poisoned (misleading ontological guidance provided).
• Models Evaluated: A representative set of 7 LLMs across three categories:
  • Commercial: Gemini-3 Flash, GPT-5.2.
  • Open-Source: Qwen3-235B, DeepSeek-3.2.
  • Domain-Specialized (Medical Fine-tuned): Baichuan-M2, Citrus-Qwen72B (both fine-tuned from Qwen2.5-72B).
  • Base Model: Qwen2.5-72B (included for comparison).
• Inference Settings: Models were tested in both Standard mode and Reasoning-enabled ("Thinking Mode") to assess the impact of explicit deliberation.

3. Key Results

A. Performance by Dimension

• Concept Identity (Synonyms): Performance is generally high but uneven.
  • Top performers (Gemini-3 Flash, GPT-5.2, Qwen3-235B) achieved ~90% accuracy.
  • Domain-specialized models (Baichuan-M2, Citrus-Qwen72B) lagged significantly (~81%), performing worse than their general-purpose counterparts.
  • Reasoning Mode: Improved performance for some (e.g., DeepSeek-3.2) but slightly degraded others (e.g., Qwen3-235B).
• Concept Hierarchy (Hypernyms): Performance drops significantly compared to identity.
  • Top accuracy: 83.8% (Gemini-3 Flash).
  • All models showed a decline of 6–10 percentage points compared to synonym recognition.
  • Domain-specialized models continued to underperform relative to general-purpose models.
• Concept Meaning (Definitions): Performance is the lowest and most volatile.
  • Baseline accuracy: 72.6% (Gemini-3 Flash).
  • Context Sensitivity: Performance improved drastically with correct guidance (Augmented: ~88%) but collapsed with misleading guidance (Poisoned: ~34–58%). This indicates a lack of stable internal semantic representations.

B. Concept-Level Fragmentation Analysis

The authors analyzed the consistency of understanding for each of the 6,252 concepts across all three dimensions using the best-performing model (Gemini-3 Flash):

• Fully Understood: Only 57.7% of concepts were understood correctly across all three dimensions (Identity, Hierarchy, Meaning).
• Partially Understood: 41.3% of concepts showed fragmented understanding (correct in 1 or 2 dimensions but not all).
• Poorly Understood: 1.1% were failed in all dimensions.
• Visualization: The distribution of understanding is highly heterogeneous and scattered, rather than forming coherent clusters, confirming that LLMs possess fragmented rather than integrated concept representations.

4. Key Contributions

1. Novel Evaluation Framework: Introduced a systematic, ontology-grounded benchmark that moves beyond downstream task performance to probe the internal structure of medical concept understanding.
2. Decomposition of Understanding: Explicitly separated concept understanding into identity, hierarchy, and meaning, revealing that high performance in one area does not guarantee competence in others.
3. Evidence of Fragmentation: Provided empirical evidence that even state-of-the-art LLMs exhibit substantial fragmentation, where strong application-level performance masks fundamental gaps in conceptual coherence.
4. Context Sensitivity: Demonstrated that definition-level understanding is highly fragile and dependent on external context, suggesting LLMs rely on associative patterns rather than robust, internalized medical knowledge.

5. Significance and Implications

• Reliability in Clinical AI: The findings suggest that current LLMs may produce correct answers in specific scenarios (e.g., exams) while lacking the systematic conceptual grounding required for safe, generalizable clinical reasoning. This poses risks for reliability and interpretability in real-world deployment.
• Role of Ontologies: The study argues that the success of LLMs does not render biomedical ontologies obsolete; rather, it highlights the necessity of integrating LLMs with explicit, structured ontological knowledge to stabilize reasoning and prevent fragmentation.
• Future Directions: The authors suggest that future medical AI development must prioritize concept-centered evaluation and hybrid architectures that combine the flexibility of LLMs with the rigor of structured knowledge bases.

In summary, the paper concludes that while LLMs are powerful tools for medical text processing, their understanding of medical concepts is inherently fragmented, and their high performance on benchmarks may be misleading regarding their true capacity for clinical reasoning.