Retrieval-Augmented Claude Opus 4.7 and GPT-5.5 Surpass Human Performance on the Nuclear Cardiology Board Preparation Exam (and Claude Drafts a Paper About it)

Next-generation large language models, specifically Claude Opus 4.7 and GPT-5.5, equipped with retrieval-augmented generation using domain-specific nuclear cardiology resources, achieved mean accuracy rates of approximately 86% on the ASNC Board Preparation Exam, surpassing both the estimated passing threshold and the average performance of human fellows-in-training.

The Gist

Imagine a high-stakes final exam for doctors who specialize in looking at pictures of the heart using special radioactive tracers. This is the "Nuclear Cardiology Board Exam." For years, Artificial Intelligence (AI) has been trying to take this test, but it kept failing, scoring lower than the average medical student.

This paper tells the story of how two new, super-smart AI models finally passed the test with flying colors, beating the average human student.

The Setup: The Exam and the "Cheat Sheet"

The exam has 168 questions. Some are just text (like a trivia quiz), but about 27 of them require looking at complex medical images of hearts.

In the past, when AI tried to take this test "cold" (without any help), the best it could do was get about 63% right. That's a failing grade. The average human medical student (a "fellow-in-training") scored 78%.

For this new study, the researchers gave the AI a massive "cheat sheet." This wasn't just a quick Google search; it was a Retrieval-Augmented Generation (RAG) system. Think of it like giving the AI a perfect, searchable digital library containing the official textbooks, atlases, and medical guidelines for nuclear cardiology. When the AI sees a question, it instantly dives into this library, finds the exact page with the answer, and uses that to formulate its response.

The Contenders

The researchers tested two new, next-generation AI models:

1. Claude Opus 4.7: A model that uses a local, transparent search system (like a librarian who shows you exactly which books it pulled off the shelf).
2. GPT-5.5: A model that uses a cloud-based search system (like a librarian who finds the books for you but doesn't show you the process).

The Results: AI Beats the Average Student

When these two AIs took the exam five times each, the results were surprising:

• The Scores: Both models scored around 86% to 87%.
• The Comparison: This is significantly higher than the average human student's score of 78%. In fact, if you lined up the 13 human students and the 2 AIs, the AIs would rank in the top 5, beating 8 or 9 of the humans.
• The Speed of Progress: This is a massive jump. Just 18 months ago, the best AI scored 63%. Now, with the "cheat sheet" (RAG), they jumped 23 percentage points.

The Two Weak Spots

Even though the AIs won, they had two specific struggles:

1. The "Image" Problem: The AIs were great at text questions (scoring nearly 89%), but they stumbled on the image questions. They got about 73–77% right on images. Humans were still better at this, scoring 81.5%.
   • Analogy: Imagine the AI is a brilliant professor who can recite the entire textbook from memory but still gets confused when looking at a blurry X-ray. It knows the theory perfectly but is still learning how to "see" the picture.
2. The "Safety" Glitch (GPT-5.5 only): GPT-5.5 refused to answer about 7% of the questions. It would say, "I'm sorry, I can't help with that," even though the questions were just standard medical exam questions about heart drugs or radiation safety.
   • Analogy: It's like a very cautious librarian who refuses to hand you a book about "how to build a bomb" even if you are asking a physics student for a legitimate exam question about nuclear energy. The AI's safety filters were too sensitive, causing it to miss points. Claude Opus 4.7 didn't have this problem; it answered everything.

What the Authors Actually Say (and Don't Say)

The paper is very careful about what this means:

• What it IS: It proves that with the right reference materials, AI can learn the facts and rules of nuclear cardiology better than the average trainee. The authors suggest these tools could be used as educational aids to help students study or as reference tools to double-check facts in a reading room.
• What it IS NOT: The authors explicitly state that passing a multiple-choice test does not mean the AI is ready to be a doctor. Real medicine involves talking to patients, handling uncertainty, and making complex judgment calls that a multiple-choice exam cannot measure. The AI is a powerful reference book, not a replacement for a human physician.

The Bottom Line

In the span of a year and a half, AI has gone from failing the nuclear cardiology board exam to beating the average human student, provided it has access to the right textbooks. However, it still struggles with interpreting medical images, and one of the models is too "scared" to answer certain legitimate questions. While it's a huge leap forward for medical education tools, the paper concludes that these machines are assistants, not replacements, for human doctors.

Technical Summary

Technical Summary: Retrieval-Augmented LLMs Surpassing Human Performance on Nuclear Cardiology Board Exams

Problem Statement
Previous evaluations of Large Language Models (LLMs) on the American Society of Nuclear Cardiology (ASNC) Board Preparation Exam revealed a significant performance gap between AI and human trainees. In earlier studies, the best-performing model without domain-specific context (GPT-4o) achieved a median accuracy of 63.1%, falling below the estimated 65% passing threshold and the 78.0% mean score of human fellows-in-training (FITs). While providing textbook context improved GPT-4o's text-only performance to 73.8%, it remained below human averages and was restricted to text-based questions. The central problem addressed by this study is whether next-generation LLMs equipped with Retrieval-Augmented Generation (RAG) can close or exceed this performance gap across the full spectrum of the exam, including complex image-based questions.

Methodology
The study administered the complete 2023 ASNC Board Preparation Exam (168 multiple-choice questions: 141 text-only, 27 image-based) to two next-generation models: Claude Opus 4.7 and GPT-5.5. The evaluation was conducted over 5 iterations for each model.

• Retrieval-Augmented Generation (RAG) Pipelines: Both models were grounded in the same authoritative domain-specific corpus: Nuclear Cardiac Imaging: Principles and Applications (textbook), its companion atlas, and ASNC Clinical Practice Guidelines.
  • Claude Opus 4.7: Utilized a transparent, local FAISS-based retrieval pipeline. PDF documents were chunked into 3-page segments, embedded using the all-MiniLM-L6-v2 sentence-transformer, and indexed via IndexFlatIP (exact inner-product search). For each query, the top 5 semantically similar chunks were retrieved and prepended to the prompt.
  • GPT-5.5: Utilized a cloud-hosted Azure Vector Store accessed via the Azure OpenAI Responses API. The retrieval mechanism (embedding model, similarity metric, chunk count) was managed by Azure's infrastructure and was not directly configurable by the researchers.
• Experimental Setup: Both models received a standardized system prompt instructing them to act as a nuclear cardiology student and select the two most likely answers with explanations. Image-based questions were processed via vision APIs (base6-encoded for Claude, data URIs for GPT-5.5).
• Scoring: The primary answer selected by the model was compared against the correct ASNC answer. Safety refusals (e.g., "I cannot assist with that request") and empty responses were scored as incorrect.
• Comparators: Results were compared against historical data from previous studies (GPT-4o without context, GPT-4o with context) and a cohort of 13 human FITs who took the exam with unlimited time and resources.

Key Results

• Overall Performance: Both models surpassed the human FIT mean (78.0%) and the estimated passing threshold (65%).
  • Claude Opus 4.7: Achieved a mean accuracy of 86.3% (SD 1.4%).
  • GPT-5.5: Achieved a mean accuracy of 86.7% (SD 2.2%).
  • This represents a ~23 percentage point improvement over the best prior LLM result (GPT-4o at 63.1% without context) in approximately 18 months.
• Question Type Breakdown:
  • Text-Only: Both models achieved high accuracy (Claude: 88.8%; GPT-5.5: 88.5%).
  • Image-Based: Performance dropped significantly but remained above the passing threshold (Claude: 73.3%; GPT-5.5: 77.0%). This indicates a persistent gap where human trainees (81.5% on images) still outperform the models in visual interpretation.
• Safety Refusals: A critical operational difference was observed. GPT-5.5 refused to answer a mean of 12.2 questions per iteration (7.3%), scoring these as incorrect. These refusals covered legitimate medical content (radiopharmaceuticals, radiation safety, clinical protocols). Claude Opus 4.7 had zero refusals.
• Model Agreement: The two models showed substantial concordance, answering the same questions correctly in 78.0–83.0% of cases per iteration. Disagreements often involved nuanced distinctions in technical definitions.

Key Contributions

1. Demonstration of Human-Level Proficiency: This is the first study to show that RAG-enhanced LLMs can surpass the average performance of human nuclear cardiology fellows-in-training on a comprehensive board preparation exam.
2. Quantification of RAG Impact: The study isolates the impact of next-generation models combined with RAG, showing a dramatic leap from 63.1% (no context) to ~86.5% (RAG) within a short timeframe.
3. Identification of Safety Limitations: The paper highlights a significant practical barrier in GPT-5.5, where overly conservative safety classifiers refuse to answer legitimate medical examination questions, artificially depressing observed accuracy and limiting utility in educational contexts.
4. Persistence of the Image Gap: The study confirms that while text-based knowledge retrieval has matured, medical image interpretation (SPECT, PET, multimodality) remains a challenging domain where LLMs (73–77%) still lag behind human trainees (~81.5%).

Significance and Claims
The paper claims that RAG-enhanced LLMs have reached a level of nuclear cardiology knowledge sufficient to serve as educational aids for board preparation and knowledge-reference tools in clinical settings. The authors suggest these models can help verify training resources and supplement teaching.

However, the authors maintain a modest stance regarding clinical deployment. They explicitly state that examination performance does not equate to clinical competence. Real-world nuclear cardiology requires integrating patient-specific context, managing uncertainty, and exercising judgment—capabilities not assessed by multiple-choice questions. Consequently, the paper concludes that LLMs should be viewed as supplementary tools to augment physician expertise rather than replacements for it. The study also notes that the persistent gap in image interpretation suggests these models are not yet ready to replace human expertise in diagnostic image reading.