AI-Benchmarked Analysis of Department-Specific ICU Exit Exam Performance: A Cross-Sectional Study
This cross-sectional study demonstrates that while the AI model DeepSeek-V4-Pro outperformed resident physicians overall on ICU exit exams, significant performance gaps varied by home department and question type, highlighting the utility of a dual-reference AI benchmarking framework for identifying targeted learning needs despite the need for further validation.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the modern hospital, the intensive care unit is a place where patients are at their most vulnerable, and the doctors who care for them must be exceptionally skilled. To ensure that trainee doctors are ready for this high-stakes environment, medical schools in many countries require residents to spend a month rotating through an ICU before they can graduate. At the end of this rotation, they take a written exam to prove they have learned what they need to know. For years, educators have wondered if the type of doctor a resident is training to be—whether a surgeon, a heart specialist, or a general practitioner—influences how well they perform on these tests. At the same time, a new kind of tool has entered the conversation: artificial intelligence. These computer systems, known as large language models, have shown they can read medical textbooks and answer complex questions with surprising accuracy. The big question for educators is no longer just whether these machines can pass a test, but whether they can serve as a reliable yardstick to measure human learning, helping to spot exactly where different groups of doctors might need more help.
A team of researchers at Wenzhou Medical University in China decided to put this idea to the test. They gathered the exam results of 412 resident doctors who had just finished their one-month ICU rotations. These doctors came from six different backgrounds, including internal medicine, surgery, and anesthesiology. The researchers wanted to see if the doctors' home departments made a difference in their scores, and they wanted to see how a specific artificial intelligence model, DeepSeek-V4-Pro, would perform on the exact same set of questions. The exam itself was substantial, consisting of 300 questions divided into different styles. Some questions asked for basic facts, others presented a short patient story and asked for a single best answer, and a final group required the doctor to analyze a complex case and select all the correct options from a list.
The researchers fed every single one of those 300 questions into the artificial intelligence model. The computer was instructed to act like a test-taker, providing only the letter of the answer without any extra explanation. When the results came in, the computer model got 85.7 percent of the questions right. The human doctors, on average, scored 80.8 percent. This means the artificial intelligence outperformed the group of doctors as a whole. However, the story was not the same for every type of question. The computer was significantly better at the basic fact questions and the single-answer story questions. But when it came to the most complex part of the exam—the case-analysis questions where doctors had to pick multiple correct answers from a list—the human doctors actually did slightly better than the machine. The doctors scored about 2.7 percentage points higher on these difficult items, a small but real advantage that suggests humans still hold a slight edge in certain types of critical thinking tasks.
The researchers then looked closer at the human scores to see if the doctors' home departments mattered. Even after accounting for the fact that different years of training took different versions of the test, clear differences remained. Doctors from the surgery department scored lower on average than those from internal medicine and other clinical specialties. The anesthesiology group also scored lower than the internal medicine and other clinical groups. This suggests that the background a doctor comes from does influence how they perform on ICU exit exams, even after they have all completed the same month of training. It does not prove that the training itself failed for these groups, but it does highlight that their starting points or their specific learning needs might differ.
To make these findings useful for teaching, the researchers created a new way to look for trouble spots. They compared the performance of each department on each type of question against two things: the average performance of all the other departments, and the score achieved by the artificial intelligence. They were looking for situations where a specific group of doctors scored lower than both their peers and the computer. They found thirteen such situations. In eight of these cases, the gap was large enough to be statistically significant. For example, the surgery group had a particularly large gap compared to the computer on the basic fact questions, scoring nearly 18 percentage points lower. Similarly, the anesthesiology group struggled with basic facts compared to the computer benchmark. These results act as a signal, pointing educators toward specific areas where certain groups of doctors might need extra review or different teaching strategies.
The study is careful to note that these findings are a starting point, not a final verdict. The research was done at a single hospital, and the design of the study means it cannot prove that the ICU rotation caused the differences in scores, only that they exist at the end of the rotation. Furthermore, the artificial intelligence was tested only once on each question, and computer models can sometimes give different answers if asked the same question again. Despite these limits, the work offers a fresh perspective. It shows that artificial intelligence can serve as a consistent benchmark for medical knowledge, performing better than the average resident on standard questions. More importantly, it demonstrates a method for using that benchmark to identify specific learning gaps in different groups of doctors. Before these signals can be used to change medical school curriculums, they will need to be confirmed in larger studies, but the approach offers a promising new tool for understanding how doctors learn.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.