Performance of Generative Artificial Intelligence and Medical Students on Undergraduate Ophthalmology Examinations: A Cross-Sectional Comparative Study
This cross-sectional study found that while five generative AI systems significantly outperformed fourth-year medical students on undergraduate ophthalmology text-based questions, most struggled with image-based items, highlighting their potential as educational tools while cautioning against equating exam performance with clinical competence.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the classrooms of modern medicine, a new kind of student has quietly entered the lecture hall. It does not sleep, it does not forget, and it can read thousands of pages in a second. This is generative artificial intelligence, a type of computer program designed to understand language and generate answers that sound like they came from a human. These systems, often called large language models, have become common tools for learning, helping students explain complex ideas or prepare for tests. But while these machines are getting better at reading and reasoning, a critical question remains: do they truly understand the visual world of medicine? In fields like ophthalmology, the study of eyes, diagnosis often depends on spotting subtle patterns in photographs of the retina or the inside of an eye. For a human doctor, seeing is believing. For a machine, seeing is a much harder challenge. As these tools become more integrated into education, educators need to know if they are merely mimicking knowledge or if they can actually see what a student sees.
To find the answer, a team of researchers at the University of Porto in Portugal put five of the most advanced artificial intelligence systems to the test against fourth-year medical students. The researchers gathered 89 multiple-choice questions from real ophthalmology exams given to students in 2025. These questions covered a wide range of eye conditions, from glaucoma to retinal diseases. The set included 77 questions that could be answered by reading text alone, and 12 questions that required the test-taker to look at a photograph or a medical image to find the right answer. The researchers then fed every single question, along with its accompanying image where applicable, into five different AI systems: GPT-4.1 Auto, GPT-5.2 Auto, Gemini 3 Flash, Grok 4.1 Auto, and Claude Sonnet 4.5. The computers were given no extra help, no internet search, and no chance to ask for clarification; they simply had to pick the best answer from the five options provided, just as a student would.
The results revealed a striking divide between how well the machines handled words versus images. On the text-only questions, the artificial intelligence systems performed with remarkable skill. Three of the five systems got more than 90 percent of the answers correct, with the top performer, Gemini 3 Flash, reaching 95.5 percent accuracy. In comparison, the average medical student got about 73 percent of the questions right. This means that on paper-based questions, the best AI systems were outperforming the vast majority of the human students, ranking in the top percentile of the class. However, the story changed completely when the questions included images. When the test required looking at a picture of an eye, the performance of four out of the five AI systems dropped significantly. While the human students maintained their level of accuracy regardless of whether the question was text or image-based, the machines struggled to interpret the visual data. For instance, one system that scored 98.7 percent on text questions fell to 75 percent on image questions. Another dropped from 96 percent to 67 percent. Only one system, GPT-4.1 Auto, showed no significant difference between the two formats, but it also had the lowest overall score of all the machines.
The researchers also looked at how the machines' answers compared to the students' answers to see if they were making the same mistakes. They found that the AI systems were not simply copying the students. In fact, the systems that got the most questions right were often the ones that disagreed most with the student majority. These high-performing machines correctly answered many questions that most students got wrong, suggesting they were accessing a different kind of knowledge or reasoning process. Conversely, the system with the lowest overall accuracy, GPT-4.1 Auto, was the one that agreed most closely with the students' pattern of answers. This indicates that while the machines are powerful, they do not necessarily think like humans, and their errors are not the same as human errors. The study also showed that the machines were better at answering questions that were easier for the students to solve, suggesting that the difficulty of the question affects both humans and machines in a similar way.
Despite the high scores, the researchers caution that these results do not mean the machines are ready to replace doctors or that they possess true clinical competence. The study was conducted on a specific set of questions from a single university, and the number of image-based questions was relatively small. The authors emphasize that while these tools can be excellent for reviewing facts and preparing for written exams, they cannot yet be trusted to interpret the complex visual findings that are central to eye care. The ability to look at a patient's eye and make a diagnosis is a skill that requires more than just selecting the right letter on a test. For now, these artificial intelligence systems serve as powerful study aids that can help students learn, but they remain distinct from the human ability to see and understand the world of medicine.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.