End-to-End Agentic AI Framework for Cross-Dataset Medical Image Classification Using CNNs, Vision Transformers, Explainable AI, Robustness Analysis, and LLM-Driven Clinical Reporting
This paper presents an end-to-end agentic AI framework that systematically compares CNN and Vision Transformer architectures across three diverse medical imaging tasks, integrating robustness analysis, explainability, and LLM-driven reporting to deliver reliable, interpretable, and clinically supportive diagnostic insights.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the quiet hum of a hospital radiology department, a new kind of assistant is learning to look at the human body. For decades, doctors have relied on their own trained eyes to spot signs of disease in X-rays, magnetic resonance images, and skin scans. Now, artificial intelligence offers a second pair of eyes, one that can scan thousands of images in seconds. But for this technology to be truly useful in a life-or-death setting, it cannot simply be fast or accurate on a test. It must be reliable. It must know when it is unsure. It must be able to explain why it thinks a shadow on a lung is pneumonia rather than a harmless artifact. It must also remain steady even when an image is slightly blurry or rotated, mimicking the imperfect conditions of the real world. This is the challenge that researchers at Stevens Institute of Technology set out to solve, moving beyond the simple question of "is the computer right?" to ask "how right is it, and why?"
The team built a comprehensive testing ground to compare two different families of artificial intelligence models. The first family, known as convolutional neural networks, works much like a human scanning an image in small, focused steps, building up a picture of the whole from local details. The second family, called vision transformers, takes a different approach by looking at the entire image at once, paying attention to how distant parts of the picture relate to one another. To see which method works best, the researchers fed these models three very different types of medical images: chest X-rays to detect pneumonia, brain MRI scans to find tumors, and dermoscopic photos of skin lesions to identify cancer. They did not just measure how often the models guessed correctly. Instead, they subjected the models to a rigorous battery of tests that checked for confidence, stability, and the ability to point out exactly which part of the image led to a decision.
The results revealed that there is no single champion that wins in every situation. When the task was to read chest X-rays, a model called DenseNet121, which belongs to the first family, proved to be the most accurate, correctly identifying pneumonia or normal lungs in nearly 99 percent of cases. This suggests that for the structured, high-contrast patterns found in standard X-rays, the method of scanning local details is highly effective. However, the story changed when the researchers moved to brain MRI scans. These images are more complex, with tumors that have irregular shapes and boundaries that blend into healthy tissue. Here, the vision transformer models, which look at the global context of the image, outperformed the others. One transformer model, Swin-Tiny, achieved the highest ability to distinguish between tumor and no-tumor cases, a critical measure for avoiding missed diagnoses. Similarly, for skin cancer detection, a transformer model named DeiT-Small showed superior ability to separate malignant from benign cases, even though another model had a slightly higher raw accuracy score.
Perhaps the most revealing part of the study was not about which model was the most accurate, but which was the most resilient. The researchers tested the models by intentionally introducing small distortions, such as blurring the image or changing its brightness, to see how well they held up. In these tests, the vision transformer models consistently proved more robust. While the most accurate model for chest X-rays saw its performance drop significantly when the images were altered, the transformer models maintained their accuracy much better. This indicates that while a model might be perfect on a clean, ideal image, it may fail when faced with the messy reality of a hospital scanner. The study also examined how well the models calibrated their confidence. A model that is 90 percent sure of its answer should be right 90 percent of the time. The researchers found that some highly accurate models were dangerously overconfident, while others, like a model called EfficientNet-B0, were much better at matching their confidence to their actual correctness.
To make sense of these complex findings, the researchers added a final layer to their framework: a large language model that acted as a research assistant. This system took all the data—the accuracy scores, the robustness tests, and the visual explanations generated by the models—and synthesized them into clear, structured reports. These reports did not offer medical diagnoses but instead provided a summary of the evidence, highlighting where each model succeeded and where it might fail. This approach ensures that the technology remains a tool for human decision-making rather than an autonomous judge. The study concludes that the future of medical AI does not lie in finding one perfect algorithm, but in understanding the specific strengths and weaknesses of different approaches. For some tasks, the detailed, local focus of one type of model is best; for others, the broad, contextual view of another is superior. By testing these systems across accuracy, reliability, and explainability, the researchers have provided a roadmap for building AI that is not just smart, but safe and trustworthy enough to stand beside a doctor.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.