From Generalist to Specialist: A Context-Fusion Framework for Endoscopic Polyp Reporting with a Frozen VLM

In the quiet, winding corridors of the human colon, a doctor performs a delicate search. Using a flexible camera, they look for small, often invisible growths called polyps. Finding these growths is a critical step in preventing cancer, but the work does not end with discovery. To manage a patient's health effectively, the doctor must record precise details: how big the growth is, what specific shape it has, and how it sits on the tissue wall. This record, known as a medical report, becomes the guide for future care, telling other doctors whether to watch the spot closely or remove it entirely. For decades, writing these reports has been a manual task, relying on the doctor's eye and memory to translate a fleeting image into a permanent document. The challenge lies in the sheer complexity of the task; a single image must yield a measurement without a ruler, a classification into a specific medical category, and a descriptive narrative, all at once.

Recently, a new generation of artificial intelligence has emerged, capable of looking at an image and writing a description. These systems, known as vision-language models, are like generalist scholars who have read millions of books and seen countless pictures. They can describe a scene fluently, but when asked to perform the specific, high-stakes duties of a medical specialist, they often stumble. They might write a beautiful sentence about a polyp while getting its size wrong or misidentifying its type. The medical community has tried to fix this by teaching these generalist models specific medical facts, but this approach often requires heavy changes to the model's internal structure, which can make it forget its general knowledge or become too rigid. A team of researchers has now proposed a different path. Instead of rewriting the model's brain, they decided to give it a better set of notes to consult while it works.

The researchers developed a system that acts as a bridge between a generalist artificial intelligence and the specific needs of a colonoscopy report. They took a powerful, pre-trained model—one that was already frozen in its current state, meaning its internal knowledge could not be changed—and equipped it with two types of context. The first type is like a reference library. When the system sees a new image of a polyp, it instantly searches a database of thousands of previously examined cases to find the most visually similar ones. It pulls up these matching images along with the expert reports that were written for them. This provides the system with concrete, real-world examples of how similar growths were measured and described in the past. The second type of context is a subtle, learned instruction. Imagine a set of invisible, continuous signals that tell the model, "Remember, you are a specialist now; pay attention to the size, the shape category, and the texture." These signals are not words, but mathematical patterns that guide the model's focus without altering its core structure.

By combining these two sources of information, the system creates a rich environment for the artificial intelligence to work in. It sees the new patient's image, it has the invisible specialist instructions, and it has a stack of similar past cases to compare against. The researchers tested this approach on a dataset of over two thousand expert-annotated endoscopic images. They compared their method against the generalist models on their own, against systems that had been heavily retrained on medical data, and against systems that tried to learn specific tasks separately. The results were clear. The new framework, which required training only a tiny fraction of the total model's parameters—less than one hundredth of a percent—outperformed all other methods. It produced reports that were not only fluent but also accurate in their measurements and classifications.

The study showed that this approach solved a specific problem that had plagued previous attempts. When the researchers looked at cases where other systems failed, they found that their method could often correct the mistake. For instance, if a standard system misidentified the type of polyp, the new framework, by looking at the most similar past cases, was able to find the correct classification in seventy percent of those difficult instances. It did this without losing the ability to write a coherent, natural-sounding report. The system managed to balance three difficult tasks simultaneously: estimating the diameter of the growth, categorizing its shape, and writing a description of its surface. In many previous attempts, improving one of these areas would hurt the others, but this method improved all of them at once.

The researchers also explored how the system used the information it was given. They found that the system did not simply copy the past cases it found; it learned to weigh them. When the retrieved examples were highly relevant to the current image, the system's accuracy jumped significantly. However, if the examples were chosen at random, the system's performance dropped, proving that the quality of the reference material mattered more than just having more material. The system also showed that asking it to perform all three tasks together actually helped it write better descriptions. By first determining the size and type, the system seemed to gain a clearer understanding of the growth, which allowed it to describe its texture and appearance more accurately.

This work suggests a new way to adapt powerful artificial intelligence tools for specialized medical work. Instead of trying to retrain the entire brain of the machine, which is expensive and risky, the researchers showed that providing the right context at the moment of decision is enough. The system remains a generalist at its core, but it acts as a specialist when it needs to, guided by the evidence of similar cases and a subtle set of instructions. The findings indicate that this method is a lightweight and effective strategy for bringing artificial intelligence into the clinical workflow. It offers a way to generate reliable, structured medical reports that can be checked for accuracy, potentially reducing the burden on doctors and improving the consistency of patient care. The study concludes that by fusing specific evidence with learned guidance, a frozen model can be transformed into a capable specialist without ever changing its fundamental nature.