INFORM-CT: INtegrating LLMs and VLMs FOR Incidental Findings Management in Abdominal CT
This paper introduces INFORM-CT, a novel plan-and-execute agentic framework that integrates large language models (LLMs) and vision-language models (VLMs) to automate the detection, classification, and reporting of incidental findings in abdominal CT scans, demonstrating superior accuracy and efficiency compared to existing pure VLM-based approaches.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are a doctor looking at a CT scan of a patient's belly. It's like looking at a very detailed, 3D map of their insides. Often, while looking for the main problem (like a broken bone or a specific pain), doctors find other small, unexpected things—maybe a tiny cyst on the liver or a small spot on the kidney. These are called "incidental findings."
Most of the time, these are harmless. But sometimes, they are serious. The problem is that there are so many of them, and the rules for deciding which ones need a follow-up are incredibly complex, written in thick manuals (guidelines) that change often. Checking every single one by hand is slow, tiring, and sometimes leads to different doctors making different decisions.
This paper introduces a new robot assistant called INFORM-CT to help with this job. Here is how it works, explained with some everyday analogies:
1. The Brain and The Hands (The "Plan-and-Execute" Team)
Think of INFORM-CT as a team of two people working together:
- The Brain (The Planner): This is a super-smart AI (a Large Language Model) that reads the complex medical rulebooks. It doesn't look at the X-rays itself; instead, it acts like a project manager. It reads the rules and says, "Okay, to check the liver, we need to do Step A, then Step B, then measure this specific thing."
- The Hands (The Executor): This is the part that actually does the work. It takes the Brain's instructions and writes a computer program (Python code) to do the job. It uses "specialized tools" (like a ruler, a magnifying glass, or a color detector) to measure the spots on the scan.
The Analogy: Imagine you are baking a cake based on a very strict, complicated recipe.
- The Brain is the head chef who reads the recipe and tells you exactly what to do: "First, sift the flour. Then, check if the eggs are fresh. If the eggs are old, stop. If they are fresh, mix for 5 minutes."
- The Hands are the sous-chefs who actually grab the sifter, crack the eggs, and turn on the mixer. They don't decide what to do; they just follow the precise instructions to get it done perfectly.
2. The Toolkit (Base Functions)
The "Hands" don't just guess. They have a toolbox of pre-built skills:
- Segmentation: Like a digital highlighter that can trace the outline of a liver or a kidney perfectly.
- Measurement: A digital ruler that measures the size of a spot in millimeters.
- Intensity Check: A tool that checks how "bright" or "dark" a spot is (like checking if a bruise is fresh or old).
- The Labeler: A smart assistant that looks at a spot and says, "This looks like a benign cyst" or "This looks suspicious," based on what it has learned from thousands of other scans.
3. The "Try, Fail, Fix" Loop
One of the coolest parts of this system is that it doesn't just run once and hope for the best.
- The Brain writes a plan.
- The Hands try to run it.
- If the plan has a mistake (like trying to measure a kidney that wasn't found), the system catches the error.
- The Brain gets the error message, thinks, "Oh, I forgot to check for that," and rewrites the plan.
- It keeps doing this until the plan works perfectly.
This is like a programmer debugging code, but the AI is doing it automatically to ensure the medical rules are followed exactly.
4. Why This is Better Than Just "Looking"
Previous AI tools tried to look at the scan and guess the answer all at once (like a student taking a test without showing their work). They often got confused or missed details.
INFORM-CT is different because it shows its work. It breaks the problem down into small, logical steps, just like a human doctor would.
- Old Way: "I think this spot is bad." (Guess)
- INFORM-CT Way: "I found a spot. It is 15mm wide. It is on the liver. The rules say anything over 10mm needs a follow-up. Therefore, I recommend a follow-up." (Logic)
The Result
The researchers tested this on thousands of real CT scans of livers, kidneys, and pancreases. They found that:
- It's more accurate: It followed the rules much better than the previous AI models.
- It's explainable: You can see exactly why it made a decision because it followed the step-by-step plan.
- It's flexible: If the medical rules change (like a new guideline from a medical association), you just update the rulebook, and the Brain automatically writes a new plan. No need to retrain the whole system.
In short: INFORM-CT is a smart, rule-following robot assistant that helps doctors manage the overwhelming number of "oops, I found something else" moments in CT scans, ensuring nothing dangerous is missed and nothing harmless causes unnecessary panic.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.