Towards Selection of Large Multimodal Models as Engines for Burned-in Protected Health Information Detection in Medical Images
This paper benchmarks three Large Multimodal Models (GPT-4o, Gemini 2.5 Flash, and Qwen 2.5 7B) for detecting Protected Health Information in medical images, finding that while they outperform traditional OCR in text extraction, their overall detection accuracy gains are most significant for complex imprint patterns rather than clearly readable text, leading to tailored selection recommendations and a scalable deployment strategy.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you have a stack of old medical X-rays or scans. Hidden inside these images are little "stamps" or "watermarks" (like dates, patient names, or ID numbers) burned right onto the picture. If you want to share these images for research or teaching, you have to erase those stamps first to protect the patient's privacy. This is called finding and removing PHI (Protected Health Information).
For a long time, doctors and tech teams used a two-step robot process to do this:
- The Scanner: A specialized tool (called OCR) that looks at the image and tries to read the text like a very fast typist.
- The Editor: A smart computer program that reads what the typist wrote and decides, "Is this a secret patient name? Yes, delete it. Is this just a date for the hospital? No, keep it."
The New Idea: The "Super-Reader"
The authors of this paper asked: What if we replace the "Editor" with a Super-Reader?
These Super-Readers are Large Multimodal Models (LMMs)—think of them as incredibly smart AI assistants (like GPT-4o, Gemini, or Qwen) that can look at a picture, understand the context, and read the text all at once. They are like a human expert who can look at a blurry note and say, "Ah, that says 'John Doe,' and that's definitely a patient name."
The researchers tested three different Super-Readers to see if they could do a better job than the old "Scanner + Editor" team. They set up two different ways to run the test:
- Setup A (The Traditional Team): Use the old, fast "Scanner" to read the text, then send that text to the Super-Reader to decide what to delete.
- Setup B (The All-in-One Team): Send the raw image crops directly to the Super-Reader and ask it to do both the reading and the deciding.
What They Found (The Results)
1. The Super-Readers are Great at Reading (Sometimes)
When the text on the images was messy, blurry, or hard to see, the Super-Readers (especially the big, powerful ones) were much better at reading the text than the old Scanner. It's like how a human can often decipher a messy handwriting note better than a standard typewriter.
2. But Being Smart Doesn't Always Mean Being Faster
Here is the catch: The Super-Readers are heavy and slow.
- The Speed Trap: When the Super-Reader tried to do both reading and deciding (Setup B), the process got significantly slower—sometimes 30% to 70% slower than the traditional team.
- The "Good Enough" Reality: On images where the text was already clear and easy to read, the Super-Reader didn't actually do a much better job at finding the secrets than the old Scanner. In fact, for some datasets, the traditional team was actually more accurate because the Super-Reader got confused by too much text or made small mistakes in reading.
3. The "Goldilocks" Models
The researchers tested three specific Super-Readers:
- GPT-4o: The "Heavyweight Champion." It was the most accurate at finding secrets, but it was also the slowest and most expensive to run. It's like hiring a world-class detective who takes a long time to solve the case.
- Gemini 2.5 Flash: The "Speedster." It was almost as good as the champion but much faster and cheaper. It's like a very sharp detective who works quickly.
- Qwen2.5-VL 7B: The "Open-Source Local Guy." This one is free to run on your own computers (which is great for privacy). It was good, but it sometimes got confused by complex situations, like telling the difference between a patient's ID and a study ID.
The Big Takeaway
The paper concludes that you shouldn't just blindly swap your old system for a Super-Reader. It depends on your situation:
- If you have messy, hard-to-read images: The Super-Reader is worth the wait because it can see things the old scanner misses.
- If you have clear, easy-to-read images: The old, fast scanner is often just as good and much quicker.
- If you need to keep data private: You might want to use the "Local Guy" (Qwen) that you can run on your own servers, even if it's slightly less perfect, because you don't have to send patient data to the cloud.
In short: The new AI tools are powerful, but they aren't a magic button that fixes everything instantly. Sometimes, the old, simple tools are actually the best choice, and the best solution is often a mix of both, depending on how messy the images are and how fast you need the results.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.