Chest2Vec: A multipurpose text encoder conditioned by instructions for chest radiograph and computed tomography reports
The paper introduces Chest2Vec, a multipurpose, instruction-conditioned text encoder available in 0.6B and 4B parameter sizes that effectively processes both chest radiograph and CT reports for tasks like retrieval, classification, and image-text supervision, demonstrating superior performance on CT reports compared to existing models.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the world of medical imaging, a picture is often worth a thousand words, but for the computers that help doctors read those pictures, the words are just as critical. When a radiologist examines a chest X-ray or a CT scan, they write a report describing what they see: whether the lungs are clear, if there is fluid around the heart, or if a bone is broken. These reports are the primary record of what the image shows, and they have become the foundation for building artificial intelligence systems that can assist in diagnosis. To teach a computer to understand these images, engineers need to teach it to understand the language of the reports first. This requires a "text encoder," a specialized computer program that translates human sentences into a mathematical format the machine can process. For years, these programs were trained on general language or specifically on chest X-rays, but they struggled with the more complex and detailed reports generated by chest CT scans, which reveal a much wider view of the body's interior.
A team of researchers has now created a new tool called Chest2Vec to fill this gap. This system is designed to understand the text from both chest X-rays and chest CT scans, acting as a universal translator for these two different types of medical imaging. The researchers built two versions of this tool, one smaller and one larger, and tested them against existing programs to see which could best grasp the meaning of medical reports. They found that their new system, particularly the larger version, was significantly better at understanding the nuances of CT scan reports than any other available tool. It could accurately match specific findings in a report to the doctor's final conclusion, spot subtle errors in text, and even focus its attention on specific parts of the body, like the lungs or the heart, just by following a simple instruction. This work suggests that by training computers specifically on the language of chest imaging, rather than just making them bigger or using general language skills, we can create more reliable and useful tools for medical diagnosis.
The challenge the researchers faced was that the language used in chest CT reports is different from that used in chest X-rays. While both describe the same organs, a CT report is often longer and includes details about areas the X-ray cannot see, such as the upper abdomen or the blood vessels in the neck. Existing computer programs were either too general to catch these medical specifics or were trained only on the simpler language of X-rays. The researchers hypothesized that if they trained a computer model specifically on the text from both types of chest reports, it would learn to represent the meaning of the words more accurately than a model trained on general language or just one type of scan. They started with a powerful base model and then taught it using thousands of real medical reports, pairing the text with instructions that told the computer what task to perform, such as "find the error" or "summarize the findings."
To test their creation, the team put Chest2Vec through a series of rigorous challenges using data from hospitals that the computer had never seen before. In one test, they asked the system to find the correct final summary of a patient's condition based only on the detailed list of findings. On chest CT reports, the new system succeeded in finding the correct summary nearly 69 percent of the time, while the next best competitor, a program specialized only for X-rays, succeeded less than 58 percent of the time. In another test, the researchers asked the computer to identify if a report had been subtly altered to include a medical error, such as changing a negative finding to a positive one. The new system correctly identified the unaltered, correct report more than 87 percent of the time, far outperforming the other tools. These results showed that the improvement came from the specific training on chest imaging text, not just from the size of the computer model.
The researchers also discovered that the system could be directed to focus on specific parts of the body without needing to be retrained. By simply changing the instruction given to the computer, they could make it prioritize information about the lungs, the heart, or the bones, effectively tuning its attention to the area of interest. This flexibility is valuable because it allows a single computer model to serve many different purposes, from checking for specific diseases to helping doctors find similar past cases. Furthermore, the team showed that this text encoder could improve the performance of image-based AI. When they used the new text tool to help teach a computer how to "see" CT scans, the resulting image-recognition system was more accurate than those trained with older, less specialized text tools.
Despite these successes, the researchers noted several limitations. The training data for CT scans came from a single public source, which means the system might not be perfectly adapted to the reporting styles of every hospital or country. Additionally, the system was tested on English-language reports, and its performance on other languages or in different medical settings remains to be seen. The team also pointed out that the errors used in their testing were created by computers rather than occurring naturally in real-world reports, which means the system's ability to catch real-life mistakes might differ. Nevertheless, the study provides strong evidence that a specialized, instruction-guided approach works better than general methods for understanding chest imaging reports. By releasing their tools and the structured data they created, the researchers hope to enable other scientists to build even more advanced systems that can assist in the complex and vital work of interpreting medical images.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.