Hierarchical temporal transformer for cancer grade prediction and cross cancer transfer learning from pathology reports
This paper introduces the Hierarchical Temporal Transformer (HTT), a two-level architecture that leverages longitudinal pathology reports and cancer-type embeddings to achieve state-of-the-art cancer grade prediction and demonstrate effective zero-shot transfer learning to unseen cancer types without performance penalties.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer
Hospitals generate a constant stream of written records. When a patient visits a doctor, the physician writes down what they see, what they suspect, and how the patient's condition has changed since the last visit. For decades, computers have struggled to read these notes in a way that helps doctors make decisions. Most computer programs that analyze medical text treat every note as a single, isolated event. They read one report, make a guess, and then move on to the next, completely forgetting the story that came before. This approach misses a crucial part of the human experience: time. A patient's health is a journey, not a snapshot. A scan showing a small change today might mean something very different if the patient had a stable scan two years ago versus one that showed rapid growth last week. Furthermore, most computer models are trained to understand only one specific type of cancer. A program built to read lung cancer reports often fails when handed a report about thyroid cancer, forcing hospitals to build and maintain dozens of separate, expensive systems.
A team of researchers set out to fix both of these problems at once. They developed a new computer system designed to read the entire history of a patient's visits, paying close attention to the actual days and months between appointments. They also wanted to see if this system could learn the general language of cancer severity so well that it could understand types of cancer it had never seen before. To test this, they created a two-part experiment. First, they built a simulated world of medical reports where they could control the story exactly, ensuring the computer had to use the timeline to get the right answer. Second, they fed the system thousands of real pathology reports from a public database, asking it to judge the severity of tumors in cancer types that were completely hidden from its training.
The system they built works in two distinct stages, much like a human reading a file. The first stage reads each individual report and turns the words into a dense summary of what that specific visit meant. The second stage looks at the sequence of these summaries for a single patient. It does not just look at the order of the reports; it measures the actual time that passed between them. If a patient had a scan three weeks after surgery and another six months later, the system understands that the second event is much more distant and significant than if the scans were only a week apart. This allows the computer to weigh the history of the disease, not just the most recent note. To handle different types of cancer, the system uses a special marker for each disease family, teaching it that while the words might change, the underlying concept of "aggressive" or "mild" remains similar across different organs.
When the researchers tested this system on their simulated data, the results were clear. The computer that could read the full timeline and the time gaps between visits was significantly better at predicting disease progression than a computer that only looked at the latest report. In a specific test where the system had to predict the outcome for a type of cancer it had never seen before, the timeline-aware system was far more accurate. It successfully used the pattern of growth it had learned from other cancers to understand the new one. This suggested that the ability to track time and history was a genuine advantage, not just a feature of the data.
The second part of the study moved from simulation to reality, using thousands of real pathology reports from patients with fourteen different types of cancer. The researchers trained the system on eleven of these types and then completely locked away the reports for the other three, including thyroid cancer, sarcoma, and a specific kind of lung cancer. They asked the system to predict the tumor grade, which is a measure of how aggressive the cancer is, for these unseen types. The results were striking. The system performed just as well on the unseen cancers as it did on the ones it had been trained on. It achieved a perfect score on thyroid cancer and very high scores on the others. This proved that the system had not simply memorized specific words for specific cancers. Instead, it had learned a deeper, universal language of cancer severity that applies across different families of disease.
The researchers then investigated exactly how this transfer of knowledge happened. They found that the ability to understand new cancer types came primarily from the system's initial training on vast amounts of medical text, which taught it the general meaning of words like "aggressive" or "poorly differentiated." The part of the system designed to track time and history, while powerful for understanding patient journeys, was not the main driver of this cross-cancer success. In the real-world test, where each patient had only one report, the time-tracking component had no history to analyze, yet the system still succeeded. This indicates that the core intelligence for recognizing cancer severity was already present in the foundational language model, waiting to be applied to new situations.
These findings suggest a new path for medical artificial intelligence. Instead of building a separate, specialized computer for every single type of cancer, hospitals might eventually be able to use one unified system. This single system could learn from common cancers and immediately apply that knowledge to rare or difficult ones, provided the concept of disease severity is expressed in the text. While the system still needs careful tuning to avoid false alarms, the study demonstrates that computers can learn to read the story of a disease across different bodies and different timeframes, moving closer to a future where a single tool can assist doctors in understanding the full complexity of cancer.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.