Footprint-Guided Exemplar-Free Continual Histopathology Report Generation

This paper introduces an exemplar-free continual learning framework for histopathology report generation that prevents catastrophic forgetting by using compact domain footprints to synthesize pseudo-WSI representations and distill linguistic styles, enabling effective adaptation to evolving clinical data without storing raw slides.

The Gist

Imagine you are a brilliant medical resident who has just finished a rotation in the Heart Department. You are an expert at writing reports for heart conditions. Suddenly, your boss says, "Great job! Now, forget everything about the heart. We are moving you to the Lung Department."

You spend months learning lung diseases. Then, you move to the Kidney Department, then the Liver, and so on.

The Problem:
Every time you learn a new department, your brain starts to overwrite the old information. By the time you finish the Liver rotation, you might forget how to describe a heart condition, or worse, you might accidentally describe a heart attack when looking at a lung scan. This is called "Catastrophic Forgetting."

In the real world, hospitals can't just keep every single patient's file (Whole Slide Images) forever due to privacy laws and storage costs. So, they can't just say, "Let's re-read all the old heart files to refresh your memory."

The Solution (The Paper's Idea):
The authors of this paper created a smart system that helps an AI doctor learn new organs without forgetting the old ones, and without needing to store the original patient files. They call this "Footprint-Guided Continual Learning."

Here is how it works, using a simple analogy:

1. The "Mental Fingerprint" (The Domain Footprint)

Instead of saving thousands of heavy, high-resolution photos of patient tissues (which takes up too much space), the AI creates a tiny, compact "Fingerprint" for each organ.

• The Analogy: Imagine you want to remember what a "Forest" looks like without carrying a photo album of every tree. Instead, you write down a small note: "Trees are mostly pine, the ground is covered in needles, and the light is dappled."
• In the Paper: The AI analyzes the new organ (e.g., the Lung) and creates a tiny "codebook" of its visual patterns (morphology) and a summary of how often certain patterns appear. This is the Footprint. It's tiny, like a single index card, but it captures the "essence" of that organ.

2. The "Dreaming" Phase (Generative Replay)

When the AI needs to learn a new organ (say, the Kidney), it needs to practice the old ones (Heart, Lung) so it doesn't forget them. But it can't look at the real old files.

• The Analogy: Instead of looking at old photos, the AI closes its eyes and dreams up fake versions of the old organs based on its "Fingerprints." It imagines, "Okay, based on my Lung fingerprint, I'll visualize a lung slide."
• The Teacher: The AI has a "Teacher" (a snapshot of itself from yesterday). The Teacher looks at these dreamed-up (fake) slides and writes a report for them.
• The Practice: The current AI then tries to write a report for these fake slides, using the Teacher's report as a guide. This keeps the old knowledge fresh without ever seeing a real patient file again.

3. The "Accent" Switch (Style Prototypes)

Medical reports aren't just about facts; they have a specific "voice" or style. A report from Hospital A might be very formal, while Hospital B might be very brief.

• The Analogy: Imagine you are an actor. When you play a character from New York, you use a specific accent. When you play a character from London, you switch accents.
• In the Paper: The AI learns a tiny "Style Token" for each hospital or organ. When it looks at a new slide, it figures out, "This looks like a Lung slide from Hospital B," and it automatically switches its "accent" to match that style. This ensures the report sounds professional and consistent, even if the hospital's writing style changes.

4. The "Mystery Guest" (Domain-Agnostic Inference)

Sometimes, the AI gets a new slide, but no one tells it which organ it is or which hospital it came from.

• The Analogy: You walk into a room and see a person. You don't know their name, but you look at their shoes, their walk, and their clothes. You think, "Hmm, those shoes look like the ones the 'Lung Team' wears." So you switch your "Lung accent" to talk to them.
• In the Paper: The AI looks at the new slide, compares it to its tiny "Fingerprints" (Footprints) of all the organs it knows, and picks the best match. It then uses that match's style to write the report. It doesn't need a label; it just figures it out on its own.

Why is this a Big Deal?

• Privacy: It doesn't need to store real patient data (which is a huge legal headache).
• Efficiency: It learns continuously as new data arrives, rather than needing to retrain from scratch every time.
• Accuracy: In tests, this method was much better at remembering old organs than other methods that tried to "forget" or "freeze" parts of the brain. It kept the AI sharp and consistent, just like a seasoned doctor who has seen it all.

In short: The paper teaches an AI how to be a lifelong learner. It creates tiny "memory cards" for each new skill, uses those cards to "dream" up practice scenarios, and switches its "voice" to match the situation—all without needing to hoard a massive library of old files.

Technical Summary

1. Problem Statement

The paper addresses the challenge of Continual Learning (CL) in automated pathology report generation from Gigapixel Whole Slide Images (WSIs).

• The Context: Current Vision-Language Models (VLMs) can generate diagnostic reports from WSIs, but they are typically trained in a static regime where all data is available simultaneously.
• The Challenge: In clinical deployment, new data arrives sequentially (e.g., new organs, institutions, or reporting conventions).
  • Catastrophic Forgetting (CF): Naïve fine-tuning on new data causes the model to forget previously learned domains.
  • Storage Constraints: Standard CL solutions use rehearsal (storing past examples), but storing raw WSIs or even intermediate patch features is prohibitively expensive due to the massive size of gigapixel images and strict patient data privacy/governance regulations.
  • Domain Agnosticism: Existing methods often require explicit domain identifiers (metadata) at inference, which may be missing or inconsistent in real-world settings.
  • Style Shift: Reporting conventions (templates, verbosity, phrasing) vary across institutions, requiring the model to adapt its linguistic style without losing diagnostic accuracy.

2. Methodology

The authors propose a Footprint-Guided Exemplar-Free Continual Learning Framework built upon a HistoGPT-style architecture. The method consists of three core components:

A. Compact Domain Footprints

Instead of storing raw images or features, the model compresses each domain into a "footprint" within a frozen patch-embedding space:

1. Codebook ($C^{(t)}$): A small set of representative morphology tokens learned via $k$-means clustering on a subsampled set of patch embeddings for the domain.
2. Histogram Bank ($H^{(t)}$): A collection of normalized histograms representing the distribution of code assignments per slide, capturing slide-level composition.
3. Statistics: Patch-count statistics (mean and variance) to reproduce realistic slide sizes during replay.
4. Style Prototype ($r^{(t)}$): A compact vector representing the linguistic style of the domain's reports, derived by mean-pooling and normalizing report embeddings from a frozen text encoder.

B. Generative Replay with Pseudo-Reports

To rehearse past domains without storing data, the model synthesizes pseudo-WSIs and pseudo-reports:

• Pseudo-WSI Synthesis: For a past domain $j$, the model samples a patch count from the stored statistics, draws a histogram from the bank, and reconstructs a synthetic set of patch embeddings by sampling from the domain's codebook.
• Pseudo-Report Generation: An "immediate teacher" (a frozen snapshot of the model from the previous episode) generates a target report for the synthetic pseudo-WSI.
• Training Objective: The current model is trained on both real data (current domain) and the synthesized (pseudo-WSI, pseudo-report) pairs using next-token cross-entropy, conditioned on the specific domain's style prototype.

C. Domain-Agnostic Inference

At test time, explicit domain labels are not required. The model:

1. Computes a reconstruction error between the test slide's patch embeddings and the codebooks of all learned footprints.
2. Selects the footprint (domain) with the minimum error.
3. Uses the selected domain's style prototype to condition the language model for generation.

3. Key Contributions

1. Exemplar-Free Framework: Introduces a method that avoids storing raw WSIs, patch features, or text reports, addressing storage and privacy constraints in clinical settings.
2. Footprint-Based Generative Replay: Proposes a novel mechanism to synthesize pseudo-WSIs and pseudo-reports using compact domain footprints (codebooks + histograms) and a teacher-student distillation approach.
3. Style-Conditioned Generation: Incorporates per-domain report-style prototypes to handle shifts in reporting conventions, enabling the model to adapt its linguistic output without explicit domain metadata.
4. Domain-Agnostic Inference: Enables the model to automatically identify the relevant domain and style based solely on the input slide content.

4. Experimental Results

The method was evaluated on two public datasets (REG2025 and PathText) across four continual learning scenarios (Organ Shift and Heterogeneous Shift).

• Performance vs. Baselines:
  • Outperforms Exemplar-Free Baselines: Significantly beats regularization-based methods (EWC, SI, LwF) which suffer from severe forgetting.
  • Competitive with Rehearsal: Matches or exceeds the performance of rehearsal-based methods (ER, DER) that use large buffers (50 samples), while outperforming those with small buffers (10-20 samples).
  • Metrics: Achieved high Average (AVG) and Incremental Learning (ILM) scores with minimal Backward Transfer (BWT) negative impact. For example, on the OS-R benchmark, the proposed method achieved an AVG of 0.6961 and ILM of 0.7319, compared to 0.6620/0.7144 for the best rehearsal baseline (DER with B=50).
• Qualitative Analysis:
  • Naïve Fine-tuning: Failed catastrophically, generating reports for the wrong organ (e.g., generating "Stomach" reports for "Bladder" slides after training on stomach data).
  • Proposed Method: Successfully preserved organ-specific context and generated accurate reports for previous domains even after training on new ones.
• Ablation Study:
  • Removing the Footprint Replay (FR) caused a massive drop in performance, confirming it as the primary driver against forgetting.
  • Adding Style Prototypes (RS) provided consistent improvements in retention and average performance, particularly in heterogeneous shift scenarios.

5. Significance

This work provides a practical solution for deploying AI in evolving clinical environments where data privacy and storage are critical constraints. By replacing heavy data storage with compact "footprints" and generative replay, the framework enables:

• Scalability: The ability to learn indefinitely new domains without linearly increasing storage costs.
• Robustness: Maintaining high diagnostic accuracy across diverse organs and reporting styles.
• Real-world Applicability: Removing the dependency on explicit domain metadata, making the system ready for deployment in settings where such metadata is often missing or unreliable.

The paper effectively demonstrates that footprint-based generative replay is a viable and superior alternative to traditional exemplar rehearsal for high-dimensional medical imaging tasks.