💻 computer science

Parameter-efficient Prompt Tuning and Hierarchical Textual Guidance for Few-shot Whole Slide Image Classification

This paper proposes HIPSS, a novel framework for few-shot whole slide image classification that combines a parameter-efficient prompt tuning method with soft hierarchical textual guidance to overcome the computational overhead and information loss limitations of existing vision-language model approaches, achieving significant performance improvements across multiple cancer types.

Original authors: Jayanie Bogahawatte, Sachith Seneviratne, Saman Halgamuge

Published 2026-03-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jayanie Bogahawatte, Sachith Seneviratne, Saman Halgamuge

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a master art critic trying to identify a specific style of painting (like "Impressionism") just by looking at a single, massive, high-resolution digital scan of a giant mural. This mural is so huge (gigapixel scale) that you can't look at the whole thing at once. You have to zoom in and examine tiny tiles (patches) of the wall to find clues.

This is exactly the challenge doctors face with Whole Slide Images (WSIs) in pathology. These are digital scans of tissue samples that are millions of pixels wide. To diagnose cancer, a doctor needs to find tiny, specific clusters of bad cells hidden somewhere in this massive image.

The problem? Doctors can't label every single tiny tile on the wall. They only have a label for the entire slide (e.g., "This slide has breast cancer"). This is called Weakly Supervised Learning. Even worse, in rare diseases, they might only have a handful of these labeled slides to learn from. This is the Few-Shot problem.

The researchers (Jayanie, Sachith, and Saman) built a new AI system called HIPSS to solve this. Here is how it works, explained with simple analogies:

1. The Old Way: The "Over-Engineered" Translator

Previously, AI systems tried to use a "Vision-Language Model" (VLM). Think of a VLM as a super-smart translator that knows how to describe images in words (e.g., "This looks like a tumor").

However, the old methods had two big flaws:

Too Heavy: To make the translator understand the specific medical jargon, they added a massive amount of new "training wheels" (parameters). It was like trying to teach a child a new language by building a whole new school for them every time. It was slow and expensive.
The "Hard Filter" Mistake: To save time, these old systems would look at a tiny tile, ask the translator, "Does this look like a tumor?" If the translator said "No, not really," the system would throw the tile away.
- The Flaw: Sometimes a tile looks boring to the translator but actually contains a tiny, crucial clue. Throwing it away meant losing vital information, especially when you only have a few examples to learn from.

2. The New Way: HIPSS (The Smart, Efficient Guide)

The authors propose HIPSS, which fixes these issues with two clever tricks:

Trick A: The "Fine-Tuner" (Parameter-Efficient Prompt Tuning)

Instead of building a whole new school for the translator, HIPSS uses a Scaling and Shifting technique.

The Analogy: Imagine the translator is a professional chef who already knows how to cook French food. You want them to cook Thai food.
- Old Way: Hire a new team of chefs and build a new kitchen (expensive and slow).
- HIPSS Way: You just give the existing chef a tiny note: "Add a little more spice (scale) and a splash of lime (shift)."
The Result: The AI learns the new task by adjusting just a few knobs on the existing model. It's incredibly fast, requires very little memory, and doesn't slow down the computer.

Trick B: The "Soft Guide" (Hierarchical Textual Guidance)

This is the most creative part. Instead of throwing away tiles that don't look like tumors, HIPSS uses a Soft Guidance system.

The Analogy: Imagine you are looking for a specific type of bird in a giant forest.
- Old Way (Hard Filter): If a bush doesn't look exactly like a bird's nest, you ignore it completely. You might miss a bird hiding in a weird-looking bush.
- HIPSS Way (Soft Guide): You have a guidebook (the text description). When you look at a bush, the guidebook says, "This bush doesn't look like a nest, but it's somewhat similar, so keep an eye on it. That other bush looks very similar, so focus on it."
How it works: The system looks at the image in two layers:
1. Small Tiles: It groups tiny tiles into "regions."
2. Regions: It groups regions into the "Whole Slide."
  At every step, it uses the text description to gently nudge the AI's attention. It doesn't say "Ignore this." It says, "Pay a little attention to this, but pay more attention to that." This ensures no potential clue is ever lost.

3. The Results: Why It Matters

The researchers tested this on three types of cancer (Breast, Lung, and Ovarian) using very few examples (sometimes just 1 or 2 slides per type).

Better Accuracy: HIPSS was significantly better at finding the cancer than previous methods. In some cases, it improved accuracy by nearly 14% when only 1 example was available.
Faster & Cheaper: Because it only tweaks a few "knobs" instead of retraining the whole brain, it uses 18% fewer computing resources and runs faster.
Better Spotting: It didn't just guess the cancer type; it could also point to where the cancer was on the slide with high precision, almost like a spotlight.

Summary

Think of HIPSS as a smart, efficient detective.

Instead of hiring a massive team (expensive training), it uses a few clever notes to guide an expert (efficient tuning).
Instead of ignoring suspicious clues just because they don't look perfect (hard filtering), it keeps an open mind and weighs every clue carefully (soft guidance).
The result is a system that can learn to diagnose rare cancers from just a handful of examples, saving time, money, and potentially lives.

1. Problem Statement

Whole Slide Images (WSIs) in computational pathology are gigapixel-scale images requiring partitioning into smaller patches (instances) for analysis. While Multiple Instance Learning (MIL) has been successful, it typically requires large volumes of slide-level annotations, which are scarce and expensive to obtain in medical domains.

Few-Shot Weakly Supervised WSI Classification (FSWC) aims to learn from limited slide-level labels. Recent approaches leverage Vision-Language Models (VLMs) like CLIP or CONCH, pre-trained on image-text pairs, using prompt tuning (e.g., CoOp). However, existing FSWC methods face three critical limitations:

High Computational Cost: Current prompt tuning methods (often using cross-attention mechanisms) significantly increase the number of trainable parameters and inference overhead, leading to overfitting in extreme few-shot settings.
Information Loss via Hard Filtering: To reduce costs, existing methods often discard instances with low alignment to text embeddings. This "hard filtering" can eliminate diagnostically relevant regions that do not perfectly match the text description, degrading performance.
Semantic Gap: There is a distribution shift between the instance-level features VLMs are pre-trained on and the global slide-level semantics required for WSI classification.

2. Methodology: HIPSS

The authors propose HIPSS (Hierarchical textual guidance based WSI representation learning and Prompt tuning with Scaling and Shifting features). The framework consists of two core components:

A. Parameter-Efficient Prompt Tuning (Scaling and Shifting Features - SSF)

Instead of using cross-attention or learnable context vectors (CoOp), HIPSS adapts the pre-trained VLM text encoder using Scaling and Shifting Features (SSF).

Mechanism: For selected layers in the text encoder, learnable scale ( $\gamma$ ) and shift ( $\beta$ ) parameters are attached. The feature transformation is defined as: $y = \gamma \cdot x + \beta$ .
Efficiency: Only these small parameters are updated; the backbone weights remain frozen.
Inference: Through re-parameterization, these linear transformations can be merged into the model weights during inference, incurring zero additional computational overhead.
Prompt Formulation: Large Language Models (LLMs) generate two specific text descriptions: one for the entire WSI (slide-level) and one for specific regions (region-level). These are concatenated and processed by the SSF-tuned text encoder.

B. Soft Hierarchical Textual Guidance

HIPSS leverages the inherent hierarchical structure of WSIs (Slide $\to$ Regions $\to$ Instances) without discarding data.

Hierarchy:
1. Region Encoder: Aggregates instance embeddings within a region (e.g., $4096 \times 4096$ pixels) into a region embedding.
2. WSI Encoder: Aggregates region embeddings into a final slide-level embedding.
Soft Guidance: Instead of hard filtering, the model uses attention weights refined by text embeddings.
- Attention weights are calculated based on instance/region features.
- These weights are refined using a score ( $s$ ) derived from the cosine similarity between the visual feature and the text embedding ( $T$ ).
- Three Regimes:
  1. Negative similarity: Suppressed (semantically inconsistent).
  2. Low positive similarity: Linearly weighted (retains uncertain but informative instances).
  3. High similarity: Amplified (emphasizes high-confidence matches).
Training: The visual and textual embeddings are jointly optimized using a contrastive loss to align them in a shared embedding space, followed by cross-entropy loss for classification.

3. Key Contributions

New Prompt Tuning Strategy: Introduction of SSF in the text encoder for VLMs, which drastically reduces trainable parameters compared to cross-attention methods while maintaining adaptability.
Soft Hierarchical Guidance: A novel WSI representation learning strategy that integrates pre-trained VLM knowledge with the spatial hierarchy of WSIs using soft attention refinement, avoiding the information loss associated with hard instance filtering.
Comprehensive Performance: The method achieves state-of-the-art (SOTA) results across three diverse cancer datasets (Breast, Lung, Ovarian) under few-shot settings (1-shot to 16-shot).
Dual Capability: Demonstrates strong performance not only in classification but also in weakly-supervised tumor localization, proving the quality of the learned feature representations.

4. Experimental Results

The method was evaluated on Camelyon16 (Breast), TCGA-Lung, and UBC-OCEAN (Ovarian) datasets.

Classification Performance (AUC):
- Camelyon16: Outperformed SOTA by up to 10.9% (4-shot) and 7.2% (1-shot).
- TCGA-Lung: Improved by 7.8% (4-shot) and 3.7% (2-shot).
- UBC-OCEAN: Achieved a massive 13.8% improvement in the 1-shot setting over the second-best method.
Parameter Efficiency:
- Reduced trainable parameters by 18.1% on Breast and Lung datasets and 5.8% on Ovarian datasets compared to the next most efficient method (TOP).
- Compared to LoRA-based tuning, HIPSS reduced parameters by 59.7% while improving AUC by 16.4% on average.
Inference Speed: Reduced average inference time per slide by 12.6% compared to ViLa-MIL.
Localization: Achieved a Dice coefficient of 0.732 for tumor localization on Camelyon16, significantly outperforming other few-shot methods (e.g., TOP at 0.203).

5. Significance

HIPSS addresses the critical bottleneck of data scarcity in computational pathology by enabling high-performance classification with minimal supervision. Its significance lies in:

Clinical Feasibility: By reducing the need for massive annotated datasets and lowering computational costs, it makes AI deployment more viable for rare diseases and resource-constrained settings.
Robustness: The "soft" guidance mechanism ensures that potentially valuable diagnostic information is not lost due to rigid filtering, a common failure point in previous VLM-based approaches.
Efficiency: It proves that complex adaptation of large VLMs can be achieved with minimal parameter updates, making it suitable for deployment on standard hardware without sacrificing accuracy.

The paper concludes that HIPSS represents a robust, efficient, and accurate framework for next-generation computational pathology, with potential extensions to other downstream tasks like tumor segmentation.