💻 computer science

Histopathology Multi-modal Embedding for Pathology Composed Retrieval

To address the architectural, task, and domain mismatches hindering effective retrieval of interleaved pathology images and text, this paper introduces HOMIE, a model-agnostic framework that adapts generative Multimodal Large Language Models into specialized retrieval experts, achieving state-of-the-art performance on the newly proposed Pathology Composed Retrieval benchmark.

Original authors: Qifeng Zhou, Wenliang Zhong, Thao M. Dang, Hehuan Ma, Saiyang Na, Yuzhi Guo, Junzhou Huang

Published 2026-07-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Qifeng Zhou, Wenliang Zhong, Thao M. Dang, Hehuan Ma, Saiyang Na, Yuzhi Guo, Junzhou Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a pathologist (a doctor who diagnoses diseases by looking at tissue samples under a microscope) as a detective trying to solve a complex case. Usually, this detective has to search through a massive library of millions of other cases to find one that looks similar to the current mystery.

The problem is that the detective's questions are rarely simple. They don't just ask, "Show me a picture of a tumor." Instead, they might say, "Show me a picture of a tumor that looks like this specific image here, but with the text description saying it's from a patient with high fever."

Current AI tools are bad at this. They are like librarians who can only handle one type of request at a time: either "Find a picture" OR "Find a sentence," but never "Find a picture plus a sentence together."

Here is how the paper "HOMIE" solves this problem, explained simply:

1. The Problem: The "Two-Headed" Librarian vs. The "One-Headed" Genie

The paper identifies three main reasons why current AI fails at these complex, mixed questions:

The Architectural Mismatch (The Two-Headed Librarian): Old AI models use two separate "brains" (encoders). One brain reads text, and the other looks at images. When you give them a mixed question, they can't really talk to each other. They just guess based on the text or the image separately, missing the connection. It's like asking a librarian to find a book based on a cover photo and a plot summary, but the photo-reader and the text-reader are in different rooms and never share notes.
The Task Mismatch (The Creative Writer): Newer, powerful AI models (called MLLMs) are great at writing stories or generating images. They are like creative writers. But a retrieval system needs to be a search engine that ranks things by similarity, not a writer that invents new things. Using a creative writer to do a librarian's job often leads to "hallucinations" (making things up) or poor search results.
The Domain Mismatch (The Generalist): Most powerful AI models are trained on general internet data (cats, cars, movies). They don't know what a "cellular nucleus" or a "staining artifact" looks like. If you ask a general AI to find a specific type of cancer cell, it might get confused because it's used to looking at pictures of dogs, not microscopic biology.

2. The Solution: HOMIE (The Specialized Detective Assistant)

The authors propose HOMIE, which isn't a brand-new AI model from scratch. Instead, it's a training framework—a special set of instructions and exercises designed to turn a generic, creative AI writer into a specialized pathology search expert.

Think of HOMIE as a two-stage boot camp for a smart but untrained AI:

Stage 1: Learning to Search (The Task Fix)
First, the AI is trained only on text. It learns to stop "writing stories" and start "ranking answers." It learns that when you ask a question, the goal isn't to generate a new sentence, but to find the exact existing document that matches the meaning. This fixes the "Task Mismatch."
Stage 2: Learning Pathology (The Domain Fix)
Next, the AI is trained on thousands of pathology images and text descriptions. But the training is very specific:
- Native Resolution: Instead of squishing the images into tiny, blurry squares (like old AI does), HOMIE lets the AI see the images at their full, high-definition size. This is crucial because tiny details in cells matter.
- Stain Training: Pathology slides are dyed with different colors (stains). HOMIE teaches the AI to ignore the color differences and focus on the shape of the cells, so it doesn't get confused if one slide is pink and another is purple.
- Curriculum Learning: The AI learns in steps. First, it learns basic shapes of cells. Then, it learns how to combine those shapes with complex medical descriptions.

3. The Result: The "Omni-Embedding"

After this boot camp, HOMIE can take a "mixed" query (e.g., "This image of a gland + this text about fever") and turn it into a single, unified search code (an embedding).

Imagine a universal translator that can take a picture, a sentence, and a video, and compress them all into one single "ID card." When the pathologist asks a question, HOMIE creates this ID card and instantly finds the matching cases in the database, understanding the combination of clues perfectly.

4. Why It Matters (According to the Paper)

The paper tested HOMIE on a new benchmark they created called PCR (Pathology Composed Retrieval).

The Winners: HOMIE (even a small, lightweight version) crushed the competition. It beat the best existing "two-headed" models and the best "creative writer" models by huge margins.
The Efficiency: It did this with a model that is much smaller (2 billion parameters) than the specialized 7-billion-parameter models it beat. This proves that the training method (the boot camp) was the key, not just making the AI bigger.
The Safety: Because it is a retrieval system (it finds existing, real medical records) rather than a generative system (it doesn't invent new diagnoses), it is safer and more trustworthy for doctors. It acts as a "computational consultant" that provides evidence for the doctor to review, rather than making the final decision itself.

In short: HOMIE takes a smart but untrained AI, teaches it how to search instead of write, and then teaches it how to see microscopic details, turning it into a super-powered assistant that can understand complex, mixed medical questions better than any previous tool.

Technical Summary: HOMIE for Pathology Composed Retrieval

Problem Statement

The paper addresses the limitations of current AI models in computational pathology, specifically regarding retrieval-based clinical workflows. While retrieval systems offer an interpretable alternative to "black-box" predictive AI and the hallucination risks of generative models, existing approaches fail to handle real-world clinical queries.

The core problem is defined as Pathology Composed Retrieval (PCR): the task of retrieving relevant items from a database given a query composed of an interleaved sequence of pathology images, texts, and videos. Current solutions suffer from three fundamental mismatches:

Architectural Mismatch: Standard dual-encoder models (e.g., CLIP-based) process images and text separately. They lack a mechanism to deeply fuse interleaved multi-modal inputs, often reducing complex queries to shallow visual matching.
Task Mismatch: Multimodal Large Language Models (MLLMs) possess the necessary deep-fusion architecture but are optimized for token generation. Their latent spaces lack the discriminative metric structure required for effective retrieval.
Domain Mismatch: General-domain MLLMs and even some pathology-specific generative models lack the specific priors to interpret subtle cellular morphology, staining artifacts, and high-resolution pathological features.

Methodology: The HOMIE Framework

To resolve these mismatches, the authors propose HOMIE (Histopathology Multi-modal Embedding for Pathology Composed Retrieval), a model-agnostic adaptation framework. HOMIE is not a monolithic model but a systematic process that transforms any generative MLLM into a specialized pathology retrieval expert.

Architecture

The framework utilizes a vision encoder ( $f_v$ ), an MLP-based projector ( $f_p$ ), and a Large Language Model ( $f_\phi$ ).

Native Resolution & Video Processing: Unlike fixed-resolution models (e.g., 336x336), HOMIE uses the SigLIP-2 ViT architecture to process images at their native resolution by dynamically adjusting dimensions to multiples of a base patch size (14x14) and employing 2D Rotary Positional Embeddings (2D-RoPE). Video inputs are handled via frame rate sampling and interleaved multi-RoPE.
Omni-modal Embedding Extraction: To generate a single dense representation from the causal LLM, the framework employs the Explicit One-word Limitation (EOL) prompting strategy. By appending "in one word:" to the prompt, the model is forced to compress the entire multi-modal context into the hidden state of the final token, overcoming the sub-optimality of standard pooling for causal attention.

Two-Stage Adaptation Strategy

HOMIE employs a two-stage contrastive learning process to align the model:

Stage 1: Text-Only Retrieval Adaptation (Solving Task Mismatch):
- The LLM is adapted using LoRA on curated, pathology-specific text data (NLI, MedNLI, MedMCQA).
- This forces the generative model to learn a discriminative metric space suitable for retrieval before it ever processes images.
Stage 2: Pathology-Specific Curriculum Tuning (Solving Domain Mismatch):
- The entire MLLM is fine-tuned on curated pathology image-text pairs.
- Data Curation: A bootstrap approach filters noisy web data (e.g., Quilt-1M) by discarding pairs with low image-text similarity scores.
- Progressive Knowledge Curriculum: Training proceeds from foundational morphology (PathGen-1.6M) to high-level diagnostic knowledge (PathCap, filtered Quilt-1M).
- Domain Adaptations: Includes stain augmentation/normalization to ensure stain-invariant morphology and native resolution processing to retain fine-grained details.

The training objective uses InfoNCE loss for contrastive learning in both stages.

Key Contributions

Formal Definition of PCR: The paper formally defines the Pathology Composed Retrieval task and introduces the PCR Benchmark, a suite of tasks covering multi-image-to-text, image-text-to-image, image-text-to-text, and video-to-text retrieval.
HOMIE Framework: A model-agnostic framework that successfully bridges the gap between generative MLLMs and discriminative retrieval tasks in the pathology domain.
Empirical Validation: Extensive experiments demonstrating that HOMIE outperforms specialized dual-encoders and generative MLLMs on composed retrieval while maintaining strong performance on simple retrieval.

Results

Evaluated on the new PCR Benchmark, HOMIE demonstrates significant superiority over existing paradigms:

Composed Retrieval: A lightweight 2B-parameter HOMIE variant (based on Qwen3-VL-2B) achieves 75.2% Recall@1 on the Multi-Image to Text task. This substantially outperforms the best dual-encoder models (e.g., MUSK at ~44.3%) and specialized 7B pathology MLLMs (e.g., PathoR1 at 14.7%).
Simple Retrieval: HOMIE maintains strong performance on traditional image-to-text and text-to-image tasks, establishing new State-of-the-Art (SOTA) on benchmarks like Bookset, Educontent, and MMUpubmed, outperforming specialized dual-encoders like CONCH and Patho-CLIP.
Generality: The framework is effective across different backbones (Qwen2.5, Qwen3, HuatuoGPT, Lingshu, PathoR1), showing that the adaptation strategy is transferable and scales with model size.
Ablation Studies: Removing components like the progressive curriculum, native resolution processing, or stain augmentation leads to consistent performance degradation, confirming the necessity of each design choice.

Significance

The paper claims that HOMIE provides a reliable, interpretable retrieval system for pathology. By generating a single, unified multi-modal embedding for complex, interleaved clinical queries, it empowers pathologists to access relevant evidence without relying on hallucination-prone generative outputs or black-box predictions. The framework offers a direct pathway to adapt increasingly capable generative MLLMs for specialized clinical retrieval tasks, moving beyond the rigid constraints of dual-encoder architectures. The authors position this as a step toward a clinical AI assistant that preserves clinical autonomy and fosters trust.