🧬 biology

HERO: Hypothesis-Driven Evidence Retrieval from Omics for Multi-Task Breast Cancer Analysis

HERO introduces a hypothesis-driven framework that leverages matched multi-omics data to generate testable morphology priors for retrieving and verifying relevant regions in whole-slide images, achieving state-of-the-art performance in multi-task breast cancer analysis while ensuring lexical auditability and reducing reliance on purely semantic matching.

Original authors: Xiangyu Li, Ran Su

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xiangyu Li, Ran Su

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine you are a detective trying to solve a complex medical mystery: Breast Cancer. You have two main pieces of evidence:

The Crime Scene Photos: High-resolution images of the tumor tissue (called Whole Slide Images or WSIs). These are massive, gigapixel-sized photos where you can zoom in to see individual cells.
The Witness Statements: Molecular data from the patient's DNA and RNA (called "omics"). This tells you what the cancer cells are doing at a genetic level.

The problem with current detective methods is that they often look at the photos and the witness statements separately, or they just guess which part of the photo is important. They might zoom in on a bright, colorful spot in the photo that looks interesting but isn't actually relevant to the specific genetic clues.

HERO is a new detective system that changes the game. Instead of guessing, it uses the "Witness Statements" (the genetic data) to write a specific hypothesis about what the "Crime Scene" (the tissue image) should look like. It then goes hunting for exactly those features.

Here is how HERO works, broken down into simple steps:

1. The "Intent Vector" (The Detective's Wishlist)

First, HERO reads the patient's genetic data (DNA methylation and miRNA). It doesn't just read the numbers; it translates them into a 16-point checklist of what the tissue should look like.

Analogy: Imagine the genetic data tells the detective, "The suspect is likely to be aggressive and have a lot of cell division." The system translates this into a specific wishlist: "Look for high cell density, signs of rapid division, and specific structural chaos."
This wishlist is called an Intent Vector. It's a strict, pre-defined plan based on biology, not a random guess.

2. The "Keyword Search" (Finding the Right Photos)

The tissue image is too huge to look at every single pixel. So, HERO first picks 300 small "sample" patches from the image.

It asks an AI (a Visual Language Model) to write a short caption for each patch, describing what it sees (e.g., "lots of dead cells," "dense clusters," "inflammation").
Then, it uses a classic search engine technique (TF-IDF) to match the Intent Vector's checklist against these captions.
Analogy: It's like searching a library. Instead of reading every book, you search for the specific keywords from your wishlist. The system picks the top 64 patches that best match the genetic clues.

3. The "Consistency Gate" (The Reality Check)

This is the most clever part. After picking the top 64 patches, HERO checks: "Does what we actually see in these photos match our genetic hypothesis?"

It calculates a "consistency score." If the score is high, great! The photos confirm the genetics.
The "Deficit-Driven Repair": If the score is low (meaning the photos don't match the genetic clues), the system doesn't give up. It figures out what is missing (e.g., "We have plenty of inflammation, but we are missing the cell division signs").
It then goes back, finds 32 new patches that specifically fill those missing gaps, and swaps them in.
Analogy: Imagine you are building a puzzle based on a picture on the box. You put in the pieces, but the sky is missing. Instead of staring at the incomplete puzzle, you go back to the box, find the specific "sky" pieces you missed, and swap them in. You only do this once to keep things efficient.

4. The Final Diagnosis

Once the system has a perfect, verified set of 64 image patches that match the genetic clues, it feeds this "evidence mosaic" into a final AI doctor.

This AI looks at the verified evidence and the genetic story together to make the final diagnosis (predicting things like tumor type, risk level, or treatment response).

Why is this better?

No Guessing: Old methods often get distracted by pretty colors in the image. HERO only looks for what the genetics say is important.
Auditable: Because the system uses a checklist and specific keywords, you can see exactly why it picked certain patches. It's not a "black box"; it's a transparent process.
Self-Correcting: If the first search misses something, the "repair" step fixes it automatically.

The Results

The researchers tested this on 930 breast cancer cases. HERO beat all other methods, including those that just mash all the data together or use powerful AI without this specific "hypothesis-driven" search.

It was especially good at finding the right answers for tricky cases where the visual signs are rare or scattered (like HER2 status or risk prediction).
It proved that using genetics to guide the search for visual evidence is much smarter than just looking at the pictures or the genetics separately.

In short: HERO is a detective that uses the suspect's genetic profile to write a specific "Wanted" poster, searches the crime scene photos for exactly what's on that poster, double-checks if the photos match the description, and fixes the search if anything is missing, all before making the final arrest (diagnosis).

Technical Summary: HERO – Hypothesis-Driven Evidence Retrieval from Omics for Multi-Task Breast Cancer Analysis

Problem Statement

Breast cancer diagnosis requires synthesizing evidence from H&E whole-slide images (WSIs) and molecular biomarkers (ER/PR/HER2, subtype, prognosis). While matched multi-omics data can improve prediction, existing pipelines face three primary limitations:

Weakly Supervised MIL: Relies on slide-level labels where attention mechanisms are post-hoc and fail to verify evidence sufficiency, often highlighting non-diagnostic regions due to intratumoral heterogeneity.
VLM-Based Readers: While making reading explicit, they often rely on prompt-level guidance rather than fixed, molecularly conditioned retrieval priors, leading to bias toward visually salient but endpoint-irrelevant regions.
Multimodal Fusion: Treats omics as parallel feature streams or appended text, failing to use omics to explicitly control which regions are retrieved or to verify the completeness of that retrieval.

The core challenge is endpoint-aware evidence retrieval on gigapixel WSIs that ensures the observed morphology explicitly supports the molecular conclusion, creating an auditable evidence trail.

Methodology: The HERO Pipeline

HERO (Hypothesis-Driven Evidence Retrieval from Omics) is a four-stage, closed-loop pipeline that treats observed omics as a testable morphology hypothesis. It operates in a shared 16-dimensional evidence space ( $K=16$ ).

Stage 1: Omics to Intent and Keywords

Pathway Scoring: DNA methylation ( $x_{dna}$ ) and miRNA ( $x_{mirna}$ ) vectors are z-normalized. Pathway scores are computed using pre-defined binary masks derived from MSigDB Hallmark gene sets and validated miRNA-target interactions. The scoring maps hypermethylation and miRNA-mediated repression to pathway down-regulation.
Morphology Prior ( $W$ ): A fixed, signed, weighted sparse prior matrix ( $W \in \mathbb{R}^{K \times P}$ ) maps the 50 pathway scores to 16 morphology axes (e.g., mitotic activity, nuclear pleomorphism, stromal desmoplasia). This prior is guided by WHO terminology and biological associations (e.g., E2F/MYC $\to$ proliferation) and is never optimized with endpoint labels, ensuring the mapping remains auditable and fold-invariant.
Intent Generation: The intent vector $m$ is generated via $m = \ell_2\text{-normalize}(\text{ReLU}(Ws))$ . A committee of LLM calls generates a molecular narrative and a structured keyword set $Q$ (15–40 terms) based on the pathways and axes.

Stage 2: 10× Retrieval (300 $\to$ Top-64)

Representative Mining: The WSI is encoded with UNI embeddings. MiniBatchKMeans ( $k=300$ ) clusters these to select 300 representative patches covering global morphological diversity.
Captioning & Ranking: These 300 patches are revisited at 10× magnification and captioned by a VLM (Qwen2.5-VL-7B) using a structured prompt. A TF-IDF vectorizer (fitted per slide) ranks these captions against the keyword set $Q$ . The top 64 patches form the initial evidence set.

Stage 3: 20× Consistency Gate and Deficit-Driven Repair

Visual Evidence Vector ( $v$ ): The top 64 patches are cropped at 20×, arranged into an 8×8 mosaic, and captioned again with a focus on cytologic features. A visual evidence vector $v \in \mathbb{R}^K$ is constructed by checking for the presence of checklist terms for each of the 16 axes in the captions.
Consistency Gate: A cosine similarity score $c = \cos(m, v)$ measures alignment between the molecular intent and visual evidence.
Deficit-Driven Repair: If $c < \tau_c$ (threshold $\tau_c=0.5$ ), the system identifies under-supported axes ( $m - v$ ). It re-ranks the remaining unselected representatives based on their ability to cover these deficits, merges the top 32 repair candidates with the original 64, and performs one additional 20× captioning pass. No further repair rounds occur.

Stage 4: Diagnosis Module

The final mosaic, molecular narrative, and consistency summary are fed into a LoRA-tuned Qwen2.5-VL-7B. The model outputs structured JSON predictions for ER, PR, HER2, subtype, and risk. Only this stage is tuned; Stages 1–3 remain fixed retrieval and verification modules.

Key Contributions

Omics-Guided Retrieval: Converts DNA methylation and miRNA into a sparse pathway-to-morphology intent vector, enabling endpoint-aware patch selection via TF-IDF over structured captions rather than embedding-based semantic matching.
Consistency Gate with Deterministic Repair: Introduces a cosine gate to detect molecular-visual mismatches. When a mismatch is detected, a deterministic, deficit-driven re-ranking mechanism retrieves patches to cover under-supported morphology axes, bounding compute to a single repair round.
Robustness and Audibility: The design bounds VLM calls, reduces reliance on black-box semantic matching, and ensures every retrieval step is lexically auditable. The fixed prior $W$ prevents shortcut learning and ensures the mapping is interpretable.

Experimental Results

Evaluated on TCGA-BRCA (930 WSIs, 878 patients) with patient-level 5-fold cross-validation:

State-of-the-Art Performance: HERO outperformed all baselines (including SNN, TransMIL, MCAT, SurvPath, and VLM-based agents) across all endpoints: ER, PR, HER2, Subtype, and Risk.
- Notable gains were observed in HER2 (+0.050 AUC over SurvPath) and Risk (+0.030 C-index), where morphology is sparse and heterogeneous.
Ablation Studies:
- Retrieval Mechanism: Replacing HERO's retrieval with TransMIL attention or random sampling significantly degraded performance (e.g., HERO improved HER2 by +0.090 over VLM + TransMIL Top-64), confirming gains stem from molecular-guided retrieval.
- Components: Removing the consistency gate reduced HER2 AUC by 0.043. Random patch repair recovered only a fraction of the performance, proving that deficit-driven re-ranking is critical.
- Sensitivity: The system is robust to perturbations in the prior $W$ (noise/dropout caused $\le 0.009$ degradation), whereas shuffling the pathway-to-axis assignments caused significant drops, validating that $W$ encodes informative biological structure.

Significance and Claims

The paper claims that HERO establishes a new paradigm where omics data acts not merely as a parallel feature stream, but as an explicit retrieval constraint and verification mechanism. By closing the loop between molecular hypotheses and visual evidence, HERO achieves state-of-the-art accuracy while providing a lexically auditable evidence trail.

The authors explicitly note limitations:

Evaluation is restricted to internal TCGA-BRCA cross-validation; robustness across institutions, scanners, and populations is not established.
The 16-axis checklist is specific to breast cancer and requires auditing for other tumor types.
VLM captions remain fallible; while structured prompts make evidence inspectable, they do not guarantee pathologic correctness.
Inference cost is high (57s/WSI on 2×A100), necessitating faster multi-stage VLM inference for clinical deployment.

The work concludes that mapping omics to intent, retrieving via TF-IDF captions, and verifying alignment with a cosine gate allows for a robust, interpretable, and high-performance multi-task breast cancer analysis system.