💻 computer science

Clinically-Grounded Counterfactual Reasoning for Medical Video Diagnosis

This paper introduces MedVCR, a clinically-grounded counterfactual reasoning framework that mimics diagnostic thinking by synthesizing pathological tissue evolution and integrating clinical rules to significantly improve medical video diagnosis performance under both fully and weakly supervised settings.

Original authors: Jianzhe Gao, Churan Wang, Weiyi Zhang, Jianghua Li, Li-An Li, Wenguan Wang, Yixin Zhu, Yizhou Wang

Published 2026-05-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jianzhe Gao, Churan Wang, Weiyi Zhang, Jianghua Li, Li-An Li, Wenguan Wang, Yixin Zhu, Yizhou Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a doctor looking at a video of a patient's internal tissue, like a cervix or the inside of the colon. Their job is to spot a disease (like cancer) by watching how the tissue changes over time as different liquids (reagents) are applied.

Current computer programs trying to do this are like over-enthusiastic tourists. They see a color change or a weird texture and immediately shout, "That's a disease!" But they often get fooled by harmless things, like a sudden flash of light from the camera or a drop in water temperature. They lack the experience to ask, "Is this really a problem, or just a trick of the light?"

This paper introduces a new system called MEDVCR (Medical Video Counterfactual Reasoning). Think of MEDVCR not as a tourist, but as a seasoned detective who plays "What If?"

Here is how it works, broken down into three simple steps:

1. The "What If" Simulator (The Counterfactual Generator)

In real life, a doctor might look at a suspicious spot and think, "If this were a harmless bump, what would it look like right now? If it were cancer, how would it change?" They mentally simulate different scenarios.

MEDVCR does this with a digital time machine.

It takes the actual video frame.
It uses a special AI (called a Diffusion Model) to generate a "What If" version of that same moment.
It creates two imaginary futures: one where the tissue is healthy and one where it is sick.
Analogy: Imagine looking at a muddy puddle. The system creates a mental image of what that puddle would look like if it were clean water (healthy) and what it would look like if it were thick sludge (sick).

2. The "Rulebook" (Counterfactual Representation Learning)

Just generating fake images isn't enough; the system needs to learn how to think like a doctor. The paper teaches the AI three strict rules (Clinical Rules) to follow while it studies the video:

Rule 1: The "Steady Hand" (Temporal Consistency)
- The Logic: A disease doesn't suddenly vanish or appear just because the camera moved or the lighting changed.
- The Analogy: If you are tracking a specific car in traffic, it shouldn't disappear just because a cloud passed over the sun. The system learns to ignore the "clouds" (lighting changes) and focus on the "car" (the tissue's actual identity).
Rule 2: The "Clear Line" (Pathological Separability)
- The Logic: Sick tissue and healthy tissue are fundamentally different. They shouldn't look the same in the computer's brain.
- The Analogy: Think of a red apple and a green apple. Even if they are both wet or both dry, the system learns to keep the "Red" and "Green" categories strictly separate so it never confuses them.
Rule 3: The "Reality Check" (Counterfactual Alignment)
- The Logic: The real video should look like the "sick" imaginary version if the patient is sick, and like the "healthy" imaginary version if they are well. It shouldn't match the wrong one.
- The Analogy: If you are holding a real apple, it should match the picture of an apple, not the picture of a pear. The system constantly checks: "Does this real tissue match my 'sick' hypothesis or my 'healthy' hypothesis?"

3. The "Final Verdict" (Dual Diagnostic Prediction)

Finally, the system combines everything. It looks at the whole video (the big picture of how things are moving) AND it compares the real frame against the "What If" frames it generated.

The Analogy: It's like a judge listening to the whole story of a trial (the video timeline) but also asking, "If the suspect were innocent, would the evidence still make sense?" If the answer is "No, the evidence only makes sense if they are guilty," the system makes a confident diagnosis.

The Results

The paper tested this "detective" on two real-world medical tasks:

Colposcopy (Cervical Cancer Screening): The system had to find exactly where to take a tissue sample. It got 93% of the locations right, which was a huge jump (over 10% better) compared to previous best methods.
Colonoscopy (Colon Cancer Screening): The system had to find frames with polyps (growths) in a long video, even without being told exactly which frames they were. It achieved a 94.8% success rate, beating the previous best by a small but significant margin.

Why This Matters

The paper argues that previous AI models were just "pattern matchers"—they guessed based on what they saw. MEDVCR is different because it reasons. It simulates alternative realities and uses medical rules to decide what is real and what is a trick. This makes the AI more reliable, especially when there isn't a huge amount of data to learn from, because it relies on logic and "what-if" thinking rather than just memorizing pictures.

In short: MEDVCR doesn't just watch the video; it imagines the "what ifs," checks them against a rulebook, and uses that mental simulation to make a smarter, more trustworthy diagnosis.

Technical Summary: Clinically-Grounded Counterfactual Reasoning for Medical Video Diagnosis (MEDVCR)

1. Problem Statement

Clinical video diagnosis, such as in colposcopy and colonoscopy, requires physicians to assess dynamic tissue responses across procedural stages to detect diseases like cervical and colorectal cancers. While recent spatiotemporal models have advanced end-to-end medical video analysis, they suffer from three fundamental limitations:

Misinterpreting Pathological Evolution: Existing models often treat videos as sequences of visual changes, analyzing pixel-level variations while overlooking how tissues evolve across specific examination stages (e.g., transient reactions to reagents like acetic acid or iodine).
Ignoring Clinical Diagnostic Principles: Purely data-driven approaches frequently conflate meaningful diagnostic cues with non-pathological variations caused by illumination shifts, reagent color changes, or camera motion, lacking explicit clinical constraints.
Lacking Hypothesis-Driven Reasoning: Current models correlate observed patterns with diagnostic outputs but fail to emulate the "hypothesis-driven" reasoning of clinicians. Physicians mentally simulate counterfactual scenarios (e.g., "What if this tissue were benign instead of malignant?") to distinguish causal pathological cues from incidental correlations. This capacity is particularly critical when learning from limited clinical cases but is absent in most automated systems.

2. Methodology: MEDVCR

The authors propose MEDVCR, a hypothesis-driven counterfactual reasoning framework that integrates factual observations with clinically plausible hypothetical scenarios. The architecture consists of three core components:

2.1. Counterfactual Generator (CG)

Inspired by the clinical practice of mentally comparing observed progressions with alternative outcomes, the CG synthesizes realistic tissue transitions conditioned on specific health states (benign or malignant).

Mechanism: It is formulated as a conditional diffusion model. Given a reference frame $x_t$ from stage $s_t$ and a target health condition $h$ , the generator produces the corresponding frame $\tilde{x}^h_{t+1}$ at the subsequent stage $s_{t+1}$ .
Process: It employs a forward diffusion process to corrupt real tissue frames with Gaussian noise and a learnable reverse denoising process (using a U-shaped network) to iteratively restore diagnostic information consistent with the specified pathological state. This allows the synthesis of both benign and malignant variants for the same input, enabling the simulation of alternative diagnostic trajectories.

2.2. Counterfactual Representation Learning (CRL)

The CRL module learns spatiotemporal representations from both factual video sequences and the generated counterfactual hypotheses. It enforces three specific clinical rules to ensure the representations are physiologically coherent and diagnostically interpretable:

Temporal Consistency: Ensures that diagnostic information remains stable across examination stages for the same tissue region, regardless of reagent reactions or illumination changes. This is enforced via a temporal contrastive loss.
Pathological Separability: Enforces a clear separation between benign and malignant conditions in the latent space, ensuring features are independent across pathological categories while preserving strong diagnostic dependency. This is enforced via a soft separability loss.
Counterfactual Alignment: Requires the encoder to align factual observations with their pathology-consistent counterfactual counterparts while separating them from incompatible assumptions. This is enforced via a triplet-based alignment loss.

2.3. Dual Diagnostic Prediction (DDP)

The DDP strategy mimics the hierarchical reasoning of clinicians by integrating global temporal context with frame-level counterfactual contrast.

Video-level Assessment: Aggregates global temporal dynamics from the entire examination sequence.
Frame-level Analysis: Independently analyzes factual keyframes and their counterfactual counterparts (e.g., a suspected malignant frame vs. a generated benign hypothesis).
Fusion: The final prediction combines the video-level progression dynamics with the frame-level evidence supporting the true pathology, while simultaneously suppressing cues aligned with the alternative hypothesis. This realizes differential diagnostic reasoning at the prediction level.

2.4. Training Objective

The framework utilizes a hybrid loss function:

Generator Loss: Minimizes the mean-squared error between predicted and reference noise during the diffusion process.
Visual Encoder Loss: Combines temporal contrastive, separability, and counterfactual alignment losses derived from the clinical rules.
Diagnostic Loss: A standard supervised binary cross-entropy loss comparing predicted probabilities with ground-truth labels.

3. Key Contributions

Novel Paradigm: Introduces a counterfactual reasoning paradigm for medical video diagnosis that explicitly models tissue evolution under different disease states, bridging the gap between data-driven learning and expert clinical reasoning.
Clinical Integration: Embeds explicit clinical diagnostic principles (temporal consistency, pathological separability, and counterfactual alignment) as regularization constraints within the representation learning process.
Architecture: Proposes a unified framework comprising a diffusion-based Counterfactual Generator, a rule-regularized Representation Learning module, and a Dual Diagnostic Prediction strategy.
Versatility: Demonstrates effectiveness in both fully supervised (biopsy-site localization) and weakly supervised (polyp-frame detection) settings, as well as in static image analysis (mammography).

4. Experimental Results

The authors evaluated MEDVCR on two representative tasks and an additional static imaging task:

Fully Supervised Colposcopy (Biopsy-site Localization):
- Dataset: In-house dataset of 623 patient-specific colposcopy videos.
- Performance: Achieved 93.0% Recall@1, outperforming the best state-of-the-art (SOTA) baseline (SurgFormer) by 10.2%.
- Comparison: Significantly surpassed general video understanding models (e.g., TimeSformer, VideoSwin) and medical-specific baselines.
Weakly Supervised Colonoscopy (Polyp-Frame Detection):
- Dataset: Combined Kvasir and LDPolypVideo datasets (over 1 million frames).
- Performance: Attained 94.8% Average Precision (AP), surpassing the strongest baseline (Fadmb) by 2.6%.
- Significance: Demonstrated the advantage of counterfactual reasoning in modeling subtle pathological dynamics under weak supervision.
Ablation Studies:
- Confirmed that both the Clinical Rules (CRL) and the Dual Diagnostic Prediction (DDP) strategy contribute positively.
- Showed that the Counterfactual Alignment rule provided the largest performance gain, highlighting its critical role in linking factual and counterfactual representations.
- Validated that video-based architectures (specifically I3D) outperformed vision-language and self-supervised models for this task.
Generalization (Mammography):
- When adapted for bilateral mammography classification (static images), MEDVCR achieved 93.4% AUC, outperforming previous breast-specific methods (90.9% AUC), demonstrating versatility across modalities.

5. Significance and Claims

The paper claims that MEDVCR addresses the "black box" nature of current automated diagnostic systems by emulating the cognitive process clinicians employ in practice. By explicitly simulating and contrasting tissue evolution under different pathological conditions, the framework:

Improves Reliability: Substantially improves diagnostic robustness, particularly in data-scarce settings where distinguishing causal cues from confounding factors is difficult.
Enhances Interpretability: Provides transparent decision support by contrasting factual observations with hypothetical alternatives, making the diagnostic reasoning process more aligned with clinical principles.
Opens New Avenues: Suggests that this counterfactual reasoning paradigm can be extended to other vision-based clinical diagnostic tasks, moving beyond simple pattern recognition toward hypothesis-driven, clinically grounded AI.

The authors acknowledge limitations, noting that the generator currently models short-range transitions between adjacent stages and that the incorporated clinical rules do not yet encompass the full range of expert reasoning strategies. Future work is suggested to expand the scope of generative modeling and coverage of clinical knowledge.