💻 computer science

Are Video Models Emerging as Zero-Shot Learners and Reasoners in Medical Imaging?

This paper demonstrates that large autoregressive video models can exhibit remarkable zero-shot capabilities in medical imaging, performing competitively on tasks like organ segmentation, denoising, and super-resolution, and even achieving state-of-the-art accuracy in 4D CT motion prediction without ever being trained on medical data.

Original authors: Yuxiang Lai, Jike Zhong, Ming Li, Yuheng Li, Xiaofeng Yang

Published 2026-04-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuxiang Lai, Jike Zhong, Ming Li, Yuheng Li, Xiaofeng Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The "Universal Translator" for Medical Movies: A Simple Explanation

Imagine you have a world-class professional dancer who has spent their entire life watching nothing but Hollywood action movies and ballet performances. They have never seen a medical textbook, and they have never stepped foot in a hospital.

Now, imagine you show this dancer a series of X-ray "snapshots" of a person breathing. Even though they’ve never seen a human lung or liver before, you ask them: "Based on how the person moved in the first few shots, can you draw what they will look like in the next five shots?"

To everyone's shock, the dancer doesn't just guess—they perform. They predict the movement of the organs with incredible accuracy, almost as if they understood the "physics" of the human body.

That is essentially what this research paper has achieved.

The Core Idea: The "Smart Observer"

Traditionally, medical AI is like a specialized tool: you have a "hammer" for finding tumors, a "screwdriver" for measuring organs, and a "level" for tracking motion. If you want to do something new, you have to build a whole new tool from scratch.

The researchers decided to try something different. They took a Large Vision Model (LVM)—a massive AI that was trained on millions of regular videos (like YouTube clips, movies, and nature footage)—and dropped it into a medical environment without giving it any "medical school" training. This is called "Zero-Shot Learning." It’s like dropping a genius polymath into a specialized lab and seeing if they can figure things out just by observing.

What did the AI actually do?

The researchers tested this "uneducated" AI on four different "medical chores":

The Sketch Artist (Segmentation): They asked it to trace the outlines of organs like the liver or lungs. Even though it had never seen a CT scan, it could "see" where one organ ended and another began.
The Photo Editor (Denoising & Super-Resolution): They gave it blurry or grainy medical images and asked it to clean them up. It acted like a high-end Photoshop filter, making the images crisp and clear.
The Fortune Teller (Motion Prediction): This was the "superpower" moment. In cancer treatment (radiotherapy), doctors need to know exactly how a tumor moves as a patient breathes so they don't hit healthy tissue by mistake. The AI watched the first few "frames" of a patient's breathing cycle and successfully "predicted the future," drawing what the organs would look like in the next stages of the breath.

Why is this a big deal? (The "Aha!" Moment)

The most amazing part? In the motion prediction task, this "uneducated" AI actually beat specialized medical models that had been trained specifically for that job.

Why did it win?
Because the AI understands "The Rhythm of the World." By watching billions of seconds of regular video, it learned the fundamental rules of how things move: how objects stretch, how they flow, and how they maintain their shape over time. It applied those "universal rules of motion" to the human body.

The Metaphor: From "Specialists" to "Generalists"

Old Way (Specialists): A library filled with thousands of tiny, thin books. One book only tells you about the liver; another only tells you about lung shadows. If a new disease appears, you have to write a new book.
New Way (The Video Model): A single, massive, "living" encyclopedia that understands the concept of shape, movement, and light. It doesn't need a new book; it just needs to look at the new data and use its "common sense" to figure it out.

The Bottom Line

This paper suggests that we might not need to build a thousand different AI models for a thousand different medical tasks. Instead, we might be able to build one "Medical Foundation Model"—a digital brain that understands the "movie" of the human body, capable of seeing, cleaning, and predicting medical images with almost no specific training.

Technical Summary: Are Video Models Emerging as Zero-Shot Learners and Reasoners in Medical Imaging?

1. Problem Statement

Traditionally, medical imaging AI relies on highly specialized, task-specific architectures (e.g., U-Net for segmentation, DVF-based models for motion) that require extensive domain-specific training and manual pipeline engineering. This fragmentation makes it difficult to create scalable, unified systems. While Large Language Models (LLMs) have achieved remarkable zero-shot reasoning, the medical imaging field lacks a unified framework capable of performing diverse tasks—ranging from low-level image restoration to high-level temporal reasoning—without task-specific retraining. The authors investigate whether large-scale, general-purpose Large Vision Models (LVMs), trained only on natural videos, can exhibit emergent zero-shot capabilities when applied to complex medical datasets like 4D CT scans.

2. Methodology

The researchers utilized a Large Vision Model (LVM) as a backbone. LVM is an autoregressive Transformer trained on massive natural image/video datasets, treating visual sequences as "visual sentences."

Core Architecture: The model uses a decoder-only Transformer (based on the LLaMA architecture) with approximately 120 million parameters. It employs Vector Quantized Generative Adversarial Networks (VQGAN) to tokenize 3D CT slices into a compact sequence of discrete visual tokens, making autoregressive modeling computationally tractable.
Task Formulation:
- Motion Prediction: Formulated as an autoregressive generation problem where the model predicts the next CT phase ( $X'_t$ ) conditioned on all previous phases ( $X_0, \dots, X_{t-1}$ ).
- Other Tasks: Segmentation, denoising, and super-resolution were performed via prompting (e.g., text prompts for organ segmentation) without fine-tuning.
Data & Pre-processing: The study used 4D CT data from 122 patients (including lung, heart, and liver scans). To enhance performance, they tested a multi-modal input approach combining raw CT intensities with anatomical segmentation masks (generated via nnUNet) to provide both texture and structural priors.

3. Key Contributions

Zero-Shot Generalization: Demonstrated that a model with zero exposure to medical data can successfully perform diverse medical tasks.
Unified Framework: Proved that a single video modeling pipeline can replace multiple specialized architectures for segmentation, enhancement, and motion modeling.
Temporal Reasoning: Identified that autoregressive video models are inherently superior at capturing patient-specific, non-linear physiological dynamics (like respiration) compared to traditional deformation-based models.
Ablation Insights: Established that combining pixel-level (CT) and semantic-level (Mask) information significantly improves the accuracy of motion modeling.

4. Results

Motion Prediction (State-of-the-Art): The LVM significantly outperformed specialized baselines (such as ConvLSTM, RMSim, and diffusion-based models like DiffuseRT). In predicting lung and liver motion, the LVM achieved higher Dice Similarity Coefficients (DSC) and Intersection over Union (IoU), while maintaining lower error rates in Hausdorff Distance (HD) and Surface Distance (SD).
Multi-phase Stability: Unlike supervised models that suffer from "error accumulation" (where accuracy drops sharply as they predict further into the future), the LVM maintained high temporal coherence and smooth transitions across all predicted phases.
Low-level & Structural Tasks:
- Segmentation: Achieved a DSC of 91.52% in a zero-shot setting.
- Denoising/Super-resolution: Produced competitive PSNR and SSIM values, demonstrating an ability to infer anatomical textures and spatial priors.

5. Significance

This paper provides foundational evidence that video models are emerging as general-purpose visual reasoners capable of transferring knowledge from natural domains to the highly specialized medical domain.

The findings suggest a paradigm shift: instead of building hundreds of small, task-specific medical models, the future of medical AI may lie in Medical Foundation Models built on large-scale video architectures. Such models could provide adaptive, real-time support in clinical workflows, such as motion-adaptive radiotherapy, where the ability to reason about how organs move in space and time is critical for patient safety and treatment precision.