Technical Summary: SliceWorld

Problem Statement

CT report generation (CTRG) requires models to synthesize three-dimensional anatomical context and pathological findings from hundreds of axial CT slices. Existing approaches typically treat CTRG as a direct image-to-text mapping problem or rely on sequential aggregation of slice features. These methods often overlook the spatial and axial-sequential structure inherent in volumetric CT data. Consequently, they lack mechanisms to model how CT evidence evolves across adjacent slices or how generated reports respond to controlled changes in latent lesion-related factors. Furthermore, current methods do not provide a structured internal state that supports predictive analysis or intervention, limiting their ability to perform diagnostic reasoning or robust report generation under partial observation.

Methodology

The authors propose SliceWorld, a CT-specific world-state framework that reframes CTRG as a world-modeling problem over ordered axial CT observations. Rather than simulating physical disease progression or synthesizing HU-valued images, SliceWorld learns a latent predictive state that captures slice-level visual evidence and supports intervention analysis.

Core Architecture

1. CT Encoding and World-State Construction:

   • Input CT slices are encoded into visual features using a DINOv2 ViT encoder.
   • A causal slice-sequence encoder (Mamba) processes the prefix sequence of features to generate a contextualized representation $h_t$ for each slice $t$, ensuring no access to future slices.
   • The representation $h_t$ is decomposed into a factor-aware world state $s_t = [a_t; l_t; u_t]$, comprising three distinct components:
     • Anatomy ($a_t$): Encodes stable anatomical context.
     • Lesion ($l_t$): Encodes lesion-related evidence.
     • Uncertainty ($u_t$): Encodes predictive uncertainty.
   • A lesion-presence head predicts the probability of a lesion in the current slice based on $l_t$.

2. Report Generation:

   • The factor-aware states are projected into "world tokens" via a token projector and a two-layer MLP, mapping them into the hidden space of a Large Language Model (LLM).
   • These tokens serve as a continuous prefix context for the LLM, which generates the report autoregressively.

3. Training Framework:
   The model undergoes a two-stage training process:

   • World-Model Pretraining: Conducted on CT slice sequences (without text supervision) using four objectives:
     • Multi-step Future-slice Prediction (MFP): Trains the model to predict future slice-level visual features from the current prefix state, enforcing the learning of axial dynamics.
     • Factor-aware State Learning (FAS): Encourages specialization of the state factors through anatomical continuity regularization, lesion sparsity, uncertainty alignment with prediction difficulty, and lesion-presence supervision.
     • Counterfactual Prediction (CF): Trains the model to respond to interventions. By neutralizing the lesion factor ($l_t \to 0$), the model learns to produce "lesion-zero" predictions. The objective penalizes discrepancies in non-lesion slices (invariance) and requires increased prediction error in lesion-related slices when the lesion factor is removed (effect).
   • CTRG Fine-tuning: The pretrained world-state encoder is frozen. The projection layers and the LLM decoder are fine-tuned on paired CT-report data using standard autoregressive loss to align the world-token interface with report generation.

4. Intervention Mechanism:
   At inference, the lesion factor can be manipulated (e.g., set to zero) while keeping anatomy and uncertainty factors constant. This produces a "latent lesion-neutral" report variant, allowing for controlled analysis of how specific lesion evidence influences the generated text without altering the physical CT image.

Key Contributions

1. SliceWorld Framework: The proposal of a CT world model that learns a factor-aware state space over axial slice sequences, treating CTRG as a predictive and controllable representation problem rather than a direct mapping.
2. Novel Training Objectives: The introduction of a training framework combining multi-step future-slice prediction, factor-aware state learning, and lesion-factor counterfactual prediction to encourage representations that are predictive, intervention-sensitive, and clinically structured.
3. Empirical Validation: Extensive experiments demonstrating that SliceWorld achieves state-of-the-art results on benchmark datasets (M3D-Cap and CT-RATE), outperforms baselines with significantly larger LLM backbones, and exhibits measurable properties such as multi-horizon prediction, factor alignment, and robustness under reduced-slice observation.

Results

• Report Generation Performance: On M3D-Cap and CT-RATE, SliceWorld consistently outperforms existing representative CTRG approaches (including MedVInT-TD, Med-Flamingo, RadFM, and S-LMR) across standard natural language generation metrics (BLEU, ROUGE, METEOR, BERTScore) and clinically oriented metrics (RaTEScore, GREEN). Notably, it achieves superior performance even with smaller LLM backbones (e.g., 1.7B–4B parameters) compared to baselines using 7B–14B models.
• Ablation Studies: Step-by-step ablation confirms that each component of the world-model training (reconstruction, multi-step prediction, factor-aware learning, and counterfactual prediction) contributes incrementally to final performance.
• Predictive Capability: The model demonstrates the ability to predict future slice-level visual features with lower Mean Squared Error (MSE) and higher cosine similarity than persistence and linear extrapolation baselines, validating that the learned state retains predictive spatial information.
• Factor Alignment: Probing experiments show that the anatomy, lesion, and uncertainty factors align with their intended clinical signals (e.g., lesion factors correlate with lesion presence labels, uncertainty factors with prediction difficulty).
• Controlled Intervention: Selective lesion-factor intervention (lesion-zero) successfully reduces focal-lesion content in reports for lesion-positive cases while preserving non-target findings and maintaining clinical validity, demonstrating the model's ability to modulate reports based on latent factors.

Significance

The paper argues that SliceWorld moves CTRG beyond direct volume-to-text mapping toward a predictive and controllable representation interface. By framing the problem as world modeling, the approach provides a structured internal state that:

1. Captures the dynamics of axial CT evidence across slices.
2. Supports latent lesion-factor manipulation for report-level analysis.
3. Enhances robustness when only a subset of slices is available.

The authors position this as a step toward more interpretable and clinically grounded medical AI, where the internal state of the model can be probed for diagnostic signals and manipulated to understand the causal impact of specific findings on generated reports, rather than treating the model as a black-box image-to-text converter.