TRANSPORTER: Transferring Visual Semantics from VLM Manifolds

1. Problem Statement

Vision-Language Models (VLMs) have achieved remarkable success in reasoning over complex video scenes involving diverse objects, actions, and dynamics. However, understanding and controlling their internal decision-making processes remains a significant challenge. Existing interpretability methods (e.g., prompting, linear probing, or decoding hidden embeddings) often yield text-based descriptions that are:

• Sensitive to prompt variations.
• Limited in length and detail.
• Misrepresentative of the model's actual internal visual processing.
• Lacking in causal representation of how specific token predictions (logits) influence visual output.

The paper addresses the gap between high-level semantic predictions (logits) and low-level visual fidelity, proposing a way to visualize what a VLM "sees" when it predicts specific tokens.

2. Methodology: TRANSPORTER

The authors introduce TRANSPORTER, a model-independent framework that solves a new task called Logits-to-Video (L2V). The core objective is to generate videos that visually represent the changes in VLM token logit distributions.

Key Components:

A. Logits-to-Video (L2V) Task Formulation

• Given a VLM's predicted logits $\omega$ for a caption $\pi$, the goal is to generate a video $x'$ that corresponds to a specific change in logit scores ($\Delta \omega$) between two token states (e.g., changing the logit for "happy" to "sad").
• This treats the VLM's probability distribution as a causal driver for video generation.

B. Coupling Network ($\Phi$)
To bridge the gap between the generator's latent space ($\mathbb{R}^\Xi$) and the VLM's embedding space ($\mathbb{R}^\Omega$), TRANSPORTER employs a coupling network $\Phi$ consisting of two modules:

1. $\Phi_{\Omega1}$ (Global Projection): Projects generator embeddings to the VLM space using a Multi-Layer Perceptron (MLP) or Mixer, minimizing Mean Squared Error (MSE) against target VLM encodings.
2. $\Phi_{\Omega2}$ (Local Structure): Uses a Transformer to preserve the relational structure between tokens. It optimizes a Gram-matrix loss to match the inner-product relationships (token dynamics) of the target VLM embeddings.

C. Learnable Optimal Transport ($\rho$-OT)
Since the generator and VLM spaces have different dimensions and structures, a direct mapping is insufficient. TRANSPORTER introduces a Learnable Entropic Optimal Transport module:

• It uses learnable projection vectors ($p_{\Omega1}, p_{\Omega2}$) to project embeddings into scalar values.
• It computes a transport plan $\tilde{T}$ by minimizing the transport cost between these projections with entropic regularization.
• This allows the model to "transport" the generator's latent noise into the VLM's semantic manifold, effectively aligning visual features with semantic tokens.

D. Concept Bank ($Q$) and Training

• Concept Vectors: A learnable bank of vectors $Q = \{q_o\}$ is trained to represent the latent direction of semantic changes (e.g., $red \to blue$, $walk \to run$).
• Training Objective: The system minimizes the divergence between the generated video's VLM logits and the target logits.
  • It calculates the logit difference $\Delta \omega$ (using Hellinger distance) between a source condition ($\pi^-$) and a target condition ($\pi^+$).
  • This $\Delta \omega$ is used as a control parameter $\delta$ to modulate the velocity field of the text-to-video (T2V) generator (based on Conditional Flow Matching).
  • The loss function encourages the generator to produce a video where the VLM's predicted logits shift exactly by the desired $\Delta \omega$.

E. Inference
During inference, a user specifies a target attribute change. The corresponding concept vector $q_o$ is added to the generation condition. The generator produces a video where the visual attributes shift to match the target logit distribution, effectively visualizing the VLM's reasoning.

3. Key Contributions

1. L2V Task: A novel controlled generation task for VLM visual explanations, moving beyond text-based explanations to video synthesis.
2. TRANSPORTER Model: A framework that learns optimal transport paths between local geometric relationships and global representations across embedding spaces (Generator $\leftrightarrow$ VLM).
3. Learnable Latents: A concept bank that modulates videos based on target logit divergence, enabling fine-grained control over object attributes, action adverbs, and scene context.
4. Empirical Validation: Comprehensive evaluations across multiple VLMs (VideoLLaMA 3, Gemma 3, Phi 4 MM) demonstrating high visual fidelity and semantic alignment.

4. Experimental Results

The paper evaluates TRANSPORTER against baseline methods (including Activation Maximization and feature visualization techniques adapted for video) using three VLMs.

• Visual Quality: TRANSPORTER significantly outperforms baselines in Fréchet Video Distance (FVD) and LPIPS. For example, on Gemma 3, FVD improved from $2.42 \times 10^3$ (Baseline) to $1.05 \times 10^2$ (TRANSPORTER), indicating generated videos are much closer to real video distributions.
• Semantic Alignment: The method achieves superior CLIPv scores and aesthetic correspondence. On VideoLLaMA 3, CLIPv scores doubled compared to the best Activation Maximization baseline (35.44 vs. 16.74).
• Embedding Similarity: TRANSPORTER generates videos whose embeddings have significantly higher cosine similarity and lower KL divergence to real video embeddings compared to baselines.
• Qualitative Analysis:
  • Attribute Control: Successfully visualizes changes in object color, material, and count (e.g., one flower to two).
  • Action Dynamics: Captures subtle changes in action performance (e.g., "walking" to "running," "wiping" to "spraying").
  • Scene Context: Modulates background elements (e.g., "parking lot" to "underpass") while preserving foreground dynamics.
  • Partial Modulation: Can apply changes to specific time segments, showing how VLMs bind objects to actions over time.

5. Significance and Impact

• New Paradigm for Interpretability: TRANSPORTER shifts the focus from "why did the model say X?" (text explanation) to "what does the model see when it predicts X?" (visual explanation).
• Causal Insight: By generating videos based on logit divergence, it provides a causal link between specific token predictions and visual features, helping researchers understand how VLMs learn object-attribute correlations and action sequences.
• Model Agnostic: The approach works across diverse VLM architectures (different vision encoders and LLM backbones), proving its generalizability.
• Future Directions: The paper suggests that L2V could evolve into a tool for zero-shot concept visualization and continuous attribute control (e.g., sliding a "speed" vector rather than switching between "slow" and "fast").

In conclusion, TRANSPORTER offers a high-fidelity, novel approach to visualizing the internal reasoning of Vision-Language Models, bridging the gap between abstract semantic predictions and concrete visual reality.