Membership Inference Attacks Against Video Large Language Models

1. Problem Definition

The paper addresses the privacy vulnerability of Video Large Language Models (VideoLLMs) regarding Membership Inference Attacks (MIAs).

• The Core Question: Can an external auditor determine, with black-box access, whether a specific video was part of a model's training dataset?
• The Challenge: Unlike text-only or image-only models, VideoLLMs face unique challenges in black-box settings:
  • Limited Observability: Attackers only see generated text, not token probabilities, logits, or internal states.
  • Confounding Factors: Generation instability in VideoLLMs is not solely due to memorization. It is heavily influenced by intrinsic video properties, such as motion complexity, temporal span (duration), and frame sampling density. A model might generate unstable text for a difficult video even if it hasn't seen it before, creating false positives for membership inference.
• Goal: Develop an attack that distinguishes between instability caused by memorization (membership) and instability caused by input difficulty (non-membership).

2. Methodology: TempVideo-MIA

The authors propose TempVideo-MIA, a black-box attack framework that couples temperature-perturbed generation with video-aware difficulty features. The method operates in three stages:

A. Temperature-Based Response Signal

The attack exploits the hypothesis that member samples (seen during training) induce a sharper, more concentrated conditional generation pattern than non-members.

• Procedure: The attacker queries the target VideoLLM with the same video and prompt under two different decoding temperatures:
  • $\tau_{low}$ (e.g., 0.0): Encourages deterministic, high-confidence generation.
  • $\tau_{high}$ (e.g., 0.8): Encourages stochastic exploration of alternative phrasings.
• Metric: The attacker calculates the semantic drift between the two generated outputs relative to a reference text (ground truth caption).
  • $sim\_low$: Cosine similarity between low-temp output and reference.
  • $sim\_high$: Cosine similarity between high-temp output and reference.
  • $temp\_diff = sim\_low - sim\_high$.
• Intuition: If a sample is a member, the model has a "memorized mode." At low temperature, it stays close to this mode (high similarity). At high temperature, it diverges significantly from the memorized mode (lower similarity), resulting in a larger $temp\_diff$. Non-members, lacking a strong memorized anchor, show less dramatic shifts.

B. Video-Aware Difficulty Features

To prevent the attack from being fooled by inherently difficult videos, the authors extract auxiliary descriptors directly from the input video:

• Motion Complexity: Calculated using optical flow statistics (average motion magnitude). High-motion videos are harder to summarize consistently.
• Video Duration: Longer videos introduce more temporal context and potential for missed events during sparse frame sampling.
• Transformation: Both features are log-transformed to reduce scale sensitivity.

C. Interaction Features & Classification

The authors explicitly model the interaction between the temperature signal and video difficulty, as the significance of semantic drift depends on the video type.

• Feature Vector: A 6-dimensional vector is constructed for each sample:
  1. $sim\_low$
  2. $temp\_diff$
  3. $complexity$ (Motion)
  4. $duration\_log$
  5. $complex\_temp$ (Interaction: Complexity $\times$ Temp Diff)
  6. $duration\_temp$ (Interaction: Duration $\times$ Temp Diff)
• Classifier: Standard binary classifiers (Logistic Regression, Random Forest, SVM, MLP) are trained on these features to predict membership ($m \in \{0, 1\}$).

3. Experimental Setup

• Target Model: LLaVA-Video-7B-Qwen2-Video-Only.
• Datasets:
  • Members: 350 samples from the LLaVA-Video-178K instruction-tuning corpus.
  • Non-Members: 350 samples from the VideoChatGPT dataset (disjoint from training).
  • Control: Length-matched sampling ensures member and non-member sets have comparable video duration distributions to prevent shortcut learning.
• Protocol: 100 repeated runs with random train/test splits to ensure robustness.

4. Key Results

The attack achieved statistically significant results across four different classifiers:

• Mean AUC: 0.68 (best achieved by Multilayer Perceptron).
• Mean Accuracy: 0.63.
• Consistency: All classifiers (including linear ones like Logistic Regression) performed similarly, suggesting the signal is stable and not dependent on complex non-linear decision boundaries.
• Significance: The performance is well above random guessing (0.5), proving that membership signals are detectable even without access to internal model states.

5. Key Contributions

1. Problem Formulation: First formalization of black-box membership inference specifically for VideoLLMs, addressing the unique challenge of disentangling memorization from video-specific difficulty.
2. Novel Attack Framework (TempVideo-MIA): Introduction of a method that combines temperature sensitivity analysis with video-level descriptors (motion, duration) and their interactions to isolate membership signals.
3. Empirical Evidence: Demonstration that VideoLLMs leak measurable membership information in a strictly black-box setting, challenging the assumption that multimodal models are inherently more private due to their complexity.

6. Significance and Implications

• Privacy Risk: The study reveals that VideoLLMs are vulnerable to privacy attacks even when the attacker has no access to logits, gradients, or training metadata. The "footprint" of training data remains in the model's generation behavior.
• Auditability: Provides a practical tool for dataset owners and regulators to audit whether proprietary or sensitive video content was used in training without needing cooperation from the model developer.
• Limitations of Current Defenses: The results suggest that standard defenses (like deduplication or differential privacy) designed for text or images may not transfer directly to video, as the interaction between visual compression, temporal sparsity, and language generation creates unique leakage patterns.
• Future Directions: Highlights the need for specialized privacy benchmarks and defenses tailored to the multimodal and temporal nature of VideoLLMs.

In conclusion, the paper establishes that VideoLLMs are not membership-private by default. The proposed attack successfully leverages the interplay between decoding temperature and video complexity to expose training data leakage, urging the community to prioritize privacy auditing in multimodal foundation models.