💻 computer science

SoK: The Security-Safety Continuum of Multimodal Foundation Models through Information Flow and Global Game-Theoretic Analysis of Asymmetric Threats

This paper proposes a unified framework for analyzing the security and safety of multimodal foundation models by using information theory to map threats and a game-theoretic minimax formulation to demonstrate that system-level bandwidth constraints are more robust against adaptive attacks than model-centric defenses.

Original authors: Ruoxi Sun, Jiamin Chang, Hammond Pearce, Chaowei Xiao, Bo Li, Qi Wu, Surya Nepal, Minhui Xue

Published 2026-02-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ruoxi Sun, Jiamin Chang, Hammond Pearce, Chaowei Xiao, Bo Li, Qi Wu, Surya Nepal, Minhui Xue

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a high-tech, futuristic Smart City. This city isn't just made of bricks and mortar; it’s run by a massive, super-intelligent "Brain" (the Multimodal Foundation Model) that can see through cameras, hear through microphones, read text, and even move robotic arms to perform tasks.

This paper is a "Systematization of Knowledge" (SoK)—essentially a master manual that explains how this Smart City can be attacked and, more importantly, how to stop it from falling apart.

Here is the breakdown of the paper using the Smart City analogy:

1. The Problem: The "Blurry Line" between Safety and Security

In a normal city, Safety means preventing accidents (like a car hitting a pedestrian because of a slippery road), while Security means preventing crimes (like a thief breaking into a bank).

In our AI Smart City, these two get mixed up. If a hacker sends a "fake" image to the Brain that looks like a green light but the Brain reads it as a red light, is that a technical glitch (Safety) or a targeted crime (Security)? The authors argue that in AI, safety and security are two sides of the same coin.

2. The Framework: The "Information Pipe" Theory

To understand how the city works, the authors use a concept from "Information Theory." Imagine every piece of information (a command, a picture, a sound) travels through a pipe.

The Signal (The Message): This is the clear, useful information (e.g., "The light is green").
The Noise (The Static): This is the junk or interference (e.g., a blurry camera or a hacker adding "static" to a command).
The Bandwidth (The Pipe Size): This is how much information the system is allowed to act upon.

The Attacks:

Model-Level Attacks (Messing with the Message): This is like someone spray-painting a "Stop" sign so it looks like a "Go" sign. They are messing with the Signal or adding Noise to confuse the Brain.
System-Level Attacks (Hijacking the Pipe): This is much scarier. This is like a hacker tricking the Brain into thinking it has permission to open the city's dam gates. They aren't necessarily confusing the Brain; they are exploiting the Bandwidth—the "authorized pathways" the system uses to take action.

3. The Big Discovery: The "Asymmetry of Defense"

This is the most important part of the paper. The authors found a fundamental unfairness between attackers and defenders.

The Model-Level Defense (The "Filter" approach):
Imagine trying to stop a flood by putting increasingly fine filters on your pipes to catch every tiny piece of dirt. The problem? As the flood (the attack) gets stronger and more sophisticated, you have to use more and more filters, which eventually slows down the water (the useful information) until the city stops working. This defense has "diminishing returns."

The System-Level Defense (The "Gatekeeper" approach):
Instead of trying to filter every drop of water, you simply install a heavy-duty steel gate that only opens for verified, authorized trucks. It doesn't matter how much "noise" or "dirt" is in the water; if the truck doesn't have the right key, the gate stays shut. This is much more effective because it sets a hard limit on what can happen.

4. The "Nuclear Option": The Circuit Breaker

Finally, the authors suggest that if the city is under a massive, unstoppable attack that bypasses all filters and gates, the system needs a "Self-Destruction Threshold."

Think of it like a Circuit Breaker in your house. If there is a massive electrical surge that threatens to burn the whole house down, the breaker "trips" and cuts the power instantly. The authors propose that if an AI system detects that the "harm" has reached a certain level, it should perform a "graceful shutdown"—essentially "killing" its own processes to prevent a catastrophic disaster (like a robot arm hurting someone or a bank account being emptied).

Summary in a Nutshell:

Don't just try to make the AI "smarter" at spotting lies (Model Defense). That's a losing battle.
Instead, build "walls and gates" around what the AI is allowed to do (System Defense).
And if things go totally wrong, give the system a "kill switch" (Circuit Breaker) to prevent total chaos.

Technical Summary: The Security-Safety Continuum of Multimodal Foundation Models

1. Problem Statement

As Multimodal Foundation Models (MFMs) evolve from standalone predictors into autonomous, agentic systems, the traditional boundaries between safety (preventing unintentional harm) and security (protecting against malicious attacks) are blurring. Current research often treats these as isolated issues, focusing on specific attack primitives (e.g., prompt injection or adversarial examples) without addressing how threats propagate across modalities, agents, and memory. There is a critical lack of a unified analytical framework to reason about the "cascading failures" that occur when individually benign components interact to produce unsafe or insecure system-level behaviors.

2. Methodology

The authors propose a novel systematization of knowledge (SoK) grounded in Information Theory and Game Theory.

Information-Theoretic Framework: The paper adapts the Shannon-Hartley theorem to characterize the "Harm Capacity" of an MFM. It defines four key variables to analyze information flow:
- Channel Capacity ( $C$ ): The effective semantic capacity of the model to transmit correct concepts.
- Signal ( $S$ ): The task-aligned semantic information extracted from input.
- Noise ( $N$ ): Irrelevant, non-semantic, or disruptive data (adversarial perturbations).
- Bandwidth ( $B$ ): The "Authorized Information Pathway"—the capacity for safe, policy-compliant interactions at the system level.
Minimax Game Formulation: The interaction between attackers and defenders is modeled as a deterministic minimax game. Attackers aim to maximize damage (by increasing $N$ or decreasing $S$ ), while defenders aim to minimize it (by suppressing $N$ , enhancing $S$ , or constraining $B$ ).
Taxonomy Construction: The authors categorize threats into a hierarchy: Model-level (targeting prediction, learning, and extraction flows) and System-level (targeting agent action, interaction, and memory flows).

3. Key Contributions

Unified Taxonomy: A comprehensive mapping of 42 model-level and 31 system-level attacks, organized by their impact on information-theoretic axes ( $S, N, B$ ).
Structural Defense Asymmetry Theorem: A mathematical proof demonstrating that model-centric defenses (aiming to improve $S/N$ ) suffer from logarithmic diminishing returns against adaptive attackers. In contrast, system-level safeguards (aiming to constrain $B$ ) provide a linear reduction in harm and can act as a "Zero-Bandwidth Veto," providing deterministic protection.
Defense Coverage Index (DCI): A new metric to quantify how effectively a defense covers an expanding attack surface.
Architectural Principles: The introduction of compartmentalization (to prevent lateral movement) and self-destructive circuit breakers (to trigger system termination when a "self-destruction threshold" of harm is reached).

4. Key Results

Model-Level Vulnerability: Empirical testing shows that while purification techniques (like JPEG compression or denoising) work against simple perturbations, they fail against sophisticated "Co-attacks" that coordinate multiple modalities.
System-Level Superiority: The study confirms that system-level bandwidth constraints (e.g., using specialized tools like DataSentinel for prompt injection) provide significantly stronger and more consistent protection than model-level alignment.
Defense Effectiveness:
- Misleading Attacks: Combining purification (image) with safety filters (system) is the most robust defense.
- Mislearning Attacks: Alignment-score-based detection is highly effective at identifying poisoned data.
- Prompt Injection: While LLM-based filters are effective, they suffer from high False Positive Rates (FPR); fine-tuned system-level constraints achieve near-perfect detection with minimal FPR.

5. Significance

This paper shifts the paradigm of MFM defense from probabilistic mitigation (trying to make the model "smarter" or "more aligned") to deterministic constraint (controlling the pathways through which the model acts). By proving that system-level bandwidth control is mathematically superior to model-level noise reduction, the authors provide a roadmap for building resilient AI agents. Their proposal of "circuit breakers" offers a concrete, actionable strategy for managing existential or catastrophic risks in highly capable, autonomous multimodal systems.