Physics-based phenomenological characterization of cross-modal bias in multimodal models

This position paper proposes a physics-based phenomenological framework to characterize and address cross-modal bias in multimodal large language models, arguing that analyzing transformer dynamics through physical surrogates reveals systematic distortions and error-attractor patterns that conventional embedding-level analyses miss.

The Gist

The Big Idea: When AI Gets "Deaf" to One Sense

Imagine you are trying to understand a movie scene. You have your eyes (seeing the actors' faces) and your ears (hearing their voices). In a perfect world, your brain combines both to get the full picture. If the actor looks sad but says "I'm happy," you might guess they are lying or confused.

This paper argues that Multimodal Large Language Models (MLLMs)—AI that can see and hear—often fail at this. Instead of blending the two senses perfectly, they tend to ignore one and obsess over the other.

The authors call this "Cross-Modal Bias." It's like a person who is so focused on reading a script that they completely ignore the actor's facial expression, even when the expression tells a different story.

---

Part 1: The Experiment (The "Emotion Test")

To prove this, the researchers played a game with two different AI models (Qwen2.5-Omni and Gemma 3n).

The Setup:
They showed the AI videos of actors acting out emotions (happy, sad, angry, etc.). They tested three scenarios:

1. Face + Voice: The full video with sound.
2. Face Only: The video with the sound muted.
3. Voice Only: The audio with the video replaced by a blank screen.

The Surprise:
You might think that adding the voice to the face would help the AI understand better. But the results showed something weird:

• When the AI saw the Face, it made decisions based almost entirely on the face.
• When they added the Voice to the Face, the AI didn't change its mind. It didn't say, "Oh, the voice sounds angry, so maybe the face is lying." It just stuck to what the face told it.
• The "Voice" input was treated like background noise. It didn't help; it just got ignored.

The Analogy:
Imagine you are at a party. You are looking at a friend's face (which looks happy) while they are shouting "I'm furious!"

• A human would pause and think, "Wait, their face says happy, but their voice says angry. Something is up."
• This AI acts like a stubborn person who only looks at the face and says, "They look happy, so they must be happy," completely tuning out the shouting.

The researchers found that the AI has a "favorite" sense (usually vision) and treats the other sense as if it doesn't exist. This is dangerous because in real life (like medical diagnosis), if an AI ignores an X-ray because it's too focused on the text description, it could miss a life-threatening condition.

---

Part 2: The "Physics" Explanation (The Orchestra Metaphor)

The authors didn't just say "the AI is biased." They wanted to know why it happens inside the machine's brain. To do this, they used a Physics-Based Model.

The Analogy: The Chaotic Orchestra
Think of the AI's internal processing like a massive orchestra playing music.

• The Musicians: Each note or "token" in the AI is a musician.
• The Sections: There are two sections: the Strings (representing the Video) and the Brass (representing the Audio).
• The Conductors: The AI has two types of conductors:
  1. Self-Attention: The conductor telling the Strings to listen to other Strings.
  2. Cross-Attention: The conductor telling the Strings to listen to the Brass.

What Went Wrong?
The researchers modeled this system using equations similar to how pendulums or chaotic weather systems move (called the Lorenz system).

They found that for the orchestra to play a beautiful, accurate song (a correct prediction), the conductors need to be very active. The "Strings" and "Brass" need to talk to each other constantly and loudly.

However, in many current AI models:

• The conductors are lazy or weak.
• The Strings (Video) are so loud and confident that they drown out the Brass (Audio).
• The Brass tries to speak up, but the "Cross-Attention" mechanism is too weak to let them be heard.

The Result:
The music becomes unbalanced. The AI predicts the outcome based only on the loud section (Video), ignoring the quiet section (Audio). The "physics" of the model shows that unless the connection between the two groups is strong enough, the system naturally collapses into relying on just one side.

---

Part 3: Why This Matters (The "Black Box" Problem)

Usually, when we check if an AI is fair, we look at the final score: "Did it get 90% of the answers right?"

The Problem:
The AI might get 90% right, but it might be getting them right for the wrong reasons. It might be ignoring the audio completely and just guessing based on the video. Standard tests don't catch this because they only look at the final grade, not how the student studied.

The Solution Proposed:
The authors suggest we need a new way to look at AI. Instead of treating the AI like a "black box" that magically thinks, we should treat it like a physical machine with moving parts (like gears, springs, or oscillators).

By using this "physics" view, we can see the hidden distortions in how the AI processes information. We can see that the "gears" for cross-modal communication are slipping, causing the bias.

Summary in One Sentence

This paper argues that current AI models often ignore one sense (like hearing) in favor of another (like sight) because their internal "wiring" isn't strong enough to blend them, and we need to use physics-based tools to fix this hidden imbalance before it causes real-world harm.

Technical Summary

1. Problem Statement

While Multimodal Large Language Models (MLLMs) have advanced significantly in understanding, reasoning, and generation, they suffer from systematic cross-modal bias.

• The Phenomenon: Instead of integrating information from multiple modalities (e.g., text, audio, video) to improve robustness, MLLMs often exhibit modality dominance, where decisions are driven primarily by a single modality (usually text or video), rendering other modalities useless or even detrimental to performance.
• The Gap: Traditional evaluation methods (aggregate accuracy metrics) and cognitivist approaches (analyzing static embeddings or symbolic representations) fail to capture the inconspicuous distortions in the model's decision-making dynamics. These distortions arise from the complex, non-linear interactions within the transformer architecture (self-attention and cross-attention) but are invisible in standard performance reports.
• The Goal: The authors argue that these biases are not merely representational artifacts but emergent properties of transformer dynamics. They propose a physics-based phenomenological approach to characterize these dynamics, moving away from metaphysical or purely symbolic explanations.

2. Methodology

The paper employs a two-pronged methodology: Empirical Diagnostic Analysis on real-world MLLMs and Theoretical Dynamical Modeling using a physics-based surrogate.

A. Empirical Diagnostic Analysis

• Models Tested: Two architecturally distinct MLLMs: Qwen2.5-Omni and Gemma 3n.
• Task: Zero-shot emotion classification using the CREMA-D dataset (7,442 samples of actors expressing emotions with synchronized audio and video).
• Experimental Conditions:
  1. Multimodal: Video (Face) + Audio (Voice).
  2. Unimodal Video: Video only (Audio replaced with silence).
  3. Unimodal Audio: Audio only (Video replaced with blank frames).
• Perturbation Strategy: The authors applied systematic label perturbation, explicitly prohibiting the models from selecting specific subsets of emotion labels (e.g., removing "Neutral" or "Happy" from the output space) to observe how error patterns shift.
• Visualization: Used directed graphs to map error attractors (misclassification patterns) and Sankey diagrams to visualize bias hierarchies.

B. Physics-Based Surrogate Modeling

• Concept: The authors developed a multi-oscillator dynamical model to simulate transformer dynamics. This model treats token representations as phase oscillators.
• Mechanism:
  • Self-Attention ($V_{SA}$): Modeled as intra-group interactions within a modality.
  • Cross-Attention ($V_{CA}$): Modeled as inter-group interactions between distinct modalities.
  • Network Topology: Based on Watts–Strogatz small-world networks to reflect the semantic structure of LLMs.
• Task: The model was tested on Lorenz chaotic time-series prediction.
  • Oscillators in Group $X$ were driven by the $x$-component of the Lorenz system.
  • Oscillators in Group $Y$ were driven by the $y$-component.
  • The goal was to predict the $z$-component.
• Metrics:
  • Normalized Mean Squared Error (NMSE): To measure prediction accuracy.
  • Dynamical SHAP: A metric defined to quantify the contribution of each modality ($X$ or $Y$) to the prediction, allowing the authors to measure modality dominance ($\phi(Y) - \phi(X)$).

3. Key Contributions

1. Phenomenological Framework: Introduces a novel physics-based phenomenological approach to explain AI bias. Unlike cognitivist views that treat AI as encoding external symbols, this approach focuses on the "lived experience" of the machine (internal physical dynamics during inference) to explain bias.
2. Surrogate Physics Model: Develops a mathematical model linking transformer attention mechanisms (self/cross-attention) to coupled oscillator dynamics, providing a tractable way to analyze complex multimodal interactions.
3. Discovery of Error Attractors: Demonstrates that model failures are not random but follow structured error-attractor patterns (hierarchical biases) that are only revealed under perturbation.
4. Modality Reinforcement Hypothesis: Provides evidence that adding a second modality often reinforces the dominance of the primary modality rather than mitigating bias, leading to "lock-in" effects where the secondary modality is ignored.

4. Key Results

Empirical Findings (Qwen2.5 & Gemma 3n)

• Hierarchical Bias: Both models exhibit a strong bias toward the "Neutral" emotion category. When "Neutral" is removed via perturbation, models fall back to a specific secondary hierarchy (e.g., "Happy" for Qwen2.5) rather than distributing errors randomly.
• Modality Dominance:
  • Qwen2.5: The error structure of the Video+Audio condition closely mirrors the Video-only condition. The audio input does not correct the video bias; it is effectively suppressed.
  • Gemma 3n: Shows even stronger asymmetry. The Voice-only input causes a massive collapse to "Neutral," but the presence of video in Video+Audio completely suppresses this audio bias, forcing the model to behave like the Video-only case.
• Conclusion: Multimodal inputs do not act as corrective signals; they often reinforce the dominance of the stronger modality.

Dynamical Modeling Findings

• Attention Levels Matter:
  • Low Attention ($\beta_{self}, \beta_{cross} \approx 10^{-4}$): The system is dominated by one modality (Group $X$), resulting in high prediction error (NMSE).
  • High Attention ($\beta_{self}, \beta_{cross} \approx 100$): The contributions of both modalities become equal ($\phi(X) \approx \phi(Y)$). The system achieves optimal accuracy, and the prediction successfully reproduces the chaotic attractor structure.
• Implication: Systematic bias in MLLMs arises when self- and cross-attention mechanisms are insufficiently tuned to balance the interaction between modalities.

5. Significance and Implications

• Beyond Aggregate Metrics: The paper argues that standard fairness metrics (which compare group performance) miss non-comparative unfairness (arbitrariness and inscrutability in individual decision dynamics). The proposed graph-based and dynamical analysis reveals these hidden failure modes.
• Algorithmic Fairness: By identifying that bias is an emergent property of interaction dynamics rather than just data representation, the paper suggests that fixing MLLM bias requires architectural adjustments to attention mechanisms (specifically balancing self- and cross-attention strengths) rather than just data re-balancing.
• New Paradigm for Explainability: The shift from "symbolic/cognitivist" explanations to "physics-based phenomenological" explanations offers a more rigorous framework for understanding the internal "black box" of transformer dynamics, particularly regarding how machines process conflicting or redundant information.

In summary, the paper demonstrates that multimodal bias is a dynamical instability where one modality suppresses another due to imbalanced attention mechanisms. It provides a physics-based toolkit to diagnose and potentially correct these instabilities to achieve truly fair and robust multimodal AI.