🤖 AI

Omni-Perception Policy Optimization for Multimodal Emotion Reasoning

The paper introduces OPPO, a reinforcement learning framework that enhances multimodal emotion reasoning by optimizing for reliable omni-modal perception through a specialized reward system and a loss function designed to reduce cross-modal hallucinations, validated by the new MEP-Bench diagnostic benchmark.

Original authors: Zhiyuan Han, Beier Zhu, Wenwen Tong, Pengyang Shao, Peipei Song, Xinyi Wang, Jiangnan Chen, Lewei Lu, Xun Yang

Published 2026-06-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhiyuan Han, Beier Zhu, Wenwen Tong, Pengyang Shao, Peipei Song, Xinyi Wang, Jiangnan Chen, Lewei Lu, Xun Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Honest Detective" vs. The "Daydreamer"

Imagine you are hiring a detective to solve a mystery about how a person is feeling. This detective has access to three types of clues: what they see (visual), what they hear (audio), and what is said (text).

The paper argues that current "AI detectives" (called Omni-MLLMs) are actually quite bad at their jobs. They suffer from two main problems:

The "Lazy Detective" (Underutilization): The detective ignores most of the clues. If a person is crying in the video but has a neutral face, the lazy detective might just say, "They are sad because they are crying," completely ignoring the fact that their face looks calm. They only look at the loudest clue and ignore the rest.
The "Daydreaming Detective" (Unfaithfulness/Hallucination): The detective makes things up. If the audio sounds angry, the detective might look at a video of a person with a neutral face and confidently say, "I see a frown on their face!" even though the video is actually blank or the person is smiling. They are "hallucinating" visual evidence based on what they heard, rather than what they actually saw.

The Solution: OPPO (The Training Camp)

The authors created a new training method called OPPO (Omni-Perception Policy Optimization) to turn these daydreaming, lazy detectives into honest, thorough ones. They did this using a "Reinforcement Learning" framework, which is like a strict coach giving rewards and penalties during practice.

OPPO uses two main tools to fix the AI:

1. The "Evidence Checklist" (Omni-Perception Reward)

The Analogy: Imagine a teacher grading a student's essay. Instead of just giving a grade for the final answer, the teacher has a specific checklist of facts that must be mentioned.

How it works: The AI is given a video and audio clip. The "ground truth" (the correct answer) has a list of specific clues, like "furrowed brow," "shaky voice," or "tears."
The Reward: The AI gets a bonus point for every single clue it mentions in its reasoning. If it ignores the "shaky voice" and only talks about the "tears," it gets a lower score. This forces the AI to stop being lazy and actually look at all the evidence.

2. The "Blindfold Test" (Omni-Perception Loss)

The Analogy: Imagine you are testing if a detective can really see. You put a blindfold over their eyes (mask the video) and ask, "What do you see on the person's face?"

The Problem: If the detective says, "I see a frown!" while blindfolded, they are lying. They are guessing based on the sound they heard, not what they saw.
The Fix: OPPO creates a special penalty. It takes the AI, hides the video (or the audio), and asks it to describe that specific part.
- If the AI changes its answer when the video is hidden (e.g., it stops saying "frown" because it can't see the face anymore), it gets a reward.
- If the AI keeps saying "frown" even though the video is gone, it gets a heavy penalty.
The Goal: This teaches the AI to be faithful. It learns that if it can't see the evidence, it shouldn't claim it exists. It stops "daydreaming" visual details based on audio cues.

The Results: A Better Detective

The authors built a special test called MEP-Bench to measure these two skills: "Utilization" (did you use all the clues?) and "Faithfulness" (did you make things up?).

Before OPPO: The AI was like a student who only studied the first page of the textbook and guessed the rest. It missed about half the clues and made up facts about 35–50% of the time.
After OPPO: The AI became a top-tier student.
- It started catching 70% of the clues (up from 50%).
- It stopped making up visual facts when the video was hidden, improving its honesty score significantly.
- It also got better at the actual job of identifying emotions, beating all previous records on standard tests.

Summary

The paper claims that for an AI to truly understand human emotions from video and sound, it must be trained to look at everything (not just the loud parts) and only say what it can actually perceive (not what it imagines). OPPO is the training method that forces the AI to do exactly that, resulting in a model that is both smarter and more honest.

Technical Summary: Omni-Perception Policy Optimization for Multimodal Emotion Reasoning

1. Problem Statement

Current Omni-modal Large Language Models (Omni-MLLMs) designed for emotion recognition, such as GPT-4o and Qwen-Omni, have driven a paradigm shift from simple label prediction to Multimodal Emotion Reasoning (MER). MER requires models to generate explicit reasoning chains grounded in visual, acoustic, and textual cues. However, this paper identifies a critical gap: existing emotion-oriented Omni-MLLMs lack reliable omni-modal perception.

The authors argue that reliable perception hinges on two principles which current models violate:

Utilization: Models often underutilize multimodal cues, relying on "thinking inertia" where they ignore fine-grained visual or acoustic evidence in favor of a single dominant modality (e.g., predicting anger solely based on voice volume while ignoring a neutral facial expression).
Faithfulness: Models exhibit unfaithful behavior, often hallucinating modality-specific statements. For instance, a model might claim a video shows a "frown" based solely on an angry audio tone, even when the video input is masked or the face is neutral. This stems from spurious cross-modal correlations rather than causal attribution to the specific modality.

To quantify these issues, the authors introduce MEP-Bench (Multimodal Emotion Perception Benchmark), which measures Utilization (via cue recall) and Faithfulness (via unimodal masking probes). Experiments on MEP-Bench reveal that baseline models capture only ~50% of human-annotated cues and fail faithfulness probes with accuracy rates as low as 26–35%.

2. Methodology: OPPO

The authors propose OPPO (Omni-Perception Policy Optimization), a Reinforcement Learning (RL) framework designed to explicitly optimize multimodal perception. The pipeline follows a standard Supervised Fine-Tuning (SFT) stage on MER-Caption+ data, followed by a GRPO-style RL phase augmented with two novel components:

A. Omni-Perception Reward ( $R_{omni}$ )

To address Utilization, this reward encourages the model to recover ground-truth evidence in its reasoning trajectory.

Mechanism: Ground-truth reasoning paragraphs are decomposed into atomic evidence cues ( $C_V$ for visual, $C_A$ for acoustic, $C_E$ for emotion). The model's generated reasoning is segmented into clauses.
Computation: An Evidence-Routing Matrix computes the semantic similarity (cosine similarity) between ground-truth cues and generated clauses. A "hit" is recorded if a cue's similarity to any clause exceeds a threshold $\delta$ .
Objective: The reward aggregates the fraction of covered cues across visual, audio, and emotion sets, encouraging the model to explicitly reference diverse multimodal evidence.

B. Omni-Perception Loss ( $J_{omni}$ )

To address Faithfulness, this loss penalizes cross-modal hallucination by contrasting the model's behavior under full input versus unimodal-masked inputs.

Mechanism: Counterfactual inputs are created by masking a portion ( $\rho$ ) of either the visual or acoustic stream.
Targeted Supervision: Unlike prior work (e.g., PAPO) that applies global KL penalties to all tokens, OPPO uses the Evidence-Routing Matrix to identify modality-specific evidence tokens (tokens describing visual or acoustic facts).
Objective: A KL divergence penalty is applied only to these specific tokens. The goal is to maximize the divergence between the token distribution under the full input and the masked input. If a model hallucinates visual evidence from audio, masking the video will not change the distribution of the "visual claim" token, resulting in a low KL penalty. Conversely, a faithful model will retract or change the claim when the source modality is removed, yielding a high KL penalty.

The total objective combines the task reward (accuracy), format reward, Omni-Perception Reward, and Omni-Perception Loss.

3. Key Contributions

Diagnostic Benchmark (MEP-Bench): The construction of a benchmark that quantifies the "Utilization" and "Faithfulness" of emotion reasoning, revealing that state-of-the-art models suffer from significant underutilization of cues and cross-modal hallucinations.
OPPO Framework: The proposal of an RL framework that explicitly optimizes perception through:
- A fine-grained Omni-Perception Reward for exhaustive cue coverage.
- A targeted Omni-Perception Loss that applies KL penalties exclusively to modality-specific evidence tokens to suppress hallucinations.
Theoretical Insight: The paper provides an information-theoretic analysis showing that the Omni-Perception Loss acts as a regularizer that expands the "information channel" (mutual information between tokens and modalities), thereby raising the upper bound for the Omni-Perception Reward to be effective.

4. Experimental Results

The authors evaluate OPPO on MER-UniBench (covering sentiment analysis, basic emotion recognition, and fine-grained recognition) and MME-Emotion.

State-of-the-Art Performance: OPPO achieves a mean score of 81.05% on MER-UniBench, outperforming the reproduced AffectGPT-R1 baseline by 5.45% and other strong baselines (e.g., Qwen-Audio, Emotion-LLaMA). It also achieves the best performance on MME-Emotion, particularly in reasoning and Chain-of-Thought (CoT) metrics.
Perception Diagnostics: On MEP-Bench, OPPO significantly improves both metrics:
- Utilization: Cue recall increases from 58.00% (Baseline) to 70.44%.
- Faithfulness: Accuracy on masked probes improves from 59.20% to 76.40% (video) and from 33.00% to 50.60% (audio).
Ablation Studies:
- Adding the Omni-Perception Reward alone improves performance, but adding the Omni-Perception Loss yields the most significant gains in faithfulness and overall task performance.
- Generic perception losses (like PAPO) that apply global penalties fail to improve performance and can degrade faithfulness, highlighting the necessity of OPPO's token-level, modality-specific targeting.
- Hyperparameter analysis confirms that a sufficient weight ( $\beta \ge 8e^{-3}$ ) for the loss is required to stabilize the optimization and unlock the reward's potential.

5. Significance and Claims

The paper claims that sufficient and faithful omni-perception is the key to unlocking robust reasoning in multimodal emotion tasks. The authors argue that previous methods relying on coarse-grained rewards or global penalties fail because they do not enforce the causal link between specific sensory inputs and the model's reasoning statements.

By enforcing that reasoning statements about a modality must depend on that modality's input (Faithfulness) and that the model must actively retrieve diverse cues (Utilization), OPPO demonstrates that reliable perception is not merely a byproduct of better language modeling but a distinct capability that must be explicitly optimized. The results suggest that future advancements in affective computing and human-AI interaction depend on ensuring models do not hallucinate evidence but rather ground their emotional interpretations in verifiable, multimodal signals.