🤖 machine learning

CONFER: Conflict-Aware Evidence Negotiation for Regime-Calibrated Weak Supervision in Multimodal Emotion Recognition

The paper proposes CONFER, a graph-based framework that negotiates cross-modal conflicts and calibrates weak supervision by dynamically estimating modality reliability and categorizing samples into consensus, dissent, and ambiguity regimes to achieve robust multimodal emotion recognition.

Original authors: Bojing Hou, Ruohao Li, Yitong Zhu, Luwen Yu, Yuyang Wang

Published 2026-08-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bojing Hou, Ruohao Li, Yitong Zhu, Luwen Yu, Yuyang Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out how a friend is feeling. You might look at their face, listen to their voice, and watch their body language. Usually, all these clues agree: a frown, a shaky voice, and slumped shoulders all scream "sadness." But what if your friend is smiling (face says happy) while their voice cracks (voice says sad) and they are trembling (body says scared)? This is the messy reality of Multimodal Emotion Recognition (MER), a field of computer science where machines try to understand human feelings by combining different types of data like video, audio, and brain signals.

The tricky part is that computers often treat the "answer key"—the label a human gives to say how they feel—as perfect truth. But humans are messy! We might lie, forget, or just have a bad day when we rate our emotions, making our "answers" unreliable. Furthermore, when the computer's different sensors disagree (like the face vs. the voice), most existing programs just smash the data together and hope for the best, ignoring the conflict. This paper tackles that confusion, asking: How can a computer know which clue to trust when they are fighting each other, and how can it realize that the "answer key" might be wrong?

Enter CONFER, a clever new system that acts like a skilled mediator for a group of arguing experts. Instead of forcing the different sensors (the "experts") to agree immediately, CONFER lets them negotiate. Imagine a roundtable discussion where a Face Expert, a Voice Expert, and a Brainwave Expert are trying to decide if a person is happy or sad. In the old days, the computer would just take a vote. But CONFER is smarter. It knows that sometimes the Face Expert is having a bad day (maybe the lighting is poor) and sometimes the Voice Expert is more reliable.

CONFER uses a "dynamic graph" to let these experts talk to each other. They don't just shout their opinions; they share "evidence" and "uncertainty." If the Face Expert is very unsure (high uncertainty) but the Voice Expert is very confident, the Face Expert listens and adjusts their opinion. If they are both confident but disagree, CONFER flags this as a serious conflict. It doesn't just ignore the disagreement; it uses it to figure out which expert is actually telling the truth and which one might be hallucinating.

The system also has a special trick for handling the "answer key" (the self-reported labels). It realizes that if all the experts are arguing fiercely, the human's label might be unreliable too. So, CONFER sorts every moment into one of three "regimes":

Consensus: Everyone agrees, and the label is likely correct.
Dissent: The experts disagree, but the system can still figure out who is right.
Ambiguity: Everything is a mess, and the system decides to trust the human label very little because it's probably wrong.

The researchers tested CONFER on three major datasets involving real people showing emotions (AMIGOS, MAHNOB-HCI, and DEAP). They found that when the sensors were fighting (high conflict), CONFER was a lifesaver. For example, on the AMIGOS dataset, it achieved an accuracy of 0.873 under strict testing conditions where it had to guess emotions for people it had never seen before. On the MAHNOB dataset, it hit 0.854. These numbers are impressive, beating other top methods.

Perhaps the most exciting finding is how the system handles "corrupted" data. The researchers pretended to mess up the human labels (like flipping the answers randomly) to see if the computer would get confused. While other methods crashed, CONFER stayed stable, maintaining an accuracy of 0.723 even when 30% of the labels were wrong. This suggests that the system isn't just memorizing answers; it's actually learning to trust the right clues and ignore the noise.

In short, CONFER proves that conflict isn't just a bug to be fixed; it's a useful signal. By letting different sensors negotiate and by realizing that human labels aren't always perfect, the system becomes much better at understanding the complex, messy reality of human emotion. It shows that sometimes, the best way to find the truth is to listen to the arguments, not just the final vote.

Technical Summary: CONFER

Problem Statement

Multimodal Emotion Recognition (MER) traditionally relies on self-reported valence and arousal scores as ground-truth supervision. However, the paper argues that these labels are inherently unreliable "weak observations" due to individual rating habits, cognitive biases, memory distortions, and experimental context. Furthermore, existing methods often overlook cross-modal conflict—disagreements among predictions from different modalities (e.g., facial, audio, physiological).

Current approaches typically address modality heterogeneity through attention mechanisms or gating but forcibly unify conflicts via fusion weights without determining which modality should influence another for a specific sample. Similarly, weak-label methods estimate supervision reliability based on global statistics or confidence scores, failing to leverage pairwise agreement and conflict among modality-specific predictions. This disconnect prevents efficient orchestration of information passing and robust learning from noisy self-reports.

Methodology: CONFER Framework

The authors propose CONFER (Conflict-Aware Evidence Negotiation for Regime-Calibrated Weak Supervision), a unified graph-based framework that models modality experts as nodes in a dynamic negotiation graph. The framework operates in three main stages:

1. Modality Expert Construction

Each modality is treated as an independent expert.

Representation: Encoders map temporal windows to embeddings. A constructor aggregates these into trial-level representations using global mean pooling, sparse top- $k$ attention, and temporal dispersion.
Evidence Generation: Each expert produces a predictive belief ( $\mu$ ) and a boundary-based uncertainty score ( $u$ ), where uncertainty increases as the prediction approaches the decision boundary.
Runtime Reliability: Reliability ( $c$ ) is estimated by combining historical Out-of-Fold (OOF) performance with current sample uncertainty. This ensures reliability is sample-specific rather than static.

2. Conflict-Aware Evidence Negotiation

Experts interact via a dynamic graph to resolve conflicts iteratively.

Node State: Defined by predictive belief, uncertainty, and runtime reliability.
Compatibility & Edge Weights: Pairwise compatibility ( $A_{mn}$ ) is calculated based on prediction disagreement normalized by uncertainty. High disagreement between confident experts yields low compatibility.
Asymmetric Message Passing: Compatibility is converted into receiver-specific directional edge weights ( $\gamma_{m \leftarrow n}$ ). This allows reliable senders to influence less reliable receivers, while the receiver's own reliability dictates how much it concedes to incoming information.
Iteration: The process runs for two rounds ( $\mu^{(0)} \to \mu^{(1)} \to \mu^{(2)}$ ), updating beliefs based on peer-supported aggregation.

3. Regime-Calibrated Weak Supervision

The negotiation outcome characterizes the sample into one of three regimes to calibrate the loss function:

Consensus: High agreement and low residual conflict.
Dissent: Persistent disagreement after negotiation.
Ambiguity: High overall uncertainty.
Sample Weighting: The inferred regime probabilities determine the weight ( $\omega_i$ ) of each sample's contribution to the weak-label loss. Samples in the "Consensus" regime receive full weight, while "Dissent" and "Ambiguity" samples are down-weighted (but not discarded) to prevent the model from overfitting to unreliable supervision.

The framework is trained in three phases: (A) training encoders for initial beliefs, (B) OOF calibration to estimate reliability priors and regime coefficients, and (C) joint optimization of the negotiation module and prediction readout.

Key Contributions

Unified Framework: CONFER integrates modality reliability estimation, cross-modal negotiation, and weak-label calibration into a single graph-based architecture.
Dynamic Negotiation Mechanism: It introduces a method to convert uncertainty-aware compatibility into reliability-directed, asymmetric message passing, allowing cross-modal conflict to guide both expert coordination and supervision reliability estimation.
Regime-Based Calibration: By defining Consensus, Dissent, and Ambiguity regimes, the method dynamically adjusts the contribution of weak labels based on sample-specific conflict resolution, rather than treating all labels as equally reliable or unreliable.

Experimental Results

The authors evaluated CONFER on three benchmarks: AMIGOS, MAHNOB-HCI, and DEAP, using both subject-dependent 10-fold cross-validation and strict Leave-One-Subject-Out (LOSO) protocols.

Performance: CONFER achieved competitive results, notably reaching 0.873 accuracy and 0.879 F1 on AMIGOS-V (Valence) and 0.854 accuracy on MAHNOB-V under strict LOSO evaluation, outperforming baselines like Late Fusion, TFN, MISA, and various Expert Routing (MoE) methods.
Robustness: The model demonstrated superior robustness to synthetic weak-label corruption. Under 30% symmetric label corruption, CONFER maintained an accuracy of 0.723, significantly outperforming baselines.
Conflict Analysis: Ablation studies confirmed that removing negotiation, sender reliability, or receiver concession degraded performance. Analysis showed that negotiation yields the largest gains (5.4–8.9 points) on high-conflict samples, correcting wrong predictions far more often than it disrupts correct ones.
Supervision Detection: The inferred conflict regimes (Consensus, Dissent, Ambiguity) were shown to be effective indicators of supervision inconsistency, outperforming predictive entropy in detecting unreliable labels.

Significance and Claims

The paper claims that CONFER demonstrates that cross-modal conflict is not merely a source of error but a useful information signal. By explicitly modeling the negotiation between modalities, the framework can:

Coordinate Modalities: Selectively update expert predictions based on relative reliability and compatibility, rather than forcing a uniform fusion.
Estimate Supervision Reliability: Use the resolution (or lack thereof) of cross-modal conflict to calibrate the trustworthiness of self-reported labels on a per-sample basis.

The authors conclude that their approach provides a more robust mechanism for learning from weak supervision in multimodal settings, particularly when dealing with the inherent unreliability of self-reported emotional states and the heterogeneity of sensory data.