C$^2$MOE: Consistency and Complementarity-guided Mixture of Experts for Incomplete Multimodal Emotion Learning

Imagine you are trying to guess how a friend is feeling just by watching a video of them talking. You have three clues: what they are saying (text), how their voice sounds (audio), and what their face looks like (video). In the world of artificial intelligence, this is called Multimodal Emotion Recognition. It's like having a super-smart detective that combines all these clues to understand human emotions better than any single clue could alone. But here's the catch: in the real world, things go wrong. Maybe the microphone cuts out, the camera freezes, or the internet connection drops. Suddenly, your detective is missing half the clues. If the AI isn't built to handle these missing pieces, it gets confused and makes bad guesses. This is the problem scientists are trying to solve: how do we teach computers to "fill in the blanks" of missing information without losing the true meaning of the emotion?

This paper introduces a new solution called C2MOE (Consistency and Complementarity-guided Mixture of Experts). Think of it as a team of specialized detectives working together to solve the case when some evidence is missing. The researchers realized that previous methods tried to fix missing data by just copying what they knew from other clues, but this often led to boring, repetitive guesses that missed unique details. C2MOE does something smarter. It splits the job into two distinct roles. One expert, the "Consistency Detective," looks for the shared patterns between the clues—like how a sad voice usually matches a frowning face. The other, the "Complementarity Detective," looks for the unique details that only one clue can provide—like a specific word choice that only the text can reveal. By using a smart "manager" (a routing network) to decide which detective to listen to at any given moment, the system can reconstruct missing information much more accurately. The authors tested this on two famous datasets of video comments (CMU-MOSI and CMU-MOSEI) and found that C2MOE consistently outperformed existing methods, especially when large chunks of data were missing. They suggest that by balancing the need for shared patterns with the need for unique details, the AI can stay robust even when the data is incomplete.

The Detective Team: How C2MOE Works

To understand how C2MOE works, let's imagine you are trying to rebuild a broken puzzle, but you only have half the pieces. In the past, AI models tried to guess the missing pieces by just looking at the ones they had and hoping for the best. Sometimes they got lucky, but often they just guessed the same thing over and over, missing the unique details that made the picture special.

The authors of this paper argue that there are two different kinds of information we need to recover:

1. Consistency: This is the "shared language" between the clues. If someone is shouting, their voice is loud, their face is red, and their words are aggressive. These three things are consistent with each other.
2. Complementarity: This is the "unique flavor" of each clue. The text might tell you what they are shouting about, while the audio tells you how angry they are, and the video shows who they are yelling at. These are different pieces of the puzzle that don't overlap but are both necessary.

The problem is that these two goals often fight each other. If you focus too much on making everything consistent, you might erase the unique details. If you focus too much on the unique details, you might lose the connection between the clues.

Enter the Mixture of Experts (MoE).
Instead of one giant brain trying to do everything, C2MOE uses a "Mixture of Experts" approach. Imagine a detective agency with two specialists:

• The Consistency Expert: This detective is great at spotting patterns. If the text says "I'm happy," this expert knows the voice should sound cheerful and the face should be smiling. Their job is to make sure the missing pieces fit perfectly with the ones we already have.
• The Complementarity Expert: This detective is great at spotting the weird, unique stuff. If the text says "I'm happy," but the voice sounds sarcastic, this expert notices that the voice has a unique "flavor" that the text doesn't capture. Their job is to make sure we don't lose those unique details when we fill in the blanks.

The Smart Manager
How does the AI know which detective to listen to? It uses a routing network, which acts like a smart manager. When the AI sees a piece of data, the manager asks: "Do we need to match the pattern, or do we need to find a unique detail?" Based on the answer, it assigns a score to each expert. If the missing piece needs to match the others, the Consistency Expert gets a high score. If it needs to add something new, the Complementarity Expert gets the nod. The final answer is a mix of both, weighted by how much the manager trusts each one.

The Experiment: Putting C2MOE to the Test

The researchers tested their new system on two large collections of video comments from the internet (CMU-MOSI and CMU-MOSEI). These datasets contain thousands of videos where people talk about various topics, and the AI has to guess if they are happy, sad, angry, etc.

To make the test realistic, they simulated "missing data" by randomly hiding parts of the videos. Sometimes they hid the audio, sometimes the video, and sometimes the text. They even tested scenarios where up to 70% of the data was missing!

The Results
The results were impressive. In almost every scenario, C2MOE beat the other top methods.

• On the CMU-MOSI dataset, when all three clues (text, audio, video) were available, C2MOE achieved an accuracy of 88.5% (binary accuracy) and 47.4% (7-class accuracy).
• When they hid the video and audio, leaving only text, C2MOE still managed to get 87.3% accuracy, which was better than the next best method.
• Even when the missing rate was high (up to 0.7, meaning 70% of the data was gone), C2MOE held its ground much better than the others. While other models saw their performance drop sharply as more data disappeared, C2MOE's performance only dipped slightly.

The authors also looked at how well the AI filled in the missing pieces. They used a visual tool called t-SNE to see how the "reconstructed" data looked compared to the real data. They found that C2MOE's guesses were much closer to the real thing, keeping the shape and structure of the data intact, whereas other methods produced messy, scattered guesses.

Why This Matters

The main takeaway is that trying to force everything to be the same (consistency) isn't enough. You also need to respect the unique differences between the clues (complementarity). By using a team of specialized experts and a smart manager to switch between them, C2MOE can handle the messy, incomplete reality of the real world much better than previous models.

The paper suggests that this approach is a significant step forward for making AI that can understand human emotions even when the data is imperfect. While the authors don't claim to have solved the problem of missing data forever, their experiments strongly suggest that this "Consistency and Complementarity" strategy is a powerful way to make AI more robust and reliable. As they put it, the best results came when they balanced both the shared patterns and the unique details, proving that a little bit of teamwork between different types of experts goes a long way.