💻 computer science

CogniRoute: Learning to Route Social Evidence in Omni-Modal Models

The paper introduces CogniRoute, a schema-guided Mixture-of-Experts framework enhanced with route-aware reinforcement learning and trained on the new OmniSocialBench dataset, which significantly improves social video question answering by effectively routing and coordinating multi-modal evidence for complex reasoning tasks involving gestures, tone, and temporal cues.

Original authors: Yifan Shen, Pei Tian, Xinzhuo Li, Bowen Fang, Shujun Xia, Bingxuan Li, Ana Jojic, Wenming Ye, Xu Cao, James Matthew Rehg, Ismini Lourentzou

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yifan Shen, Pei Tian, Xinzhuo Li, Bowen Fang, Shujun Xia, Bingxuan Li, Ana Jojic, Wenming Ye, Xu Cao, James Matthew Rehg, Ismini Lourentzou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a mystery in a crowded room. You have a video camera, a microphone, and a transcript of what people are saying. To solve the mystery, you need to know what to look at, how to interpret it, and when it happened.

Most current AI models are like detectives who have all this equipment but don't know how to use it. They might look at the video when they should be listening to the tone of voice, or they might focus on the wrong moment in time. They can often guess the right answer by luck or by memorizing patterns, but they don't truly understand why the answer is correct.

This paper introduces CogniRoute, a new way to train AI to become a better detective. Here is how it works, broken down into simple concepts:

1. The Problem: The "Jack of All Trades, Master of None" AI

Current AI models can see, hear, and read at the same time. But when you ask them a tricky social question (like, "Is that person actually happy, or just pretending?"), they often get confused. They might ignore a subtle facial expression because they are too focused on the words being spoken, or they might miss a crucial 2-second pause in the conversation. They have access to all the clues, but they don't know which clue is the "smoking gun."

2. The Solution: A "Smart Traffic Controller" (CogniRoute)

The authors built a system called CogniRoute. Think of the AI model as a massive factory with hundreds of specialized workers (called "experts"). Some workers are great at analyzing faces, others at understanding tone, and others at tracking time.

In a normal factory, the manager (the router) just sends work to whoever is free. In CogniRoute, the manager is trained to be a smart traffic controller. Before sending a task to a worker, the controller asks three specific questions about the task:

Evidence Source: Do I need to look at the video, listen to the audio, or compare both?
Reasoning Demand: Do I need to just "see" what happened, or do I need to figure out someone's hidden feelings or social rules?
Time Scope: Do I need to look at a split-second moment, or do I need to watch the whole scene?

3. The Training: Learning with a "Cheat Sheet"

To teach this traffic controller, the researchers created a special "cheat sheet" called a Cognitive Schema.

The Analogy: Imagine a student taking a test. Usually, they only get a grade (Right/Wrong) at the end. With CogniRoute, the teacher gives them a cheat sheet during the test that says, "For this question, you must look at the audio and check the timing."
The Process: The AI is trained to match its internal "traffic control" decisions with this cheat sheet. It learns that if a question requires understanding a sarcastic tone, it must route the data to the "audio expert" and the "social expert," not just the "visual expert."

4. The Reinforcement: "Good Job, But Did You Do It Right?"

Even if the AI gets the right answer, it might have gotten there by guessing. To fix this, the researchers added a second training phase called Route-Aware Reinforcement Learning.

The Analogy: Imagine a coach watching a player score a goal. If the player scored by tripping the opponent, the coach says, "Good goal, but bad technique." If they scored by a perfect pass, the coach says, "Great goal, great technique!"
The Reward: The AI gets a "high score" only if:
1. It gets the answer right.
2. It used the right type of evidence (e.g., didn't ignore the audio).
3. It focused on the right moment in time.

5. The New Test: OmniSocialBench

To prove this works, the team built a new test called OmniSocialBench.

The Analogy: Most video tests are like a multiple-choice quiz where you just pick the right letter. OmniSocialBench is like a detective exam where you have to show your work. The test includes 118,000 examples where the "correct answer" is paired with a detailed map of which clues were used and when.
The Result: On this test, CogniRoute scored 59.38%, beating the best existing AI models by a huge margin (over 15% better than the top proprietary model). It was especially good at questions requiring it to mix audio and video or resolve conflicts (like when someone says "I'm fine" but looks sad).

Summary

CogniRoute is like teaching an AI to stop guessing and start thinking like a human detective. Instead of just looking for the right answer, it learns to:

Identify what kind of clue it needs (visual, audio, or both).
Understand how to process that clue (simple observation vs. complex social inference).
Pinpoint when that clue happened.

By forcing the AI to route its "brain power" to the right specialists based on these clues, it becomes much better at understanding the messy, complex world of human social interaction.

Technical Summary: CogniRoute

Problem Statement

Omni-modal models, which ingest video, audio, and text within a unified autoregressive system, have demonstrated strong capabilities in multimodal perception. However, a critical gap remains between having access to multiple modalities and effectively utilizing the right evidence to answer a specific question. This issue is particularly acute in social video question answering (QA), where the correct answer often hinges on subtle cues such as gestures, vocal tone, temporal ordering, or conflicts between spoken words and visual expressions.

Existing benchmarks typically supervise only the final answer, leaving the underlying evidence structure implicit. Consequently, models may produce correct answers by relying on the wrong modality, attending to irrelevant temporal segments, or exploiting dataset priors. Furthermore, standard Mixture-of-Experts (MoE) routing mechanisms, which are typically optimized via next-token prediction and load balancing, lack explicit sample-level supervision to distinguish between different types of multimodal reasoning (e.g., visual-only vs. audio-visual conflict resolution).

Methodology: CogniRoute

To address these challenges, the authors introduce CogniRoute, a schema-guided MoE training framework designed for social omni-modal reasoning. The framework aligns expert routing with the latent evidence structure of each input through two primary stages:

1. Schema-Aligned Predictive Routing (SAPR)

During supervised fine-tuning (SFT), CogniRoute introduces a training-only Cognitive Schema that factorizes each example into three dimensions:

Evidence-Source: The relation between audio and visual evidence (Visual, Audio, Joint, or Conflict).
Reasoning-Demand: The required reasoning operation (e.g., Direct Perception, Social Norm, Mental State, Sarcasm).
Temporal-Scope: The time window of relevant cues (Momentary, Local Window, Long Range, Multi-Segment).

The method computes a global routing signature by pooling the router's probability distributions over input tokens (video, audio, question) across all MoE layers. A lightweight prediction head maps this signature to a target embedding derived from the schema tags. The SAPR loss encourages the router to produce distinct signatures for samples with different evidence structures, effectively teaching the model to "know" which modality and reasoning type a question requires before generating the answer.

2. Route-Aware MoE Reinforcement Learning (RMRL)

Standard answer likelihood is insufficient for optimizing expert allocation, as correct answers can be generated via incorrect internal routes. CogniRoute introduces a second training stage using Route-Aware MoE Reinforcement Learning. This stage jointly optimizes token generation and expert routing using a composite reward signal:

Answer Correctness ( $\mathcal{R}_{ans}$ ): A binary reward for the final answer.
Modality-Consistent Reasoning ( $\mathcal{R}_{mcr}$ ): A reward (computed via a frozen LLM judge) that verifies whether the generated reasoning trace explicitly grounds the answer in the required modality evidence (visual, audio, or both) and correctly characterizes their relationship.
Cognitive Temporal Grounding ( $\mathcal{R}_{ctg}$ ): A reward that measures whether the model's final reasoning state focuses on the annotated temporal evidence span. This is calculated using the model's own causal attention weights over audio-visual tokens, converted into a temporal distribution and compared against the ground-truth evidence span via KL divergence.

The RL objective updates both the token generation policy and the router's gate selection using rollout-level advantages, ensuring that successful generations are those that are not only correct but also evidence-aware and temporally grounded.

Key Contributions

CogniRoute Framework: A schema-guided MoE framework that aligns expert routing with the evidence structure of social video questions, enabling the model to distinguish when to rely on specific modalities, resolve conflicts, or perform temporally grounded inference.
Route-Aware Reinforcement Learning: A novel RL objective that jointly optimizes answer generation and expert allocation using rewards for correctness, modality consistency, and temporal grounding, moving beyond simple answer supervision.
OmniSocialBench: A diagnostic social video QA resource containing 118K structured training examples and a manually verified evaluation split. Unlike standard benchmarks, it provides explicit supervision for modality evidence, audio-visual relations, reasoning demands, and temporal evidence spans.

Results

Evaluated on OmniSocialBench, CogniRoute achieves an average accuracy of 59.38%.

It outperforms the strongest proprietary baseline (Gemini 3.1 Pro) by 15.33 percentage points.
It outperforms the strongest open-source omni baseline (Qwen3 Omni) by 26.77 percentage points.
The most significant gains are observed in categories requiring complex coordination, such as Action Goal (+18.79% over the best baseline) and Social Norm inference (+24.80%).

Beyond OmniSocialBench, CogniRoute improves performance on 8 out of 10 public audio-visual and video reasoning benchmarks (including OmniBench, WorldSense, and Video-MME-v2), demonstrating that the proposed routing and reinforcement objectives generalize to broader multimodal tasks.

Significance and Claims

The paper claims that evidence-structured routing is an effective training signal for improving social omni-modal reasoning. By explicitly supervising the model's internal routing behavior according to the evidence structure (modality relation, reasoning demand, and temporal scope), CogniRoute forces the model to engage in more interpretable and appropriate computation.

The authors emphasize that their approach addresses the "black box" nature of multimodal reasoning, where models often arrive at correct answers via spurious correlations. CogniRoute demonstrates that aligning expert allocation with the cognitive schema of the task leads to substantial improvements in tasks requiring cross-modal coordination, conflict resolution, and temporally grounded social inference. The work positions social video QA as a strong testbed for omnimodal understanding, highlighting that access to multiple modalities is necessary but insufficient without mechanisms to select and integrate the correct evidence.