💬 NLP

Cross-Modal Coreference Alignment: Enabling Reliable Information Transfer in Omni-LLMs

This paper identifies cross-modal coreference as a critical bottleneck in Omni-LLMs, introduces the CrossOmni dataset to evaluate this capability, and demonstrates that both training-free and training-based strategies significantly improve robust omni-modal reasoning by inducing coreference-aware thinking patterns.

Original authors: Hongcheng Liu, Yuhao Wang, Zhe Chen, Pingjie Wang, Zhiyuan Zhu, Yixuan Hou, Yanfeng Wang, Yu Wang

Published 2026-04-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hongcheng Liu, Yuhao Wang, Zhe Chen, Pingjie Wang, Zhiyuan Zhu, Yixuan Hou, Yanfeng Wang, Yu Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are at a busy party where three friends are trying to tell you a story about a specific person, let's call him Chase.

Friend A (The Text) hands you a biography that says: "Chase was in a car accident in 2015."
Friend B (The Video) shows you a clip of a man sitting in a hospital bed, looking pale.
Friend C (The Audio) plays a recording of a voice saying, "Do you hurt anywhere else?"

The Problem:
Current "Super-Brain" computers (called Omni-LLMs) are great at listening to Friend C, watching Friend B, or reading Friend A individually. They are like party guests who can perfectly understand one conversation at a time.

However, when you ask them, "What does the voice of the person who was in the car accident sound like?", they get confused. They look at the text, see "Chase," look at the video, see a man, and listen to the audio, but they fail to connect the dots. They don't realize that the "man in the video" is the same "Chase" mentioned in the text, and that the voice in the audio belongs to that same man. They treat the three friends as strangers talking about different people, rather than a team describing one person.

This paper calls this failure a lack of "Cross-Modal Coreference Alignment." In simple terms: They can't point to the same thing across different senses.

The Solution: A New Training Manual (CROSSOMNI)

The researchers realized that to fix this, they needed a new kind of training manual. They created a dataset called CROSSOMNI.

Think of this dataset as a "Detective's Training Course." Instead of just giving the computer the raw party footage, they gave it a workbook with step-by-step reasoning guides (called rationales).

Old Way: "Here is a video and a question. Guess the answer." (The computer often guesses wrong because it doesn't know how to link the clues).
New Way (CROSSOMNI): "Here is the video. First, find the man in the bed. Second, look at the text to confirm his name is Chase. Third, listen to the audio and match it to Chase. Finally, describe his voice."

They built 9 different types of "detective puzzles" where the computer has to jump between text, video, and audio to solve a mystery.

The Fix: Teaching the Computer to "Think"

The researchers tested 13 different "Super-Brain" computers and found they were all bad at this linking game. They tried two ways to fix it:

1. The "Cheat Sheet" Method (In-Context Learning)

This is like giving the computer a sample solved puzzle right before the test.

Analogy: "Here is how we solved a similar case: We found the person in the text, then looked for them in the video. Now, you try this new case."
Result: The computer got much better immediately, just by seeing an example of how to think step-by-step.

2. The "Drill Sergeant" Method (SFT + GRPO)

This is a more permanent fix. They didn't just show examples; they retrained the computer's brain.

SFT (Supervised Fine-Tuning): They taught the computer the correct "detective steps" using the new workbook.
GRPO (Group Relative Policy Optimization): This is like a coach watching the computer practice. If the computer tries to jump to the answer without linking the clues, the coach says, "No, go back! You missed the link between the text and the video!" The computer learns to reward itself for doing the linking correctly.

The Big Discovery

The most important finding is that being smart at one thing doesn't make you smart at connecting things.

A computer can be a genius at reading text.
It can be a genius at watching videos.
But if it doesn't have a specific "Linking Habit" (a thinking pattern that says, "Wait, this person in the text is the same person in the video"), it will fail at complex tasks.

The Takeaway

This paper is like telling us: "Don't just build bigger brains; teach them how to hold hands."

To make AI truly understand our world (which is a mix of talking, seeing, and reading), we need to stop just feeding it more data and start teaching it how to connect the dots between different types of information. Once we teach the AI this "Linking Habit," it becomes much better at solving real-world problems, from diagnosing medical issues to understanding complex movies.

1. Problem Definition

Omni-LLMs (Large Language Models capable of processing text, audio, and video simultaneously) have shown strong capabilities in holistic multi-modal perception. However, they consistently fail in complex scenarios requiring synergistic omni-modal reasoning.

The core issue identified is a lack of fine-grained cross-modal alignment. While current models can aggregate global context, they struggle to perform cross-modal coreference: the ability to locate a specific referent (e.g., a person or object) in a source modality (e.g., text) and re-identify/track that same referent in a target modality (e.g., video or audio) to synthesize evidence.

Current Failure Mode: Models often treat modalities as separate streams or perform coarse aggregation, failing to establish explicit links between entities across modalities. This leads to referential inconsistency and reasoning errors.
Formalization: The authors formalize this as a Cross-Modal Coreference Problem, where the model must:
1. Identify a referent in a source modality ( $m_s$ ).
2. Re-identify and align that referent in a target modality ( $m_t$ ).
3. Extract the answer from the target modality based on the alignment.

2. Methodology

A. The CROSSOMNI Dataset

To address the lack of evaluation and training data for this specific capability, the authors introduce CROSSOMNI, a comprehensive dataset containing:

Scale: 4,147 videos and 39,726 question-answer (QA) pairs.
Modalities: Audio (A), Video (V), and Text (T).
Task Structure:
- 9 Task Types: 3 single-modality tasks (Au, Vi, Te) and 6 cross-modality tasks (e.g., Audio→Video, Text→Audio).
- Data Construction: Built upon TVQA clips, enhanced with detailed annotations for visual frames, audio characteristics (pitch, emotion), and text biographies.
- Key Feature: Every instance includes human-designed reasoning rationales that explicitly guide the model through a four-step alignment process:
  1. Identify source and target modalities.
  2. Locate the referent in the source.
  3. Find the aligned referent in the target.
  4. Extract the answer.
Quality Control: Rigorous human verification ensures 93%+ agreement on audio labels and 96%+ correctness on video descriptions.

B. Diagnostic Analysis

The authors evaluated 13 state-of-the-art Omni-LLMs on CROSSOMNI.

Finding: Cross-modal tasks are ~21% harder than single-modal tasks.
Root Cause: Standard Chain-of-Thought (CoT) prompting yields only marginal gains (1.7%). The analysis reveals that models lack "coreference-aware thinking patterns." They fail to explicitly bridge the gap between the entity in the source modality and its manifestation in the target modality.

C. Proposed Solutions

To induce these missing thinking patterns, two strategies are proposed:

Training-Free: In-Context Learning (ICL)
- A dedicated prompt structure is designed to guide the model step-by-step through the coreference alignment process (Identify → Locate → Align → Answer) using few-shot examples.
- Goal: To show the model how to think without updating weights.
Training-Based: SFT + GRPO Framework
- Supervised Fine-Tuning (SFT): Models are fine-tuned on the CROSSOMNI dataset using the structured reasoning rationales as ground truth.
- Group Relative Policy Optimization (GRPO): A reinforcement learning stage applied on top of SFT.
  - Reward Model: Uses a large LLM (Qwen3-Next-80B) to evaluate outputs.
  - Reward Signals: The objective includes rewards for Output Format, Reasoning Accuracy, Contextual Consistency, and Logical Coherence (specifically checking if the visual localization matches the text retrieval).
- Goal: To explicitly train the model to internalize the coreference-aware thinking patterns.

3. Key Contributions

Problem Formulation: Defined Cross-Modal Coreference Alignment as a critical, missing component for robust Omni-LLM reasoning, shifting focus from coarse aggregation to fine-grained entity linking.
CROSSOMNI Dataset: Created the first dataset specifically for evaluating and training cross-modal coreference, featuring 9 task types and human-annotated step-by-step reasoning rationales.
Thinking Pattern Induction: Demonstrated that the primary bottleneck is not data volume but the lack of specific reasoning patterns. Introduced both ICL and SFT+GRPO methods to induce these patterns.
Generalization: Showed that improving coreference alignment on CROSSOMNI generalizes to broader collaborative reasoning benchmarks (Daily-Omni, WorldSense), suggesting this is a foundational capability.

4. Experimental Results

Baseline Performance: Across 13 models, cross-modal accuracy was significantly lower than single-modal accuracy (e.g., average drop from ~64% to ~46%).
ICL Impact: Applying the ICL prompt reduced the performance gap between single and cross-modal tasks from 24.37% to 11.01%, yielding a 21% relative improvement on cross-modal tasks.
SFT + GRPO Impact:
- Qwen2.5-Omni-3B: GRPO improved overall accuracy by 28.43% (from 45.62% to 58.59%).
- Qwen2.5-Omni-7B: GRPO improved overall accuracy by 24.23% (from 50.36% to 62.56%).
- Ablation Studies: Removing CoT supervision during SFT or the thinking-pattern reward during GRPO caused significant performance drops, confirming that explicitly modeling the reasoning path is crucial.
Generalization: Models trained with GRPO showed consistent gains on out-of-distribution (OOD) benchmarks like WorldSense and Daily-Omni, whereas SFT alone sometimes degraded performance due to domain mismatch, highlighting GRPO's role in learning robust alignment rather than overfitting.

5. Significance

This paper identifies that the evolution of Omni-LLMs has stalled because models can perceive modalities individually but cannot reliably transfer information between them via entity alignment.

Paradigm Shift: It argues that future progress depends less on simply adding more modalities and more on fine-grained cross-modal coreference.
Practical Pathway: The proposed SFT+GRPO framework provides a concrete, effective method to instill "thinking patterns" that allow models to reason across modalities, moving them closer to human-like omni-modal understanding.
Resource: CROSSOMNI serves as a vital benchmark for the community to diagnose and improve this specific capability.