💻 computer science

OmniEncoder: See, Hear, and Feel Continuous Motion Like Humans With One Encoder

The paper introduces Omni-Encoder, a unified Transformer backbone that co-embeds visual and audio signals at a symmetrical 25 fps to overcome the limitations of modality-specific encoders, thereby enabling holistic, human-like perception of continuous motion and significantly improving performance on fine-grained visual tasks while maintaining competitive audio-visual benchmarks.

Original authors: Detao Bai, Shimin Yao, Weixuan Chen, Chengen Lai, Yuanming Li, Zhiheng Ma, Xihan Wei

Published 2026-05-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Detao Bai, Shimin Yao, Weixuan Chen, Chengen Lai, Yuanming Li, Zhiheng Ma, Xihan Wei

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to understand a movie by looking at a single, frozen photo every second while listening to the soundtrack at full speed. That's essentially how most current "omni-modal" AI models (systems that see and hear) work today. They look at video frames slowly (1 or 2 per second) but listen to audio very quickly (25 times per second). This creates a mismatch: the AI sees the world in slow motion but hears it in real-time, making it hard to connect a specific hand gesture with the exact sound it makes.

The paper introduces Omni-Encoder, a new AI brain designed to fix this by seeing, hearing, and "feeling" motion all at once, just like humans do.

Here is how it works, using simple analogies:

1. The Problem: The "Slow-Motion Camera" vs. The "Fast Ear"

Current AI models are like a security guard who checks a hallway every 10 seconds but listens to every footstep instantly. If someone waves their hand quickly between checks, the guard misses it.

The Issue: Existing models sample video too slowly to catch fast movements (like sign language or gymnastics) while processing audio at full speed. This causes them to miss the "dance" between what you see and what you hear.

2. The Solution: The "25-FPS Super-Brain"

Omni-Encoder is built to process video and audio at the same high speed: 25 frames per second. It doesn't just look at pictures; it understands the movement between them.

To make this possible without the computer crashing from too much data, the authors invented three clever tricks:

A. The "Three-Lane Highway" (Omni-Encoder Token Template)

Instead of treating video and audio as one messy pile of data, the system organizes the information into three distinct "lanes" for every single moment in time:

The Audio Lane: Carries the sound.
The "Motion" Lane (Visual Continuous): This is the magic ingredient. These are special data points that act like "motion sensors." They don't care about what the object looks like (its color or shape); they only care about how it moves from one frame to the next. Think of them as tracking the path of a dancer's hand rather than the dancer's face.
The "Static" Lane (Visual Base): This carries the details of what things look like (textures, shapes, background).

The Efficiency Trick: The system keeps the "Motion" and "Audio" lanes running at full speed (25 fps) so no movement is missed. However, it slows down the "Static" lane to just 2 frames per second. Why? Because the background doesn't change that fast. This saves massive amounts of computer power while keeping the fast motion data intact.

B. The "3D GPS" (Omni-RoPE)

In a normal AI, it's hard to tell if a piece of data is a sound, a moving hand, or a static wall because they all look like the same numbers.
Omni-Encoder gives every single piece of data a unique 3D GPS coordinate (Time, Height, Width).

Sounds get a coordinate at the "origin" (0,0).
Moving hands get a coordinate just next to them (0,1).
Static walls get coordinates further out (1,1).
This allows the AI to instantly know, "Oh, this data point is a sound, and that one is a moving object," so it can understand how they relate to each other without getting confused.

C. The "Sliding Window" (Temporal Window Shifting)

Processing 25 frames of video and audio all at once is like trying to read every book in a library simultaneously—it's too heavy.
The authors use a technique called Temporal Window Shifting. Imagine reading a book by looking at a small window that covers 16 pages at a time.

First, you read pages 1–16.
Then, you shift the window halfway and read pages 9–24.
Then, you shift again.
This allows the AI to see how the story flows from one moment to the next without having to memorize the entire movie at once. It keeps the thinking process fast and efficient.

3. The Results: What Did They Prove?

The team tested this new system on tasks that require catching fast, detailed movements:

Sign Language: Recognizing signs is like reading a book written in fast hand movements. Old models got about 1% to 37% right. Omni-Encoder got 90% to 97% right, beating even specialized systems built just for sign language.
Sports & Action: In tasks like recognizing diving moves or gymnastics, it matched or beat the best existing models.
Audio-Visual Reasoning: When asked questions that require both hearing and seeing (e.g., "Who is speaking?"), it performed just as well as complex systems that use separate "ears" and "eyes."

The Bottom Line

The paper claims that by unifying how AI sees and hears—treating them as a single, synchronized stream of data rather than two separate, mismatched streams—we can build models that understand the world more like humans do. They can catch the "feeling" of continuous motion, making them much better at understanding fast actions, gestures, and the relationship between sound and sight.

Technical Summary: OmniEncoder

1. Problem Statement

Current omni-modal large language models (OmniLLMs) typically employ a modality-specific modular design, utilizing separate encoders for vision and audio. This architecture often results in a "video-coarse, audio-dense" sampling strategy, where visual frames are sampled at low frequencies (1–2 fps) while audio is processed at high frequencies (25 fps).

The authors argue that this design creates a fundamental discrepancy with human perception. Humans perceive the world holistically, seeing, hearing, and feeling continuous motion simultaneously, with cross-modal interactions occurring at the earliest stages of sensory processing. The prevailing "frame-by-frame, modality-by-modality" approach leads to:

Impoverished cross-modal interaction during the encoding phase.
Inability to capture fine-grained visual motion, which is critical for tasks like sign language recognition, fine-grained sports analysis, and lip reading.
Computational inefficiency when attempting to increase visual frame rates to match audio density using global attention mechanisms, which scale quadratically ( $O(T^2)$ ) with sequence length.

2. Methodology: Omni-Encoder

To address these challenges, the authors propose Omni-Encoder, a unified Transformer backbone designed to co-embed visual and audio signals at a symmetrical 25 fps within a shared latent space. The architecture processes raw video and audio simultaneously rather than fusing them at a later stage.

Core Architectural Components

A. Omni-Encoder Token Template

The input video is decomposed into three distinct token streams at each temporal position ( $t$ ):

Audio Tokens ( $a_t$ ): Extracted from log-Mel spectrograms via lightweight 1D CNNs.
Visual Continuous (VC) Tokens ( $vc_t$ ): Frame-wise learnable vectors introduced to capture inter-frame motion dynamics (e.g., motion trajectories, gesture onsets, micro-movements).
Visual Base (VB) Tokens ( $vb_t$ ): Generated via patch embedding to capture static appearance features (texture, shape, layout).

Token Sparsification: To maintain computational feasibility while preserving motion fidelity, a Token Sparsifier is applied. It retains all Audio and VC tokens at the full 25 fps resolution but downsamples VB tokens to 2 fps. This drastically reduces the total token count forwarded to the downstream LLM decoder without sacrificing the high-frequency motion information captured by the VC tokens.

B. Omni-RoPE (3D Rotary Positional Encoding)

Standard 3D Rotary Positional Embedding (RoPE) assigns coordinates $(t, h, w)$ . However, applying this directly to heterogeneous modalities creates a conflict where Audio and VC tokens lack independent spatial coordinates.

Solution: The authors remap the spatial coordinates. VB tokens are shifted to start at $(t, 1, 1)$ , while Audio tokens occupy $(t, 0, 0)$ and VC tokens occupy $(t, 0, 1)$ .
Benefit: This ensures all three modalities receive non-overlapping positional encodings within the same 3D RoPE formulation, allowing the model to distinguish between modalities and model cross-modal relationships while preserving resolution-agnostic extrapolation.

C. Temporal Window Shifting

To handle the computational cost of 25 fps video, the authors replace global spatiotemporal attention with a Temporal Window Shifting mechanism inspired by the Swin Transformer.

Mechanism: The token sequence is partitioned into non-overlapping temporal groups ( $G=16$ frames). Joint spatiotemporal attention is computed within these groups.
Shifting: In alternating layers, the window boundaries are shifted by $G/2$ frames, allowing information to flow across adjacent groups.
Complexity: This reduces the attention complexity from quadratic $O(T^2)$ to linear $O(T \cdot G)$ , enabling dense cross-modal interaction at 25 fps with manageable computational overhead.

Training Strategy

The model employs a single-stage, end-to-end training strategy:

Base: The LLM component of Qwen2.5-Omni-3B is used as a fixed decoder.
Initialization: The Omni-Encoder is initialized with weights from self-supervised video representation learning (V-JEPA). New components (audio embeddings, VC learnable queries) are initialized randomly.
Optimization: Only the final language modeling (LM) head is trainable; all other LLM parameters are frozen. This avoids complex multi-stage pre-training pipelines.

3. Key Contributions

Unified High-Frequency Encoding: The first unified encoder to process visual and audio signals simultaneously at 25 fps, bridging the gap between human-like continuous perception and current discrete sampling methods.
Novel Architectural Innovations: The introduction of the Token Template (separating motion and appearance), Omni-RoPE (unified spatio-temporal-modality coordinates), and Temporal Window Shifting (efficient dense attention).
Performance on Dense Motion Tasks: Demonstrating that a unified encoder can outperform specialized models on tasks requiring fine-grained temporal modeling, specifically sign language and sports action recognition.

4. Experimental Results

The authors evaluated Omni-Encoder against modality-specific baselines and existing OmniLLMs (e.g., Qwen2.5-Omni, Gemini-2.5-Pro) across several benchmarks:

Visual Continuous Understanding:
- Sign Language: On NationalCSL6707, Omni-Encoder achieved 90.32%, surpassing the previous best domain-specific model (69.61%) by over 20 percentage points. On SLR500, it reached 97.8%, outperforming the specialist SignBERT.
- Action Recognition: On Diving48, it achieved 90.8% (vs. 90.3% SOTA), and 68.7% on SthSv2, competing with strong baselines like InternVideo2-S2-1B.
- Observation: Existing OmniLLMs (even with SFT) performed near-randomly or poorly on these tasks (e.g., Qwen2.5-Omni SFT scored 25.7% on Diving48), highlighting the limitation of sparse visual encoding.
Audio-Visual Reasoning:
- Speaker Localization/Identification: Omni-Encoder achieved 98.8% localization and 82.31% identification, matching or surpassing dedicated models like HumanOmni-Speaker.
- AVQA: Achieved 82.6%, comparable to the SFT variant of Qwen2.5-Omni-3B.
Audio-Visual Recognition (LRS2):
- Without lip-cropping preprocessing, the model achieved a Word Error Rate (WER) of 45.3% (Visual-only), 10.2% (Audio-only), and 7.2% (Audio-Visual), demonstrating effective cross-modal fusion.

5. Significance and Claims

The paper posits that unified omnivorous encoding is a promising direction for building omni-modal models that more closely reflect the integrated nature of human perception.

Paradigm Shift: The results suggest that moving away from modality-specific, mismatched sampling rates toward a unified, high-frequency encoding space allows models to capture fine-grained motion dynamics that were previously inaccessible to general-purpose OmniLLMs.
Efficiency vs. Fidelity: The architecture successfully reconciles the dual challenges of modality disentanglement (distinguishing audio, static visual, and motion signals) and computational efficiency (linear scaling with video length).
Generalization: The model achieves specialist-level performance on dense motion tasks without task-specific architectural customization, validating high-density temporal modeling as an effective general-purpose representation strategy.

The authors conclude that while the model currently lags slightly behind proprietary state-of-the-art models in specific recognition tasks (attributed to limited training data scale), the unified approach offers a robust foundation for future omni-modal systems that perceive the world holistically.