🤖 AI

Aero Realtime: Fully Aligned Input-Output Streams for Low-Latency Streaming Multimodal Generation

The paper introduces Aero Realtime, a 4B streaming multimodal model that achieves fully aligned, duplex input-output interaction on a shared temporal grid, enabling low-latency, proactive generation with efficient KV-cache reuse and sub-200ms processing lag.

Original authors: Kaichen Zhang, Wei Huang, Keming Wu, Bo Li, Xiaojuan Qi

Published 2026-08-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kaichen Zhang, Wei Huang, Keming Wu, Bo Li, Xiaojuan Qi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to have a conversation with a friend who is also watching a movie with you. In a normal chat, you wait for them to finish a sentence before you speak. But in a truly natural conversation, you might interrupt them with a quick "Wow!" while they are still talking, or you might stay silent while they explain a complex plot point, only to jump in the exact second you understand something new. This is the holy grail of "real-time" interaction: the ability to listen and speak at the same time without missing a beat.

For a long time, computers have been terrible at this. Most AI models work like a very polite, but slightly slow, game of "hot potato." They wait for you to finish your entire story (the input), process it all at once, and then start talking (the output). They can't really "listen" while they are "speaking." If you try to feed them new information while they are mid-sentence, they get confused or have to stop and restart. This paper tackles the problem of building an AI that can truly be "duplex"—able to hear new things and speak new things simultaneously, just like a human does, all while keeping up with a fast-moving video stream.

The researchers behind this paper, from the University of Hong Kong and other institutions, have built a new AI model called Aero Realtime. Think of it as a model that doesn't just talk in sentences, but in tiny, rhythmic "beats" of time. Instead of waiting for a whole video to finish, Aero Realtime breaks time down into tiny 80-millisecond chunks. For every single chunk of audio it hears, it has to make a split-second decision: "Should I say a word right now, or should I stay silent?"

This is a huge shift from how other models work. Usually, an AI has to finish its "thinking" phase before it starts its "talking" phase. Aero Realtime does both at once. It aligns the video, the audio, and its own words onto a single, shared timeline. Imagine a conveyor belt where every 80 milliseconds, a new piece of video and audio arrives. At that exact same moment, the AI decides whether to drop a word onto the belt or drop a "silence token" (a placeholder for being quiet). This allows the AI to keep listening to new parts of the video even while it is in the middle of generating a sentence. It never stops the flow to "catch up."

The paper argues that previous attempts to make AI "proactive" (ready to speak) were clunky. Some systems used a "micro-turn" method, where the AI would constantly pause to ask itself, "Should I speak now?" This is like a driver who stops at every single intersection to check a map before moving forward. Others used external "gates" or switches to decide when to talk, which complicated the system and made it slow. Aero Realtime rejects these methods. Instead, it learns the timing naturally. The model is trained to treat "silence" as a valid word, just like "hello" or "cat." This means the AI doesn't need a separate switch to decide when to talk; the decision to speak or stay quiet is baked directly into the same process that chooses the words.

To make this work, the team had to solve some tricky engineering puzzles. They created a special way to train the model using a mix of real-time video data and standard video questions. They also designed a new way to run the model on computers that saves memory and speed. Instead of re-reading the whole video every time a new frame arrives, the model remembers what it already processed and only adds the new "slice" of information. This is like reading a book where you don't have to re-read the first chapter every time you turn a page; you just remember where you left off and keep going.

The results are impressive for a model of its size. The researchers tested Aero Realtime on a continuous 20-minute video stream. Even as the video played on, the model managed to keep its "lag"—the delay between what is happening in the video and what the AI is saying—at a median of 84 milliseconds. That is less than the blink of an eye. Even at the 95th percentile (meaning 95% of the time), the delay was only 173 milliseconds. This means the AI stays within 200 milliseconds of the "real" timeline, effectively listening and speaking in real-time.

However, the paper is careful not to claim this is the absolute best AI in the world at understanding video. When tested on a standard benchmark called OVOBench, the 4-billion-parameter model (which is relatively small) scored well but didn't beat the largest, most powerful 7-billion-parameter models that are designed for offline, non-real-time tasks. The authors suggest this gap exists because the model had to learn a completely new way of interacting (the "duplex" style) and didn't have as much specific training data for this exact type of real-time conversation as the older models had for traditional questions.

In short, Aero Realtime proves that it is possible to build an AI that doesn't just wait for its turn to speak, but can truly converse in real-time, hearing and speaking simultaneously without breaking a sweat. It's a step toward AI that feels less like a robot waiting for instructions and more like a friend who is actually paying attention to the world with you. While it's not yet the smartest video expert in the room, it is the first to truly master the art of keeping up with the flow of time.

Technical Summary: Aero Realtime

Problem Statement

Current multimodal language models (LMMs) largely inherit a turn-based, prefill-then-decode architecture. While recent streaming models process observations incrementally, they remain non-duplex: once generation begins, new observations cannot naturally enter the active generation stream. This creates a fundamental mismatch with continuous, real-time interaction where a model must perceive the world while simultaneously deciding when to speak and what to say.

Existing attempts to address this via micro-turn polling or external response gates suffer from significant drawbacks:

Fragmentation: They break continuous interaction into discrete turns or decision points.
Decoupling: Response timing is separated from the language generation objective.
Inefficiency: External gating mechanisms complicate deployment by misaligning with the KV-cache reuse and prefill-decode execution paths optimized in modern inference engines.

The core challenge is to unify continuous perception, response timing, and lexical generation without breaking efficient incremental inference.

Methodology: Aero Realtime

Aero Realtime is a 4B streaming multimodal model designed with a duplex architecture that aligns video, audio, and textual output on a shared temporal grid.

1. Architecture and Temporal Alignment

Shared Temporal Grid: The model divides continuous time into approximately 80-ms audio slots.
Unified Output: Each slot predicts exactly one token: either a lexical token (speech) or a silence token ([P]).
Duplex Mechanism: By predicting silence or speech within the same autoregressive objective, the model learns when to respond and what to generate simultaneously.
State Fusion: At each audio slot $t$ , the current audio representation is fused with the preceding output token embedding ( $e^a_t = z^a_t + E(y_{t-1})$ ). Visual tokens (timestamped frames) are interleaved into this stream. This allows the model to maintain listening context and response history at every slot.
Causal Recurrence: The model follows a causal audio-text recurrence where visual states are inserted by timestamp, enabling the model to "listen while speaking" without pausing perception.

2. Training Strategy

Data Curation: The authors constructed a Realtime QA dataset by sampling video prefixes and generating temporally grounded QA pairs using GPT-5.4, ensuring no future frames are exposed. This is combined with converted video QA and caption data (LLaVA-Video, LiveCC, EgoIT, QAEgo4D).
Slot-Aligned Supervision: Responses are mapped to the first audio slot at or after their timestamp. Unoccupied positions are supervised as silence tokens ([P]).
Silence-Aware Optimization: To prevent the model from becoming overly silent due to the high frequency of silence tokens, a masking ratio ( $r$ ) is applied to exclude a fraction of [P] targets from the loss calculation (optimized at $r=0.95$ in Stage 1).
Two-Stage Training:
- Stage 1: Adapts the pretrained turn-based model to the aligned stream interface using the full data mixture (2B tokens).
- Stage 2: Refines response quality and video understanding using a focused subset of realtime data and specific video QA corpora (1B tokens).

3. Inference and Serving

Delta Inference: The system appends only the newest multimodal slot (audio feature + optional visual tokens) and carries the sampled token into the next state.
KV-Cache Reuse: The scheduler removes the sampled suffix, appends the new delta, and reuses the KV-cached prefix. This avoids repeated prefill steps, enabling efficient incremental execution.
Three-Level Parallelism: To handle long multimodal streams, the authors employ:
1. Workload-balanced Frame Parallelism: Distributes visual frames across ranks based on patch workload to balance vision tower compute.
2. Unified Packed-Sequence Parallelism: Uses DeepSpeed Ulysses to partition audio and language sequences without padding.
3. Modality-Aware Distribution: Ensures all three components (vision, audio, language) are distributed to avoid memory bottlenecks.

Key Results

Performance on OVOBench

Evaluated on the OVOBench (a benchmark for online video understanding), the 4B Aero Realtime model achieves:

Realtime: 61.49
Backward: 44.07
Forward: 39.36
Average: 48.31

While it trails the strongest 7B+ baselines (e.g., Streamo, HERMES) in raw benchmark scores, it is the only evaluated architecture that admits new observations into an active decoding sequence while modeling silence and speech natively. The authors attribute the performance gap to the distribution shift from turn-based pretraining to dense aligned streams and the limited amount of native realtime supervision.

Latency and Throughput

Processing Lag: On four NVIDIA A6000 GPUs, Aero Realtime maintains a median lag of 84 ms and a P95 lag of 173 ms over a 20-minute continuous video stream.
Realtime Capability: The system remains within 200 ms of the source timeline for the entire 20-minute duration, demonstrating sustained real-time processing rather than short-clip throughput.
Training Efficiency: The training stack sustains over 1,600 tokens per GPU per second on A100-40G GPUs, completing both training stages within one day.

Significance and Claims

The paper claims that Aero Realtime demonstrates the feasibility of fully aligned input-output modeling for duplex, proactive, and hardware-aligned multimodal interaction.

Architectural Innovation: It unifies perception, timing, and generation under a single autoregressive objective, eliminating the need for external gating or micro-turn polling.
Hardware Alignment: By preserving KV-cache reuse and aligning with standard prefill-decode execution paths, it offers a practical serving recipe for real-time systems.
Modest Scope: The authors acknowledge that Aero Realtime is an architectural exploration. It is capped at 12.5 tokens/second due to the 80-ms grid and currently trails state-of-the-art video LLMs in benchmark accuracy. They note that latency reflects deployment policy and that path-level deduplication does not guarantee content-level independence from evaluation videos.

Future work suggested by the authors includes adaptive-rate decoding, more native realtime supervision, and stronger contamination audits.