💬 NLP

APB-V: Accelerating Long-Video Understanding via Sequence-Parallelism-aware Approximate Attention

APB-V is a sequence-parallel framework that accelerates long-video inference in Large Multimodal Models by distributing approximate attention across multiple GPUs, achieving significant speedups over existing methods without requiring visual compression or sacrificing performance.

Original authors: Yuxiang Huang, Mingye Li, Xu Han, Chaojun Xiao, Weilin Zhao, Ao Sun, Ziqi Yuan, Hao Zhou, Fandong Meng, Zhiyuan Liu

Published 2026-06-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuxiang Huang, Mingye Li, Xu Han, Chaojun Xiao, Weilin Zhao, Ao Sun, Ziqi Yuan, Hao Zhou, Fandong Meng, Zhiyuan Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive library of video clips, and you want a super-smart AI assistant to watch them all and answer a specific question, like "What was the secret word whispered in the car?"

The problem is that these videos are so long and detailed that the AI gets overwhelmed. It's like asking a single librarian to read every single page of a million books at once to find one sentence. The process is slow, expensive, and often forces the librarian to skip pages (summarize) just to finish, which means they might miss the important details.

This paper introduces APB-V, a new way to organize the "librarians" (computers) so they can watch these long videos together, fast, and without missing a thing.

Here is how it works, using simple analogies:

1. The Old Problem: The "Single Librarian" Bottleneck

Currently, when an AI watches a long video, it has to process every single frame. If the video is 1440p (high definition) and has many frames, the amount of data is huge.

The Bottleneck: It's like trying to fit a whole ocean into a single cup. The computer runs out of memory or takes forever to calculate.
The Old Fix: To make it faster, people used to tell the AI, "Just look at every 10th frame" or "Throw away the blurry parts."
The Downside: This is like reading a book but skipping every other page. You finish faster, but you might miss the plot twist or the secret word. The AI gets faster but gets "dumber."

2. The New Solution: APB-V (The "Team of Librarians")

APB-V changes the game by using multiple computers (GPUs) working together, like a team of librarians splitting up a massive task. But instead of just splitting the books randomly, they use a clever strategy called Sequence-Parallelism.

Think of the video as a long train of train cars.

Frame Parallelism: First, the team splits the video frames. One librarian looks at cars 1–10, another at 11–20, and so on. They all start watching at the same time.
The "Anchor" and "Passing" Trick: Here is the magic part. In a normal team, if Librarian A needs to know what Librarian B saw 10 minutes ago, they have to stop and shout across the room. This takes time.
- APB-V's Trick: Instead of shouting everything, Librarian A only shouts the most important bits (the "Passing Blocks") to the others. They keep a small, permanent "Anchor" of the beginning of the video.
- The Result: They don't need to send the whole video back and forth. They just send the "highlight reel" of what matters. This saves a massive amount of time and data traffic.

3. Balancing the Load (The "ZigZag" Strategy)

If you just split the work evenly, some librarians might end up with the heavy lifting (calculating complex scenes) while others have easy tasks.

The Fix: APB-V uses a ZigZag pattern. Imagine the librarians are arranged in a line. The first librarian gets the first chunk and the last chunk, the second gets the second and the second-to-last, and so on.
Why? This ensures everyone has an equal amount of "thinking" to do, so no one is waiting around for the slowest person to finish.

4. The Results: Fast and Accurate

The authors tested this on huge videos (like 64 frames of 1440p resolution).

Speed: It was 12.7 times faster than the current standard method (FlashAttention).
Accuracy: Unlike the old methods that skipped pages and got the answers wrong, APB-V kept all the visual details. It got the answers just as right as if it had watched the whole video slowly, but it did it in a fraction of the time.

Summary

APB-V is like organizing a team of experts to watch a marathon video. Instead of one person struggling to read every page, or everyone skipping pages to finish fast, the team splits the work, shares only the critical highlights, and balances the load perfectly. The result is that they find the answer super fast without missing a single important detail.

The paper claims this is specifically for long-video understanding on multiple computers (like in data centers for surveillance or autonomous driving), and it doesn't work on a single computer because the magic relies on the team effort.

Technical Summary: APB-V

Problem Statement

The efficiency of long-video inference in Large Multimodal Models (LMMs) remains a critical bottleneck. As video duration increases, the number of frames grows, leading to a larger batch size for the visual encoder and a significant increase in the number of generated visual embeddings. Since the attention layers in LLM backbones exhibit quadratic complexity ( $O(n^2)$ ) with respect to input length, these additional embeddings drastically extend the input sequence, slowing down inference speed and increasing computational costs.

Existing solutions generally fall into two categories, both of which face limitations:

Intrinsic Attention Optimization: Methods that optimize the attention mechanism or reduce Key-Value (KV) cache size within the LLM. These fail to address the computational costs of the visual encoder and feed-forward networks (FFNs).
Explicit Input Reduction: Methods that compress visual embeddings via token pruning or pooling. While effective at reducing sequence length, these approaches often sacrifice fine-grained video details, leading to significant performance degradation (e.g., ~25% accuracy drop in some cases) and failing to achieve satisfactory efficiency gains without compromising quality.

Single-GPU approaches struggle to overcome the efficiency-performance trade-off, necessitating a scaling dimension that leverages multiple GPUs while suppressing quadratic complexity.

Methodology: APB-V

The authors propose APB-V, a sequence-parallelism-aware approximate attention framework designed to accelerate long-video inference across multiple GPUs (hosts). The framework integrates three core components:

1. Framework Design

Frame Parallelism: Inspired by LONGVILA, the visual encoder processes video frames independently. APB-V distributes these frames across multiple hosts to handle high computational intensity.
Context Splitting & ZigZag Arrangement: The input embeddings are partitioned into an Anchor Block ( $B_a$ ), a Query Block ( $B_{qr}$ ), and Context Blocks ( $B^{(h)}$ ). To address the imbalance in decoder-only models where earlier tokens attend to more context than later ones, APB-V employs a ZigZag-style arrangement. This maps physical hosts to pairs of virtual hosts (e.g., host $h$ and $2H-1-h$ ) to ensure balanced attention computation loads.
Approximate Attention Mechanism: Instead of computing dense attention across all tokens, APB-V uses a local KV cache compression strategy:
- Anchor Block: A fixed initial segment of embeddings.
- Passing Blocks: For each host, the most essential Key-Value pairs from previous hosts are identified via query-to-context attention scores and aggregated into "passing blocks."
- Computation: Attention is computed over the Anchor, the local Context, the aggregated Passing Blocks, and the Query. Crucially, the full set of visual embeddings is preserved (no input reduction), but the cross-host communication is compressed to only essential KV pairs.

2. System-Level Optimizations

To maximize efficiency, APB-V incorporates four specific system optimizations:

Visual Load Balancing: Frames are distributed evenly across virtual hosts to prevent bottlenecks in the visual encoding stage.
Fused Context and Query Forward: Unlike previous methods that process context and query separately (often memory-bound for short queries), APB-V fuses them into a single forward pass, merging partial query results using FlashAttention's lse (log-sum-exp) outputs.
Approximate Attention Load Balancing: The ZigZag mapping ensures that the total length of passing blocks is balanced across physical hosts, preventing uneven workloads.
Overlapped Communication: The transfer of passing blocks and partial query attention results is performed concurrently with attention calculations, hiding communication latency.

Key Contributions

APB-V Framework: The introduction of a sequence-parallel framework that utilizes local KV cache compression and distributed approximate attention. This strikes a balance between efficiency and performance by avoiding input compression while reducing computation.
System Optimizations: The design of a load-balancing strategy and a two-stage attention mechanism where communication perfectly overlaps with computation, significantly unleashing the potential of multi-GPU inference.
Comprehensive Evaluation: Extensive benchmarking demonstrating that APB-V achieves the best trade-off between performance and efficiency compared to existing baselines.

Experimental Results

The authors evaluated APB-V on LongVideoBench and VNBench using models including InternVL3-2B, Qwen2.5VL-3B, and Qwen2.5VL-7B.

Performance: APB-V achieves near-lossless accuracy compared to full attention (FULLATTN). It outperforms other approximate attention methods (like APB, STARATTN, XATTN) and token pruning methods (SLOWFAST), which suffer from significant accuracy drops (e.g., SLOWFAST drops ~25%).
Speedup: APB-V delivers substantial speedups over baselines:
- 12.72× faster than FLASHATTN.
- 1.70× faster than ZIGZAGRING.
- 1.18× faster than APB.
- These gains are observed particularly in high-resolution (1440p) and long-video scenarios.
Scalability: The framework maintains stable task accuracy across different host counts ( $H \in \{2, 4, 6, 8\}$ ) and demonstrates robustness even when inter-group communication is limited by InfiniBand (IB) bottlenecks, showing the smallest performance drop compared to other methods.

Significance and Claims

The paper claims that APB-V offers a promising pathway for long-video understanding by scaling computational power while suppressing quadratic complexity. By distributing approximate attention across multiple GPUs, the framework enables the processing of more visual embeddings without the need for compression, thereby preserving fine-grained video details that are often lost in token-pruning approaches.

The authors emphasize that APB-V is specifically designed for scenarios constrained by end-to-end time (Time-to-First-Token, TTFT), such as surveillance and autonomous driving, where single-GPU inference is infeasible due to the sheer volume of tokens generated by long videos. The work demonstrates that with well-designed parallelism and system optimizations, it is possible to achieve significant acceleration without the typical trade-off of degraded model performance.

Limitations: The authors note that APB-V is currently restricted to decoder-only transformer-based LMMs and multi-GPU inference. It is not compatible with convolutional networks, and on a single GPU, it degenerates to standard FlashAttention, offering no advantage.