💻 computer science

Revisiting Token Compression for Accelerating ViT-based Sparse Multi-View 3D Object Detectors

The paper proposes SEPatch3D, a novel framework that accelerates ViT-based sparse multi-view 3D object detectors by dynamically adjusting patch sizes and enhancing coarse features with fine-grained details, achieving up to 57% faster inference than StreamPETR while maintaining comparable detection accuracy.

Original authors: Mingqian Ji, Shanshan Zhang, Jian Yang

Published 2026-04-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mingqian Ji, Shanshan Zhang, Jian Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are driving a self-driving car. To see the world, the car takes pictures from multiple cameras (like a human looking around). A smart computer program (called a Vision Transformer or ViT) looks at these pictures to find other cars, pedestrians, and obstacles.

However, this "smart computer" is currently very slow. It's like a chef trying to chop every single grain of rice in a huge bowl to make a salad. It takes too long, and the car might not react fast enough in an emergency.

This paper introduces a new method called SEPatch3D to make the chef faster without ruining the salad. Here is how it works, using simple analogies:

The Problem: The "One-Size-Fits-All" Mistake

Current methods try to speed things up by either:

Throwing things away: Cutting out parts of the picture they think are unimportant (like throwing away the background). Problem: Sometimes the background holds clues that tell the car "there is no car here," which is crucial for safety.
Smushing things together: Gluing similar pixels together. Problem: This blurs the edges, making it hard to tell exactly where a car is.
Zooming out: Looking at the picture in bigger, chunkier blocks. Problem: If you zoom out too much, you miss the small details, like a pedestrian's hand waving.

The Solution: SEPatch3D (The "Smart Zoom" Chef)

The authors propose a system that acts like a smart, adaptive chef. Instead of treating the whole picture the same way, it changes its strategy based on what is happening in the scene.

1. The "Spatiotemporal" Eye (SPSS)

Imagine you are looking out your car window.

When a car is far away: It looks small and blurry. Your brain doesn't need to focus on every tiny detail of that distant car. The system says, "Okay, let's look at that part of the picture in big, coarse chunks." This saves a massive amount of brainpower (computation).
When a car is close: It's huge and detailed. You need to see every tire and license plate. The system says, "Whoa, that's close! Let's switch to tiny, detailed chunks for this part."

The Magic: The system looks at where objects were in the previous second (history) and where they are now (motion). If objects are getting closer, it zooms in (small patches). If they are far away, it zooms out (large patches).

2. The "Highlighter" (IPS)

Even when looking at a distant car in "big chunks," the system worries it might miss something important. So, it uses a highlighter.
It scans the picture and asks: "Which parts of this big chunk are actually interesting?"

It ignores the boring, flat road (low information).
It highlights the edges of a car or a person (high information).
It only keeps the "highlighted" parts for extra attention, ignoring the rest of the big chunk.

3. The "Detail Injector" (CGFE)

This is the final trick. Even though the system is looking at the distant car in "big chunks," it still has access to the original "tiny detail" version of the image in the background.
It takes the highlighted parts from the "big chunk" view and injects the fine details from the "tiny chunk" view into them.

Analogy: It's like looking at a map of a city from space (big view) to see the general area, but then instantly zooming into the specific street corner to see the exact house number, all without actually having to scan the whole map at high resolution.

The Result: Faster and Smarter

By doing this, the system achieves two amazing things:

Speed: It runs 57% faster than the previous best methods. It's like the chef chopping only the necessary ingredients instead of the whole bowl.
Accuracy: It doesn't lose its ability to see things. It keeps the safety of seeing the background and the precision of seeing the details.

Why This Matters

In the world of self-driving cars, speed is safety. If the computer takes too long to process an image, the car might brake too late. SEPatch3D allows the car to "think" faster by being smart about what it looks at and how closely it looks, ensuring a safer and smoother ride for everyone.

1. Problem Statement

Vision Transformer (ViT)-based sparse multi-view 3D object detectors (e.g., StreamPETR) have achieved state-of-the-art accuracy but suffer from high inference latency due to the computational cost of processing dense tokens. To accelerate these models, researchers have explored token compression strategies such as:

Token Pruning: Removing low-importance tokens.
Token Merging: Aggregating similar tokens.
Patch Size Enlargement: Increasing the receptive field per token to reduce token count.

The Core Limitation: The authors argue that existing strategies fail in 3D detection contexts:

Pruning often discards informative background regions, which are crucial for learning "hard negatives" (distinguishing objects from complex backgrounds).
Merging disrupts contextual consistency and semantic alignment.
Uniform Patch Enlargement loses fine-grained semantic details necessary for accurate object localization, leading to significant accuracy degradation when patch sizes exceed a certain threshold.

The challenge is to design a compression strategy that reduces computational cost (latency) without sacrificing the fine-grained details or background context required for high-precision 3D detection.

2. Methodology: SEPatch3D

The authors propose SEPatch3D (Dynamic Patch Size Selection and Informative Patch Enhancement), a framework that dynamically adjusts patch sizes while preserving critical semantic information. The pipeline consists of three main components:

A. Spatiotemporal-aware Patch Size Selection (SPSS)

Instead of using a fixed patch size, SPSS dynamically selects between a small patch size ( $P_s$ ) and a large patch size ( $P_l$ ) for the entire frame based on scene content.

Mechanism: It uses object queries from the previous frame ( $T-1$ ) to estimate the scene's spatial layout and temporal motion trends.
Logic:
- Small Patches ( $P_s$ ): Assigned when objects are close and growing (approaching). This preserves fine-grained details for nearby objects.
- Large Patches ( $P_l$ ): Assigned when objects are far and shrinking (receding) or the scene is background-dominated. This reduces redundant computation on distant backgrounds.
Stability: A linear regression model is used to smooth depth trends over time, preventing abrupt patch size changes between consecutive frames.

B. Informative Patch Selection (IPS)

To mitigate potential detail loss from using large patches, the system identifies which specific regions require high-fidelity processing.

Mechanism: Instead of relying on learned attention scores (which may be unreliable in shallow layers), IPS uses entropy calculated from patch features. High entropy indicates texture-rich or edge areas (informative regions).
Selection: Patches with entropy values exceeding the scene's mean entropy are selected. This adaptive approach avoids fixed $K$ -selection, allowing the system to adapt to varying scene complexities.
Alignment: Selected indices from the fine-grained (vanilla) patch space are projected to the coarse patch space to ensure spatial alignment.

C. Cross-Granularity Feature Enhancement (CGFE)

This module refines the selected coarse patches by injecting fine-grained details.

Mechanism: A cross-attention mechanism is employed where the selected coarse patches act as queries and the corresponding fine-grained (vanilla) patches act as keys and values.
Process: The coarse patches retrieve detailed information from the fine patches. A residual connection is applied to preserve the original global structure while enriching the semantic features of the informative regions.

3. Key Contributions

Critical Revisit of Compression Strategies: The paper provides a systematic analysis of why standard token pruning and merging are suboptimal for 3D detection, highlighting the loss of background cues and contextual consistency.
Novel Adaptive Framework: SEPatch3D introduces a dual-stage approach:
- Dynamic Granularity: Adapts patch sizes based on spatiotemporal object distribution.
- Selective Enhancement: Uses entropy-based selection and cross-granularity attention to recover lost details in coarse patches.
Efficiency-Accuracy Balance: The method achieves significant speedups (up to 57%) while maintaining detection accuracy comparable to or better than the baseline and state-of-the-art (SOTA) methods.

4. Experimental Results

The method was evaluated on nuScenes and Argoverse 2 validation sets.

Performance on nuScenes (640 × 1600 resolution):
- SEPatch3D-faster achieved a 57.7% reduction in total inference time (from 1309.9ms to 554.4ms) compared to the StreamPETR baseline.
- It maintained competitive accuracy: 62.4% NDS and 54.2% mAP (vs. 62.7% NDS and 55.8% mAP for baseline), a negligible drop of <1 point.
- It outperformed the SOTA ToC3D-faster by 20% in inference efficiency while maintaining similar accuracy.
Performance on Argoverse 2:
- Achieved a 40.7% reduction in runtime with less than a 1-point drop in CDS (Composite Detection Score).
Generalization:
- The framework successfully generalized across different sparse detectors (DETR3D, Sparse4Dv2) and different ViT backbones (ViT-B, ViT-L), confirming its decoupling from specific model architectures.
Ablation Studies:
- SPSS contributed most to efficiency.
- CGFE was essential for recovering accuracy lost by coarse patching.
- Entropy-based selection outperformed score-based or attention-based selection methods.

5. Significance

Paradigm Shift: The paper challenges the prevailing trend of token pruning in 3D detection, arguing that preserving background context is vital for hard negative mining. It proposes that enlarging patches (coarse-to-fine) is a more viable strategy than removing tokens.
Practical Deployment: By offering a "budget-aware" analysis, the authors provide a tool for practitioners to select optimal patch configurations for specific latency/accuracy requirements without exhaustive retraining.
Real-Time ViT: SEPatch3D bridges the gap between the high accuracy of ViT-based detectors and the real-time latency requirements of autonomous driving, making high-performance 3D perception more feasible for edge deployment.

In conclusion, SEPatch3D demonstrates that intelligent, adaptive token compression—specifically through dynamic patch sizing and selective feature enhancement—can drastically accelerate ViT-based 3D detectors without compromising the semantic integrity required for safe autonomous navigation.