💻 computer science

YOSE: You Only Select Essential Tokens for Efficient DiT-based Video Object Removal

YOSE is an efficient fine-tuning framework for DiT-based video object removal that accelerates inference by adaptively selecting essential tokens via Batch Variable-length Indexing and simulating unmasked regions with a Diffusion Process Simulator, achieving up to 2.5X speedup while maintaining visual quality.

Original authors: Chenyang Wu, Lina Lei, Fan Li, Chun-Le Guo, Dehong Kong, Xinran Qin, Zhixin Wang, Ming-Ming Cheng, Chongyi Li

Published 2026-05-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chenyang Wu, Lina Lei, Fan Li, Chun-Le Guo, Dehong Kong, Xinran Qin, Zhixin Wang, Ming-Ming Cheng, Chongyi Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a video of a busy street, and you want to remove a specific person walking across the frame. In the past, AI models trying to do this were like a team of painters who, instead of just painting over the person, decided to repaint the entire street, the sky, and every car in the background every single time they made a change. This was incredibly slow and wasteful because 90% of the video didn't need to be touched.

This paper introduces YOSE (You Only Select Essential Tokens), a new method that acts like a smart, surgical editor. Instead of repainting the whole canvas, YOSE only paints the specific spots that need fixing, while still making sure the new paint blends perfectly with the untouched parts.

Here is how YOSE works, broken down into simple concepts:

1. The Problem: The "Whole Canvas" Approach

Current AI video tools (based on something called "DiT") are very good at removing objects, but they are slow. Even if you only want to remove a small bird from a video, the computer processes every single pixel of the entire video. It's like using a sledgehammer to crack a nut. The paper notes that even the best existing tools only run at about 10 frames per second (FPS), which is too slow for real-time use.

2. The Solution: Two Smart Tools

YOSE adds two special "gadgets" to the existing AI to make it faster without ruining the quality.

Gadget A: The "Smart Clipper" (Batch Variable-length Indexing)

The Analogy: Imagine you have a stack of 100 homework papers, but only 5 of them have mistakes. A normal teacher grades all 100 papers one by one. YOSE is like a teacher who instantly cuts out the 5 papers with mistakes, grades only those, and then puts them back in the stack.
How it works: The AI looks at the "mask" (the area you want to change). It uses a technique called BVI to physically select only the data points (tokens) inside that mask. It ignores the rest of the video during the heavy lifting. This allows the computer to work on a variable number of items depending on how big the object is, rather than a fixed, huge number.

Gadget B: The "Ghost Whisperer" (Diffusion Process Simulator)

The Analogy: If you only paint the small patch where the person was, the new paint might look like a sticker that doesn't match the lighting or texture of the street around it. You need to know what the rest of the street looks like to blend the new paint in.
The Problem: If you cut out the "bad" parts to save time, the AI forgets what the "good" parts look like. It loses context.
The Fix: YOSE uses a module called DiffSim. It doesn't actually process the "good" parts of the video (which would be slow). Instead, it creates a simulation or a "ghost" of what those good parts are doing. It whispers to the AI, "Hey, the sky over here is blue, and the car over there is moving left," so the AI knows how to blend the new patch seamlessly. It's like having a map of the whole city while only walking through one neighborhood.

3. The "Seamless Stitch" (Fusion Strategy)

Even with the ghost whisperer, there might be a tiny visible line where the new video meets the old video. YOSE uses a final trick: it slightly overlaps the edges and adjusts the brightness and color statistics (mean and variance) to make the transition invisible. It's like feathering the edges of a photo cutout so it disappears into the background.

The Results: Fast and Clean

The paper claims that by using these tricks:

Speed: YOSE is 2.5 times faster than the previous best methods. If the old method took 10 seconds, YOSE takes about 4.
Scalability: The speed depends on the size of the object you remove. If you remove a tiny speck, it's super fast. If you remove a huge building, it slows down, but it never gets slower than the old method.
Quality: The video quality remains just as good as the slow, full-processing methods. In fact, because it doesn't accidentally mess up the background (since it ignores it), the background often looks more stable.

Summary

YOSE is a "smart editor" that realizes you don't need to re-simulate the entire universe to remove a single object. It cuts out the work it doesn't need to do, uses a clever simulation to remember the context of the parts it ignored, and stitches everything back together so perfectly that you can't tell the difference. It turns a slow, heavy process into a fast, surgical one.

1. Problem Statement

Video Object Removal (VOR) aims to erase undesired objects from videos while maintaining spatial realism and temporal consistency. Recent state-of-the-art (SOTA) methods, such as MiniMax Remover, leverage Diffusion Transformers (DiT) to achieve high-quality results. However, these methods suffer from significant inference latency and computational inefficiency.

The Core Issue: Existing DiT-based VOR models perform dense computations over the entire spatiotemporal token space (all frames and pixels), even when only a small, masked region requires processing.
The Consequence: The computational complexity remains constant regardless of the mask size. Since real-world VOR tasks typically involve small mask ratios (e.g., ~70% of cases have masks <20%), this results in massive redundancy. For instance, MiniMax Remover operates at only ~10 FPS, limiting practical application.

2. Methodology: YOSE Framework

The authors propose YOSE (You Only Select Essential Tokens), a lightweight, plug-in fine-tuning framework designed to accelerate DiT-based VOR without re-architecting the underlying model. YOSE introduces two core components:

A. Batch Variable-length Indexing (BVI)

BVI is a differentiable dynamic indexing operator that enables the model to process only the tokens within the masked regions (essential tokens).

Mechanism: Instead of hard indexing (which blocks gradient flow), BVI uses a continuous interpolation-based selection (via grid_sample) to map masked tokens to a variable-length sequence.
Forward Index ( $Ind_F$ ): Selects essential tokens from the masked area for processing.
Backward Index ( $Ind_B$ ): Restores the processed results back to their original spatiotemporal positions.
Benefit: It supports variable-length token processing across a batch, allowing the model to handle different mask sizes efficiently while maintaining gradient flow for end-to-end training. This reduces computational complexity from $O(N_{total})$ to $O(N_{masked})$ .

B. Diffusion Process Simulator (DiffSim) Module

Processing only masked tokens risks breaking semantic continuity with the unmasked (background) regions. DiffSim addresses this by simulating the influence of unmasked regions without processing them.

Mechanism:
1. It constructs a Residual Latent ( $Res_{Nis}$ ) representing the difference between the noise latent and the masked video latent.
2. It generates proxy Key & Value (KV) features for the unmasked regions using learnable parameters: a combining parameter ( $G$ ), a scaling parameter ( $S$ ), and a bias ($Bias$).
3. The formula for the simulated KV is: $KV = G \cdot Lat_{mask} + (1-G) \cdot Res_{Nis}$ , followed by scaling and bias modulation.
Benefit: These simulated features are injected into the DiT's self-attention mechanism. This allows the "inner" (masked) tokens to attend to the "outer" (unmasked) context, preserving semantic and temporal consistency across boundaries without the cost of full-token computation.

C. Fusion Strategy

To eliminate boundary artifacts caused by statistical mismatches between processed and unprocessed regions:

The mask is dilated to create an overlapping region.
The mean and variance of the generated overlapping region are aligned with the original unmasked region.
A weighted blending strategy is applied to ensure a smooth transition.

3. Key Contributions

YOSE Framework: A mask-aware fine-tuning framework that accelerates DiT-based VOR (e.g., MiniMax Remover) by processing only essential tokens, achieving linear computational complexity relative to the mask ratio.
Batch Variable-length Indexing (BVI): A differentiable operator enabling efficient, variable-length token selection and processing, solving the gradient flow and batch parallelization issues associated with dynamic masking.
Diffusion Process Simulator (DiffSim): A novel module that approximates the diffusion influence of unmasked regions within DiT blocks, ensuring global semantic consistency without redundant computation.
Performance: The method achieves up to 2.5× speedup (reaching ~25 FPS for 20% mask ratios) while maintaining visual quality comparable to or slightly better than the baseline.

4. Experimental Results

The authors evaluated YOSE on MiniMax Remover and VACE using the DAVIS and YouTube-VOS datasets.

Speed & Efficiency:
- FLOPs & Latency: YOSE exhibits a linear relationship between inference time and mask ratio.
- Speedup: Achieves 3.3× acceleration at 5% mask ratio and 2.5× acceleration at 20% mask ratio (typical real-world scenarios).
- FPS: Increases from 10 FPS (MiniMax) to **25 FPS** (YOSE) at 480p resolution.
Visual Quality:
- Metrics: YOSE maintains comparable or slightly improved scores on VBench metrics (Motion Smoothness, Aesthetic Quality) and background metrics (PSNR, SSIM, LPIPS).
- Background Preservation: Notably, YOSE (VACE) improved background PSNR by 5.47 dB and SSIM by 0.07 compared to the original VACE, effectively preventing the "hallucination" of mask-shaped artifacts in the background.
Ablation Studies:
- Removing DiffSim leads to a drop in both motion smoothness and background reconstruction quality.
- Removing the Fusion Strategy results in visible boundary artifacts.
- Training Efficiency: BVI enables training with a batch size of 4 on 8 GPUs in ~4 hours, compared to ~11 hours for conventional approaches.

5. Significance

YOSE represents a significant step toward practical, real-time video editing. By decoupling computational cost from the total video size and tying it strictly to the area of interest (the mask), it solves the primary bottleneck of current DiT-based generative video models.

Scalability: It makes high-fidelity video object removal feasible on consumer hardware or for real-time applications where latency is critical.
Generalizability: The framework is model-agnostic (plug-and-play) and has been successfully validated on multiple DiT-based architectures (MiniMax Remover, VACE).
Quality vs. Speed: It demonstrates that aggressive acceleration does not necessarily require a trade-off in visual fidelity; in some cases, the selective processing even improves background stability by preventing the model from over-interpreting unmasked regions.