💻 computer science

FrameSkip: Learning from Fewer but More Informative Frames in VLA Training

The paper introduces FrameSkip, a data-layer framework that enhances Vision-Language-Action (VLA) training efficiency and performance by selectively sampling high-importance frames based on action variation and visual coherence, achieving a 76.15% success rate across benchmarks while retaining only 20% of the original trajectory frames.

Original authors: Bin Yu, Shijie Lian, Xiaopeng Lin, Zhaolong Shen, Yuliang Wei, Changti Wu, Hang Yuan, Haishan Liu, Bailing Wang, Cong Huang, Kai Chen

Published 2026-05-14

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Bin Yu, Shijie Lian, Xiaopeng Lin, Zhaolong Shen, Yuliang Wei, Changti Wu, Hang Yuan, Haishan Liu, Bailing Wang, Cong Huang, Kai Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to make a cup of coffee. You record a video of a human doing it from start to finish. This video is 10 minutes long.

The Old Way (The Problem)
In the past, when training AI robots, researchers would feed the computer every single frame of that 10-minute video. They treated every second as equally important.

But think about the video:

Minutes 0–2: The human walks slowly to the kitchen counter. (Nothing much happens).
Minute 3: They grab the coffee mug. (This is a critical moment!).
Minutes 4–8: They walk slowly to the coffee machine and wait. (Again, mostly just walking).
Minute 9: They press the button and the coffee pours. (Another critical moment!).
Minute 10: They walk away.

The paper argues that the "old way" is inefficient. The robot spends 80% of its study time watching the human walk slowly, and only 20% of its time watching the actual tricky parts (grabbing, pouring). It's like studying for a math test by reading the same boring chapter over and over, while only glancing at the actual formulas you need to solve the problems.

The New Solution: FRAMESKIP
The authors created a tool called FRAMESKIP. Think of it as a super-smart video editor that works before the robot starts learning.

Instead of showing the robot the whole 10-minute video, FRAMESKIP cuts out the boring parts and creates a "highlight reel." It keeps the slow walking parts very short but zooms in and repeats the important parts (like the hand grabbing the mug) so the robot sees them more often.

How does it know what to keep?
FRAMESKIP uses four simple "clues" to decide which frames are important:

Action Changes: Did the robot's hand suddenly move fast? (Keep that frame).
Visual Changes: Did the picture change because the object moved? (Keep that frame).
Task Progress: Is this the middle of the task where things usually go wrong? (Keep that frame).
Gripper Moves: Did the robot's "fingers" open or close? (Keep that frame).

The Magic Trick
The coolest part is that FRAMESKIP doesn't change the robot's brain or how it learns. It just changes what data the robot sees. It's like giving the same student a better study guide instead of trying to make the student smarter.

The Results
The researchers tested this on three different robot simulation games.

The Result: When they used the "highlight reel" (keeping only 20% of the original frames), the robots actually got better at their tasks than when they watched the full, boring video.
The Score: The robots succeeded about 76% of the time with the new method, compared to only 66% with the old method.

In a Nutshell
The paper says: "We don't need to show robots every second of a video to teach them. If we skip the boring parts and focus on the moments where the robot actually has to do something, the robot learns faster and does a better job."

It's the difference between reading a whole encyclopedia to learn how to tie a shoe versus just watching a 30-second video of someone tying a shoe. FRAMESKIP helps the robot find the "30-second video" inside the "encyclopedia."

Technical Summary: FrameSkip: Learning from Fewer but More Informative Frames in VLA Training

Problem Statement

Vision-Language-Action (VLA) policies are typically trained on dense robot demonstration trajectories collected via teleoperation. A prevailing convention in this field assumes that every recorded frame provides equally useful supervision, leading to uniform sampling of all timesteps during training.

The authors argue that this approach creates a temporal supervision imbalance. Robot trajectories often contain long, low-change segments (e.g., approaching an object, steady transport) that dominate the training stream, while manipulation-critical transitions (e.g., alignment, contact, grasping, release) appear sparsely. Consequently, under a fixed optimization budget, rare but decision-critical transitions are diluted by abundant but weakly informative observations. This leads to policies that adapt well to smooth motions but remain brittle at sparse, critical interaction stages.

Methodology: FRAMESKIP

The paper introduces FRAMESKIP, a data-layer frame selection framework designed to reallocate temporal supervision toward high-importance frames without modifying the VLA architecture, action head, loss function, or inference procedure.

1. Frame Importance Estimation

FRAMESKIP assigns an importance score to each frame based on four lightweight trajectory cues, normalized within each trajectory:

Action Variation Importance (AVI): Measures local action dynamics, including the difference from the previous action and short-range variation in future steps. High AVI indicates abrupt motion changes or contact events.
Visual-Action Coherence (VAC): Captures visually grounded transitions by comparing the magnitude of visual feature changes (extracted via DINOv2) relative to action changes. This identifies stages where the environment changes significantly due to the robot's action.
Task Progress Importance (TPI): Encodes a structural prior regarding where critical stages occur. It uses a dataset-adaptive Gaussian Mixture Model (GMM) fitted to annotated critical stage centers, or a generic Gaussian prior centered at the trajectory midpoint if annotations are unavailable.
Gripper-Transition Preservation: A multiplicative factor that boosts scores for frames involving absolute changes in gripper or end-effector states, ensuring contact-related stages are not pruned.

The final score is a weighted combination: $I(t) = \alpha \cdot \text{AVI} + \beta \cdot \text{VAC} + \gamma \cdot \text{TPI}$ .

2. Ratio-Aware Pruning

Given a target retention ratio $r$ (e.g., retaining 20% of frames), FRAMESKIP constructs a subset of timestep indices.

It calculates a threshold based on the $(1-r)$ -quantile of importance scores.
It enforces practical constraints: retaining the first/last frames, gripper transition frames, and frames with top-decile action changes.
If the quantile rule deviates from the target count, frames are added or removed based on descending importance.
Temporal consistency constraints are applied to prevent excessively large gaps between retained frames.

3. Training Integration

FRAMESKIP operates as a dataloader intervention:

Warmup: The model is trained for an initial period ( $N_{warm}$ ) on the full-frame dataset ( $r=1.0$ ) to establish stable visual-action grounding.
Pruned Sampling: After warmup, the majority of mini-batches are drawn from the compressed view ( $r < 1.0$ ).
Anchoring: To prevent overfitting to sparse transitions, a small fraction of mini-batches (e.g., 1 in 5) is drawn from the full-frame view to refresh global trajectory context.
Index Remapping: The system maps requested timesteps to the nearest retained indices in the cached trajectory, leaving the underlying dataset structure and model architecture unchanged.

Key Contributions

Identification of Temporal Imbalance: The paper identifies temporal supervision imbalance as a practical, underexplored issue in VLA training, arguing that not all frames are equally informative.
FRAMESKIP Framework: An architecture-agnostic, data-layer framework that selects informative frames using lightweight cues (AVI, VAC, TPI) and gripper preservation, operating solely within the dataloader.
Systematic Empirical Study: A comprehensive evaluation of importance-guided frame retention, including ablations on retention ratios, importance metrics, and warmup schedules, demonstrating that gains stem from supervision reallocation rather than mere data reduction.

Experimental Results

The authors evaluated FRAMESKIP on three simulation benchmarks: RoboCasa-GR1 (bimanual manipulation), SimplerEnv (WidowX generalization), and LIBERO (Franka single-arm manipulation).

Performance Gains: Across all three benchmarks, FRAMESKIP improved the macro-average success rate from 66.50% (full-frame training) to 76.15%.
Efficiency: In the main setting, the method retained only 20% of unique frames per trajectory while achieving these gains.
Ablation Insights:
- Retention Ratio: Performance peaked at moderate compression levels (e.g., 50% or 20-30%), outperforming both full-frame training and aggressive pruning (10%).
- Metrics: The full combination of AVI, VAC, and TPI outperformed variants using only random selection or single metrics (e.g., AVI alone), confirming the complementary nature of the cues.
- Warmup: A short full-frame warmup (e.g., 5,000 steps) was sufficient to stabilize learning before switching to pruned views.

Significance and Claims

The paper claims that FRAMESKIP demonstrates that frame-level supervision allocation is a practical lever for improving VLA training. By reframing frame pruning not merely as data reduction but as a mechanism to rebalance training toward fragile, critical moments, the method achieves higher success rates with significantly compressed trajectory views.

The authors emphasize that FRAMESKIP is complementary to existing scaling efforts (model architecture, action representation, data mixtures). It offers a direct way to study frame importance as a training principle without requiring architectural changes or auxiliary models for quality prediction. The ultimate goal stated is to make "frame importance a first-class object in embodied multimodal learning."