PRISM: Video Dataset Condensation with Progressive Refinement and Insertion for Sparse Motion

1. Problem Statement

Video Dataset Condensation aims to synthesize a small, representative dataset that allows models to achieve performance comparable to training on massive original datasets, thereby reducing storage and computational costs. However, video condensation faces a unique challenge compared to image condensation: the inseparable interdependence between spatial appearance and temporal dynamics.

• Limitations of Prior Work: Existing methods (e.g., Wang et al.) typically follow a static/dynamic disentanglement paradigm. They decompose videos into static content and auxiliary motion signals, optimizing them in separate stages.
• The Flaw: This approach often misrepresents real-world actions where content and motion are mutually constitutive. For example, a static frame of hands clapping is visually identical to the moment hands begin to separate; without temporal context, the static state is ambiguous. Furthermore, prior methods often rely on fixed-frame optimization (e.g., a fixed number of frames per video), leading to unnecessary redundancy or failure to capture complex non-linear dynamics.

2. Methodology: PRISM

The authors propose PRISM (Progressive Refinement and Insertion for Sparse Motion), a holistic approach that treats video as a unified spatiotemporal structure from the outset, avoiding artificial disentanglement.

Core Philosophy

Instead of optimizing every frame or using a fixed set of frames, PRISM assumes that simple or low-velocity motion can be effectively approximated by linear interpolation between key temporal anchors. The model only allocates representational capacity to frames where this linear assumption fails.

Key Technical Components

A. Sparse Initialization & Linear Interpolation

• Initialization: Each synthetic video sequence is initialized with only two temporal anchors: the start frame ($t=1$) and the end frame ($t=T$).
• Reconstruction: All intermediate frames are generated via linear interpolation between the current set of key-frames.
• Optimization: Only the designated key-frames are trainable parameters; intermediate frames are updated indirectly through the interpolation mechanism, ensuring holistic spatiotemporal coupling.

B. Gradient-Guided Frame Insertion
The core innovation is an adaptive mechanism to identify and insert "critical" frames where linear interpolation is insufficient.

• Theoretical Basis (Lemma 1): If the gradient of an intermediate interpolated frame ($g_t$) is directionally misaligned (negative correlation) with the gradients of its two adjacent key-frames ($g_i, g_{i+1}$), then updating only the endpoints cannot minimize the loss at the intermediate point.
• Detection Mechanism: During training, the system calculates the cosine similarity between the gradient of a candidate interpolated frame and the gradients of its adjacent key-frames.
  • If $\cos(\nabla \mathcal{L}(s_t), \nabla \mathcal{L}(s_{k_i})) < \epsilon$ AND $\cos(\nabla \mathcal{L}(s_t), \nabla \mathcal{L}(s_{k_{i+1}})) < \epsilon$, the frame is identified as a critical temporal junction.
• Action: These frames are promoted to the key-frame set and become directly trainable in subsequent iterations.

C. Training Phases
To ensure stability, the training process includes two specific phases:

1. Warm-Up Phase (20%): Frame insertion is disabled. Only the initial start and end frames are optimized to stabilize the global trajectory before insertion begins.
2. Progressive Refinement (60%): The gradient-guided insertion mechanism is active. Frames are added iteratively based on gradient misalignment.
3. Cool-Down Phase (20%): Frame insertion is disabled again. This ensures that frames inserted near convergence receive sufficient updates and are not left under-optimized.

D. Optimization Objective
The method minimizes the discrepancy between the feature distributions of real video batches and synthetic video batches (using a 3D CNN feature extractor). Gradients are backpropagated exclusively to the sparse set of key-frames.

3. Key Contributions

1. Holistic Paradigm: PRISM is the first video condensation method to treat video as a unified spatiotemporal entity, rejecting the static/dynamic disentanglement used in prior work.
2. Adaptive Sparse Representation: It introduces a gradient-guided insertion mechanism that starts with minimal anchors (2 frames) and adaptively inserts frames only where non-linear dynamics exist, rather than using a fixed frame count.
3. State-of-the-Art Efficiency: It achieves competitive performance while drastically reducing storage requirements by learning a sparse set of essential temporal anchors.

4. Experimental Results

The authors evaluated PRISM on standard benchmarks: MiniUCF, HMDB51, Kinetics-400, and Something-Something V2.

• Performance: PRISM consistently outperforms or matches prior methods (including Coreset selection and the only other video condensation method, Wang et al.) across various Videos Per Class (VPC) settings.
  • Example: On MiniUCF (1 VPC), PRISM achieves 17.9% accuracy, outperforming the previous best (Wang et al.) at 17.5%.
  • Large Scale: On Kinetics-400, PRISM achieves 7.1% (VPC 1), outperforming Wang et al. (6.3%) and recovering top performance at VPC 1 compared to image-based baselines.
• Storage Efficiency (The Major Win):
  • PRISM achieves 5x to 10x reduction in storage compared to baselines while maintaining or improving accuracy.
  • Example: On MiniUCF (1 VPC), PRISM requires only 20 MB of storage compared to 94 MB for the baseline, while achieving higher accuracy.
  • Unlike baselines where storage grows linearly with VPC, PRISM's storage grows sub-linearly because it only adds frames when necessary.
• Generalization: PRISM demonstrates strong cross-architecture generalization, performing well on models (ConvNet3D, CNN+GRU, CNN+LSTM) different from the training backbone, suggesting it captures intrinsic action structures rather than overfitting to specific architectural biases.
• Ablation Studies:
  • Removing frame insertion causes a significant performance drop, proving the necessity of adaptive refinement.
  • Starting with more than 2 initial frames actually degrades performance, validating the "start-small" philosophy.
  • Cosine similarity (directional alignment) is proven superior to L2 distance for frame selection.

5. Significance

PRISM represents a paradigm shift in video dataset condensation. By moving away from the flawed assumption that content and motion can be disentangled, and by replacing fixed-frame constraints with an adaptive, gradient-driven sparse representation, it solves the fundamental trade-off between storage efficiency and motion fidelity.

• Practical Impact: It enables the deployment of video learning models in resource-constrained environments (e.g., edge devices) by reducing storage footprints by orders of magnitude without sacrificing recognition accuracy.
• Theoretical Insight: It establishes that the "critical moments" in a video are defined by gradient misalignments (non-linear dynamics) rather than arbitrary temporal intervals, providing a new theoretical lens for understanding spatiotemporal data compression.