Multi-modal video data-pipelines for machine learning with minimal human supervision

Technical Summary: Multi-Modal Video Data-Pipelines for Machine Learning with Minimal Human Supervision

Problem Statement
Traditional machine learning workflows often rely on unimodal data (e.g., RGB images for segmentation or text for sentiment) and fixed benchmarks, which can lead to overfitting and poor generalization. While the real world is inherently multi-modal, capturing and correlating these modalities from raw video without extensive manual annotation presents a significant engineering challenge. Existing datasets, such as Dronescapes, often rely on manual curation or rigid scripts to generate auxiliary modalities (like depth or camera normals), creating a bottleneck for scaling multi-modal training data. Furthermore, the deployment of real-time inference on consumer hardware remains a complex trade-off between latency, compute power, and network reliability.

Methodology: The Video Representations Extractor (VRE)
To address these challenges, the authors introduce the Video Representations Extractor (VRE), an open-source, modular, and configurable data pipeline designed to generate multi-modal datasets with minimal human supervision.

• Architecture and Core Loop: VRE operates on a Directed Acyclic Graph (DAG) of representations. A "representation" is defined as a basic block equivalent to a modality, consisting of a unique name, parameters, and dependencies on other representations. The pipeline executes a topological sort of these representations, processing frames in batches. It supports two primary modes:

  • Batching Mode: Optimized for reliability and offline processing. It splits videos into chunks, computes representations, and stores them on disk (using formats like .npz and .jpg) for later training. It includes mechanisms for re-entrancy (skipping already computed frames) and atomic writes to handle interruptions.
  • Streaming Mode: Optimized for low latency. It processes frames sequentially from live sources (webcams, phone cameras) without disk I/O overhead, suitable for real-time inference.

• Scalability Strategies:

  • Multi-GPU Batching: The paper outlines two strategies for parallelizing work across multiple accelerators. Strategy 1 splits the video timeline into independent chunks, assigning each to a separate GPU process. Strategy 2 splits the representations themselves into independent groups (based on the DAG), allowing different GPUs to compute different modalities of the same video chunk simultaneously.
  • Hardware Agnosticism: VRE is designed to work on CPUs, GPUs, and potentially other accelerators (TPUs, NPUs), treating them as generic "accelerators."

• Implementation Details: The pipeline is configured via YAML files, allowing users to define complex dependency chains (e.g., deriving camera normals from depth maps, which are derived from RGB). It includes a repository of pre-implemented representations, including color spaces, edge detection, optical flow, depth estimation (DPT, Marigold), and semantic segmentation (Mask2Former, PHG-MAE).

Key Contributions

1. VRE Framework: The introduction of a modular pipeline that automates the generation of multi-modal datasets from raw video using pre-trained neural networks and procedural algorithms.
2. Engineering Architecture: Detailed strategies for scalability, including multi-GPU batching (both temporal and representational splitting) and real-time streaming architectures.
3. Dataset Extension: A case study demonstrating the extension of the Dronescapes dataset. Using VRE, the authors (and collaborators in reference [32]) expanded the dataset from ~23K to ~148K frames, adding up to 13 new modalities and 16 new UAV videos with minimal human intervention.
4. Deployment Analysis: Empirical validation of deploying real-time semantic segmentation and depth estimation models on commodity hardware (NVIDIA RTX 4050 laptop GPU) and remote cloud servers.

Experimental Results

• Batch Processing Efficiency: In experiments using the Dronescapes configuration (17 representations), a single GPU processed one frame in approximately 15.2 seconds. Using an 8-GPU setup, this dropped to an average of 2.6 seconds per frame, representing a throughput increase of 30–350× compared to manual annotation estimates.
• Hardware Performance: GPU acceleration consistently outperformed CPU processing, showing 2.5x to 5x speedups for neural network representations. Increasing batch sizes improved utilization, though memory constraints limited the maximum batch size.
• Scaling Efficiency: Multi-GPU scaling showed near-linear improvements up to 4 GPUs but plateaued at 8 GPUs (72% efficiency), attributed to bottlenecks in I/O, RAM, or CPU resources.
• Streaming and Latency: Real-time inference on a local consumer GPU achieved reliable frame rates for models like PHG-MAE-Distil and Depth DPT. However, remote processing via a network introduced significant latency, limiting frame rates to 2–3 FPS, leading the authors to conclude that local computation is preferable for real-time applications.

Significance and Claims
The paper claims that VRE successfully "closes the loop" between data and model design by providing a tool to optimize the relationship between model architecture and signal-rich training datasets. By automating the dependency management of modalities, VRE enables the creation of large-scale, multi-modal datasets that were previously infeasible to curate manually.

The authors emphasize that while they focus on UAV datasets, the procedure is applicable to any video domain, including autonomous driving and indoor robotics. They position VRE not just as a data generator, but as a critical infrastructure component for the emerging "Software 2.0" paradigm, where neural networks are integrated into larger systems to enable intelligent, autonomous actions. The work highlights the trade-offs between local and remote inference, suggesting that while cloud offloading is possible, local processing remains superior for latency-sensitive real-time tasks.