MMEdge: Accelerating On-device Multimodal Inference via Pipelined Sensing and Encoding

1. Problem Statement

Real-time multimodal inference on resource-constrained edge devices (e.g., autonomous vehicles, drones, mobile health devices) faces significant challenges due to the tight coupling between sensing dynamics and model execution, as well as complex inter-modality dependencies.

Key Challenges Identified:

• Sequential Bottlenecks: Traditional systems operate sequentially, waiting for all sensor data within a time window to be fully acquired before starting inference. This leads to accumulated delays and idle waiting times, especially when modalities have different processing speeds (e.g., video processing is slower than audio).
• Resource Contention: Sensing and inference often share computational resources on the same device. Optimizing one in isolation (e.g., increasing model complexity for accuracy) often degrades the other (increasing latency), and existing methods fail to coordinate them dynamically.
• Asynchronous Delays: Faster modalities are forced to wait for slower ones, leading to inefficient resource utilization and increased end-to-end latency.
• Accuracy-Latency Trade-off: Decomposing tasks to reduce latency often results in a loss of temporal context, causing significant accuracy degradation.

2. Methodology: MMEdge Framework

MMEdge proposes a unified system design that decomposes the entire inference process into fine-grained, pipelined sensing and encoding units. Instead of waiting for a complete time window, the system processes data segments (e.g., a single video frame or audio chunk) immediately upon arrival.

The framework consists of three core components:

A. Pipelined Sensing and Encoding Framework

• Decomposition: The inference task is split into a sequence of units corresponding to the smallest data segments.
• Parallelism: Feature encoding begins immediately as data arrives, overlapping the sensing interval with the encoding process. This eliminates idle waiting periods.
• Lightweight Encoders: Instead of heavy 3D convolutions or large temporal models, MMEdge uses lightweight encoders (e.g., 2D CNNs for individual frames) that operate on these fine-grained units.

B. Lightweight Temporal Aggregation Module

To counteract the accuracy loss caused by breaking temporal continuity, MMEdge introduces a module that captures rich temporal dynamics across the pipelined units without adding significant latency:

• Alternating Temporal Shift: Feature channels are shifted and replaced with features from neighboring units (past and future) to extend the temporal receptive field.
• Temporal Difference Features: The system extracts multi-scale difference features ($X_t - X_{t-1}$, etc.) to capture short- and long-term changes, enhancing temporal sensitivity.

C. Adaptive Multimodal Configuration Optimizer

This module dynamically selects the optimal sensing and model configurations for each modality based on real-time system conditions and input data complexity:

• Variables: It adjusts sensing granularity (e.g., frame rate, chunk size) and model complexity (e.g., ResNet-18 vs. ResNet-50).
• Mechanism: It uses an offline-trained accuracy predictor (based on modality consistency and complementarity metrics) and a pre-profiled latency lookup table.
• Goal: A greedy search algorithm selects the configuration that maximizes predicted accuracy while strictly adhering to a latency constraint ($T_{max}$).

D. Cross-Modal Speculative Skipping

To address modality asynchrony, MMEdge employs a gating classifier:

• Mechanism: When faster modalities (e.g., audio) provide sufficient information for a confident prediction, the system checks if waiting for slower modalities (e.g., video) is necessary.
• Action: If the gating classifier predicts high confidence, the system skips the remaining units of the slower modality, bypassing redundant data collection and computation.

3. Key Contributions

1. Systematic Analysis: An in-depth evaluation of end-to-end latency in on-device multimodal systems, identifying the inefficiencies of sequential processing and the potential of pipelined decomposition.
2. Novel Framework (MMEdge): A new inference framework that decomposes tasks into fine-grained units for pipelined execution, integrated with a temporal aggregation module to maintain accuracy.
3. Adaptive Optimization: The introduction of an adaptive multimodal configuration optimizer and a cross-modal speculative skipping mechanism to handle dynamic resource and data conditions.
4. Real-World Validation: Extensive evaluation on public datasets (LRW, NuScenes-Mini-QA) and a real-world UAV testbed for human tracking, demonstrating significant latency reduction without compromising accuracy.

4. Experimental Results

The authors evaluated MMEdge on an NVIDIA Jetson Xavier NX (public datasets) and a custom UAV testbed with an NVIDIA Jetson Orin Nano.

• Latency Reduction:
  • On the UAV testbed, MMEdge reduced end-to-end latency by 75.83% compared to baseline methods while maintaining high task accuracy.
  • On the LRW dataset, MMEdge achieved the lowest latency (~137 ms) among all methods, with less than a 3% accuracy drop compared to the best-performing baseline.
  • On the UAV dataset, it achieved a mean IoU of 70.47% (comparable to the blocking baseline) while reducing latency by over 80%.
• Robustness:
  • MMEdge maintained high accuracy across varying latency budgets (120ms–160ms) and resource constraints (CPU usage capped at 50%–100%).
  • It effectively adapted to input sample complexity, skipping 76.2% of "easy" samples but only 22.4% of "hard" samples, optimizing processing time based on difficulty.
• Overhead: The core design modules (temporal aggregation, optimizer, skipping) introduced negligible overhead (under 9ms latency and ~2W power), which was far outweighed by the latency savings they enabled.

5. Significance

MMEdge represents a paradigm shift in on-device multimodal AI by moving away from sequential, window-based processing to a continuous, pipelined approach. Its significance lies in:

• Bridging the Gap: It effectively bridges the gap between sensing and computation, treating them as a unified, coupled system rather than isolated stages.
• Real-Time Viability: By significantly reducing latency, it makes complex multimodal tasks (like autonomous driving or drone navigation) feasible on resource-constrained edge devices where real-time response is critical.
• Dynamic Adaptability: The ability to dynamically reconfigure sensing and model parameters based on runtime conditions ensures robust performance in unpredictable real-world environments (e.g., fluctuating power, varying lighting, or network congestion).
• Practical Deployment: The successful deployment on a real UAV testbed validates its applicability in safety-critical, time-sensitive scenarios.