Covariance-Guided Resource Adaptive Learning for Efficient Edge Inference

1. Problem Statement

Edge devices (e.g., NVIDIA Jetson) face a critical challenge in Deep Learning (DL) inference: balancing throughput (frames per second) and power consumption under strict constraints.

• Complex Configuration Space: Modern edge hardware offers dozens of tunable parameters (CPU/GPU frequencies, memory clocks, active core counts, concurrency levels). These parameters interact non-linearly, meaning changing one affects the impact of others.
• Inefficiency of Current Solutions:
  • Manufacturer Presets: Fixed power modes (e.g., "Max Power," "Default") cover only a tiny fraction of the configuration space, often leading to excessive power draw or insufficient throughput.
  • Offline Profiling: Methods like ALERT require exhaustive offline profiling for every new model-device combination, which is time-consuming and energy-intensive.
  • Single-Metric Optimization: Existing learning-based approaches often optimize for throughput or power alone, failing to satisfy simultaneous dual-constraints (e.g., "Must run at 30 FPS while staying under 6W").
• The Core Challenge: Finding a configuration that satisfies both a throughput target and a power budget without exhaustive search or prior offline training.

2. Methodology: CORAL

CORAL (Covariance-Guided Resource Adaptive Learning) is an online optimization method that discovers near-optimal configurations in real-time without offline profiling.

A. Core Insight: Distance Covariance

Instead of assuming a linear relationship or requiring extensive training, CORAL uses Distance Covariance (and its normalized form, Distance Correlation, $dCor$).

• Unlike Pearson correlation, distance covariance captures arbitrary non-linear dependencies between variables.
• It measures the statistical dependency between hardware settings (e.g., CPU frequency) and performance metrics (throughput, power) regardless of the functional form of their relationship.

B. Algorithm Workflow

CORAL operates in an iterative loop (typically 10 iterations) consisting of three steps:

1. Reward Evaluation & Feasibility Check:

   • The system executes inference with the current configuration.
   • It checks if the configuration meets the Throughput Target ($\tau \ge \tau_{target}$) and Power Budget ($p \le p_{budget}$).
   • Reward Function:
     • If feasible: Reward = Efficiency ($\tau / p$).
     • If infeasible: Penalty = Negative Inverted Ratio ($-p / \tau$). Infeasible configurations are added to a "prohibited list" to avoid re-testing.

2. Correlation Analysis:

   • Using a sliding window of recent observations, CORAL computes the distance correlation ($\alpha$ for throughput, $\beta$ for power) between each hardware parameter and the metrics.
   • It identifies which parameters have the strongest influence on performance.

3. Configuration Search (Adaptive Step):

   • Step Size: The adjustment magnitude for each parameter is proportional to the distance between the current best and second-best configurations, scaled by the correlation strength ($\gamma = \max(\alpha, \beta)$). High-correlation parameters receive larger adjustments.
   • Search Direction:
     • If throughput is met but power is too high: The algorithm reduces values (e.g., lowers frequency) to save power.
     • If throughput is insufficient: The algorithm increases values to boost performance.
   • Heuristic: Once constraints are met, CORAL minimizes CPU frequency and maximizes concurrency to leverage CPU power savings while maintaining throughput via parallelism.

3. Key Contributions

1. Offline-Profiling-Free Online Optimization: CORAL is the first method to apply distance covariance to edge inference optimization, enabling real-time adaptation without the hours of offline profiling required by prior work.
2. Throughput-Power Co-Optimization: It explicitly formulates the problem as a dual-constraint optimization task, addressing the real-world need to satisfy both power limits and throughput targets simultaneously, a gap left by single-metric optimizers.
3. Statistical Non-Linearity Handling: By utilizing distance covariance, the method effectively navigates the complex, non-linear interactions between hardware parameters that linear models miss.
4. Comprehensive Evaluation: The method was validated on two distinct NVIDIA Jetson devices (Xavier NX and Orin Nano) across three object detection models ranging from lightweight (YOLOv5-N) to heavyweight (RetinaNet-ResNet50).

4. Experimental Results

The authors evaluated CORAL against four baselines: ORACLE (exhaustive search), ALERT (offline profiling + Kalman filter), ALERT-Online (random online exploration), and Manufacturer Presets.

• Single-Constraint Scenarios (Throughput only):

  • CORAL achieved 96% – 100% of the optimal throughput found by the exhaustive ORACLE search.
  • It significantly outperformed manufacturer presets (which achieved only 33%–60% of optimal throughput).

• Dual-Constraint Scenarios (Strict Power + Throughput):

  • Xavier NX (Target: 30 FPS, Limit: 6.5W): CORAL achieved 33 FPS at 5.5W, satisfying both constraints. In contrast, ALERT exceeded the power budget (8.5W), and ALERT-Online failed to find a valid configuration.
  • Orin Nano (Target: 60 FPS, Limit: 6W): CORAL achieved 77 FPS at 5.8W, exceeding the target while staying within budget.
  • Generalization: As model size increased (from YOLO to RetinaNet), baseline methods struggled increasingly, often failing to find any valid configuration. CORAL maintained near-ORACLE performance, demonstrating its robustness in tighter, more constrained search spaces.

5. Significance

CORAL represents a paradigm shift in edge AI resource management. By replacing heavy offline profiling and complex reinforcement learning training with a lightweight, statistically driven online search, it enables:

• Rapid Deployment: Operators can deploy new models on edge devices without waiting for days of profiling.
• Energy Efficiency: It prevents the "power waste" associated with default presets and the "performance failure" of under-provisioned settings.
• Scalability: The method scales effectively across different hardware generations and model complexities, making it a practical solution for dynamic, battery-constrained edge environments like drones and smart surveillance systems.