🤖 machine learning

Hierarchical adaptive control for real-time dynamic inference at the edge

This paper proposes a hierarchical adaptive control architecture that co-optimizes model and system decisions across global and local levels to enable dynamic, energy-efficient, and low-latency machine learning inference on edge devices under varying data distributions and resource constraints, achieving significant performance gains over static baselines.

Original authors: Francesco Daghero, Mahyar Tourchi Moghaddam, Mikkel Baun Kjærgaard

Published 2026-04-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Francesco Daghero, Mahyar Tourchi Moghaddam, Mikkel Baun Kjærgaard

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a busy factory assembly line that sorts packages. In the past, every package had to go through the same heavy-duty, slow, and energy-hungry machine to be checked, no matter how obvious the package was. If a package was clearly a "shoe," it still had to wait in line for the complex machine to confirm it. This wasted time and electricity.

This paper proposes a smarter way to run that factory, specifically for small computers (called "edge devices") that run on batteries and need to work instantly, like the ones inside robots or sensors.

Here is the breakdown of their solution using simple analogies:

The Problem: The "One-Size-Fits-All" Bottleneck

Standard AI models are like a single, massive security guard who checks every single item in detail. This is accurate, but slow and tiring.

Dynamic Inference is a newer idea: It's like having a line of guards. A quick, cheap guard checks the easy items first. If they are confident, they let the item pass immediately. Only if the item is confusing does it get sent to the slow, expensive "Big Boss" guard at the end.
The Catch: This system usually needs to be tuned perfectly before it starts. But in the real world, things change. Maybe the factory suddenly starts getting mostly "shoes" instead of "books," or the factory's power supply gets weak (battery dying), or the factory gets hot (overheating). If the system isn't tuned for these changes, it might get stuck sending easy items to the Big Boss, wasting energy, or it might get so slow that it misses its deadlines.

The Solution: A Two-Level Management Team

The authors built a "hierarchical" (two-level) system to fix this. Think of it as a factory with a Site Manager and a Floor Supervisor.

1. The Site Manager (Global Controller)

This is the "brain" that sets up the factory.

What it does: Before the factory opens, it looks at the rules (how much battery we have, how fast we need to work) and picks the best lineup of guards. It decides which "Specialized Predictors" (SPs)—the quick, cheap guards—should be on duty and in what order.
The Safety Net: It makes sure that even in the worst-case scenario (where every single item is confusing), the factory never runs too slow. It sets a hard limit so the system never fails its timing deadlines.
Updates: If the factory manager gets a report that the types of packages are changing drastically, the Site Manager can send a new specialized guard to the floor to help out, without having to shut down the whole factory to rebuild it.

2. The Floor Supervisor (Local Controller)

This is the "eyes" on the ground, running directly on the device.

What it does: It watches what's happening right now. Is the battery low? Is the machine getting hot? Are we suddenly seeing 100 "shoes" in a row?
The Reaction: If the Floor Supervisor sees that a specific type of package is rare, it can tell the system to turn off the guard specialized for that package to save energy. If a new type of package becomes common, it can turn on the right guard.
The Emergency Brake: If the system is running too slow and might miss a deadline, the Floor Supervisor can instantly kick out the last few guards in the line to speed things up, ensuring the system stays safe and fast.

How It Works in Practice

The system uses a "cascade" (a line) of simple, one-task experts.

Example: Instead of one giant model trying to guess 10 different things, you have 10 tiny models, each looking for just one thing (e.g., "Is this a cat?").
The Flow: An image comes in. The first tiny model checks: "Is this a cat?" If it's 99% sure, it says "Yes!" and stops. The job is done. If it's unsure, it passes the image to the next tiny model. If none of the experts are sure, the image goes to the "Big Boss" (the full, heavy model) for a final answer.

The Results: What Did They Find?

The authors tested this on two different image datasets (one for soil types, one for general objects like CIFAR-10) using a Raspberry Pi (a small, cheap computer).

Speed: Their system was up to 2.45 times faster than the old, static way of doing things.
Energy: It used up to 2.86 times less energy. This is huge for battery-powered devices.
Accuracy: They only lost a tiny bit of accuracy (less than 4%) compared to the heavy, slow models.
Adaptability: When the "types of packages" changed (data drift), their system automatically rearranged the line of guards to stay efficient. The old "dynamic" systems got confused and slowed down, but their system kept working smoothly.

The Bottom Line

This paper presents a smart, two-layer control system that lets small, battery-powered computers run AI efficiently. It combines a "planner" who sets the rules and a "supervisor" who adjusts the rules on the fly. This ensures the computer stays fast and saves battery, even when the world around it changes unexpectedly, without needing to be constantly connected to a powerful cloud server.

1. Problem Statement

Industrial edge systems increasingly rely on Machine Learning (ML) for real-time perception and control. These systems operate under strict constraints regarding latency, energy, and memory, often on heterogeneous, battery-powered devices with intermittent cloud connectivity.

While Dynamic Inference (reconfiguring computational graphs at runtime based on input complexity) promises energy efficiency, current implementations face two critical challenges:

Static Tuning: Existing methods tune hyperparameters (e.g., exit thresholds) on a calibration dataset that assumes a stable test-time distribution. In real-world scenarios, data drift (changing class frequencies) and resource variability (thermal throttling, battery drain) render these static configurations suboptimal or even worse than static models.
Latency Violations: Dynamic inference increases worst-case latency (as it may require running multiple models). If the system cannot adapt to resource constraints or distribution shifts, it risks violating real-time deadlines.

Recalibrating models is time-consuming and introduces unacceptable delays. Therefore, there is a need for a system that can autonomously adapt to distribution shifts and resource changes without requiring full redeployment.

2. Methodology

The authors propose a two-tier hierarchical adaptive architecture that co-optimizes model and system decisions. The system consists of a Global Controller (GC) and a Local Controller (LC).

A. Core Mechanism: Budgeted SP-Cascade

Instead of a fixed cascade or early-exit networks, the system uses a Specialized Predictor (SP) Cascade:

Decomposition: A multiclass problem is decomposed into $N$ binary "one-vs-rest" classifiers (SPs).
Fallback: A generalist multiclass model (FB) acts as the final fallback.
Inference Logic: An input traverses the cascade of SPs. Each SP uses two thresholds ( $thr^-$ $t h r^{-}$ and $thr^+$ $t h r^{+}$ ) to decide:
- Accept: If confidence is high, exit early.
- Reject: If confident the input is not the class, move to the next SP.
- Undecided: If confidence is low (near 0.5), invoke the heavy FB model.
Benefit: This allows the system to skip the heavy FB model for "easy" samples while maintaining a hard worst-case latency bound (the full cascade).

B. Global Controller (GC)

The GC operates at a lower frequency (e.g., during deployment or periodic updates) and handles coarse-grained configuration:

Task Scheduling: Uses a beam search to select an optimal subset and ordering of SPs for each node. It minimizes expected MACs (computational cost) while satisfying memory and worst-case latency constraints.
Online Update: Monitors node-level signals (battery, temperature) and task-level signals (predicted class frequency). If data drift occurs, the GC can deploy new SPs (e.g., for a newly frequent class) or update the configuration without a full system redeployment.

C. Local Controller (LC)

The LC operates at the inference-time granularity on the edge device:

Resource Monitoring: Tracks CPU, memory, and temperature.
Distribution Tracking: Computes an Exponential Moving Average (EMA) of predicted classes to detect drift.
Adaptive Actions:
- Enable/Disable: Activates or deactivates locally available SPs based on class frequency thresholds.
- Reordering: Sorts the active SPs so that those for the most frequent classes are evaluated first, maximizing early exits.
Safety Check: Enforces a strict latency budget. If a sample's inference time threatens to violate the deadline ( $L_k > D_t$ ), the LC immediately reduces the active cascade (disabling the last SPs) to ensure the worst-case latency remains within bounds.

3. Key Contributions

Budgeted SP-Cascade Formulation: A novel dynamic inference approach that explicitly preserves worst-case latency constraints while optimizing average latency and energy.
Hierarchical Controller: A dual-loop system (GC + LC) that maintains efficiency under data distribution mismatch and runtime resource variability, decoupling coarse configuration from fine-grained selection.
Experimental Validation: An evaluation on embedded hardware (Raspberry Pi 4B) demonstrating significant efficiency gains with minimal accuracy loss.

4. Results

The system was evaluated on two vision datasets: Soil Classification Dataset (SCD) and CIFAR-10, under three scenarios: Base (stationary), Mismatch Minor, and Mismatch Major (severe distribution shift).

Performance Metrics: Compared to static baselines and non-adaptive dynamic inference:
- Latency: Achieved up to 2.45x reduction in average latency (SCD) and 1.18x (CIFAR-10).
- Energy: Achieved up to 2.86x reduction in energy consumption (SCD).
- Accuracy: Maintained accuracy within <4% of the static baseline.
Adaptation Effectiveness:
- In Mismatch Major scenarios (where a class not in the initial cascade becomes dominant), non-adaptive dynamic inference degraded significantly.
- The proposed framework successfully detected the shift, reordered the cascade, and deployed new SPs via the GC, recovering efficiency.
- In the SCD Mismatch Major scenario, the framework reduced total energy usage by 58% compared to static solutions and 39% compared to non-adaptive dynamic baselines.

5. Significance

This work addresses a critical gap in Edge AI: the inability of current dynamic inference systems to handle real-world volatility.

Robustness: By separating the "what" (model selection) from the "how" (runtime adaptation), the system ensures timing predictability even when data distributions change or hardware resources fluctuate.
Efficiency: It proves that dynamic inference can be more efficient than static models even under adverse conditions, provided the system can adapt its configuration online.
Practicality: The architecture is designed for heterogeneous edge nodes with intermittent connectivity, allowing local adaptation without constant cloud reliance, making it suitable for industrial IoT and cyber-physical systems.