Empowering On-Device Model Adaptation with an Edge AI Inference Accelerator

Technical Summary: Empowering On-Device Model Adaptation with an Edge AI Inference Accelerator

Problem Statement
On-device model adaptation is critical for enabling lifelong personalization on resource-constrained hardware, allowing pre-trained models to adjust to new data distributions or user-specific patterns without relying on cloud connectivity. However, the computational, power, and memory limitations of edge devices make end-to-end backpropagation for modern deep neural networks impractical. While specialized co-processors exist for inference, there is a lack of research exploiting these inference accelerators to accelerate the training process itself. Conventional edge devices often lack the capacity for full training, creating a bottleneck for advanced adaptive solutions.

Methodology
The authors propose a heterogeneous adaptation pipeline that repurposes a commercial edge AI inference accelerator, the Hailo-8L, to handle feature extraction during on-device training. The core innovation lies in partitioning the computational graph to leverage the strengths of both the accelerator and the host CPU:

1. Graph Partitioning: The model is split into a frozen backbone (feature extractor) and a trainable classification head.
   • Backbone: Pre-trained weights are quantized to INT8 and offloaded to the Hailo-8L accelerator. This component generates feature maps during the forward pass but is excluded from the backward pass.
   • Head: A lightweight FP32 classification head runs on the host CPU (Raspberry Pi 5). Only this head is fine-tuned.
2. Truncated Backpropagation: Backpropagation is mathematically truncated at the accelerator boundary. Gradients are computed and parameters updated only for the adaptive top layers on the CPU. The accelerator-generated feature tensor serves as the input to the host-side trainable head.
3. Quantization and Restoration: To mitigate accuracy loss from INT8 quantization of the backbone, the study evaluates several post-training quantization restoration strategies using the Hailo Dataflow Compiler. These include:
   • Channel Equalization (Eq.)
   • Iterative Bias Correction (IBC)
   • Knowledge-Distillation Fine-tuning (FT)
   • AdaRound (AR)
     The pipeline converts PyTorch models to ONNX, applies these strategies to generate a hardware delegate (HailoOp), and utilizes the ONNX Runtime Training API to execute the hybrid graph.

Key Contributions

1. Heterogeneous Training Pipeline: A novel ONNX Runtime-based workflow that repurposes an INT8 inference accelerator for frozen-backbone feature extraction, enabling on-device head fine-tuning.
2. Systematic Evaluation of Restoration: A comprehensive analysis of how different quantization restoration strategies impact downstream adaptation performance (accuracy, throughput, and energy) in a training context.
3. System-Level Benchmarking: A comparative evaluation against CPU-only (Raspberry Pi 5) and edge-GPU (NVIDIA Jetson Orin Nano) baselines across multiple architectures (ResNet18, EfficientNet-Lite 4, MobileNetV3 Large, FastViT-SA12) and datasets (CIFAR-100, Oxford-IIIT Pet).

Results

• Training Speed: The proposed pipeline significantly reduces wall-clock training time. For ResNet18, it achieves up to 15.4× faster training compared to the Raspberry Pi 5 CPU baseline. In favorable settings, it also outperforms the dedicated edge GPU (Jetson Orin Nano) in terms of time-per-sample (e.g., 6.04 ms/sample vs. 13.70 ms/sample for ResNet18).
• Energy Efficiency: The heterogeneous setup consistently reduces energy consumption per sample, using approximately 3.3× less energy than the edge GPU baseline for ResNet18.
• Throughput: The system demonstrates strong scalability for small batch sizes (1 and 4), matching or exceeding the throughput of the edge GPU while consuming less power. However, throughput gains vary by architecture; models with computationally intensive heads (e.g., MobileNetV3) see more modest speedups as the host-side FP32 backward pass becomes the bottleneck.
• Accuracy and Quantization: The study finds that quantization resilience varies by architecture. ResNet18 and EfficientNet-Lite are resilient to simple restoration methods (Eq., IBC). In contrast, architectures using Hard-Swish or self-attention (MobileNetV3, FastViT-SA12) are more sensitive to INT8 degradation and require advanced restoration strategies (FT or AdaRound) to maintain accuracy close to FP32 baselines.

Significance and Claims
The paper claims that repurposing inference-oriented edge accelerators offers a practical approach to efficient on-device adaptation. By strictly partitioning the execution graph, the method overcomes key computational bottlenecks, enabling frequent, small-scale model updates under strict energy and latency constraints.

The authors position this work as a solution for edge scenarios where the core perceptual backbone remains fixed while label semantics evolve (e.g., human activity recognition personalization, local environmental sensing, or industrial defect detection). They emphasize that performance restoration techniques, typically used for inference, are equally crucial in training contexts to stabilize upstream gradients and mitigate accuracy loss.

The study acknowledges limitations, noting that effectiveness depends on architecture-specific robustness to INT8 degradation and that host-side overheads (data transfer, quantization/dequantization) can limit speedups for models with complex heads. Future work is proposed to extend the framework to parameter-efficient fine-tuning paradigms and dynamic continual learning scenarios.