🤖 machine learning

What changes after deployment? A survey on On-device Learning in TinyML

This paper surveys approximately 70 works on On-device Learning (ODL) for TinyML, categorizing them by distribution change regimes to analyze how different change types influence applicable scenarios, hardware choices, and solution structures, while highlighting the persistent gap between methodological benchmarks and real-world deployment.

Original authors: Massimo Pavan, Luca Pezzarossa, Fabrizio Pittorino, Manuel Roveri, Xenofon Fafoutis

Published 2026-06-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Massimo Pavan, Luca Pezzarossa, Fabrizio Pittorino, Manuel Roveri, Xenofon Fafoutis

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a tiny, super-smart robot assistant living inside a small device, like a smartwatch or a factory sensor. This robot is trained in a big, fancy school (a powerful computer) to recognize things—like your voice, a specific sound, or a type of movement. Once the training is done, the robot is shipped to its new home (the tiny device) to do its job.

The Problem: The "New Neighborhood" Shock
The paper argues that there's a big problem with this setup. The school where the robot learned was perfect and clean. But the real world is messy. Once the robot starts working, the environment changes. Maybe the background noise gets louder, your voice sounds different when you're tired, or a machine starts vibrating in a new way.

In the old days, the robot would just keep doing what it was taught, even if it was wrong. This is like a student who memorized the answer key for a test but gets confused when the teacher asks a slightly different question. The paper calls this a "distribution change." Basically, the data the robot sees after it's deployed is different from the data it saw in school.

The Solution: The Robot Learns on the Job
To fix this, the authors surveyed about 70 different projects where these tiny robots are taught to learn while they work. This is called "On-Device Learning." Instead of being static, the robot adapts to its new neighborhood in real-time.

However, the paper points out that not all "new neighborhoods" are the same. They categorize the changes into three distinct scenarios, like three different types of moving experiences:

The "One-Time Move" (Single-Change Regime):
Imagine you move to a new house, and the furniture is arranged differently, but then it stays that way forever. The robot just needs to learn this new layout once, quickly, and then it's done. It doesn't need to keep learning forever.
- Example: Teaching a smart speaker to recognize your specific voice or a new keyword you just said.
The "Constant Weather Shift" (Concept Drift Regime):
Imagine you are driving a car where the road conditions change every few minutes—rain, then sun, then fog, then ice. The robot has to constantly adjust its driving style, forgetting the old rules (like "drive fast on dry roads") and learning the new ones immediately. It's a never-ending cycle of adaptation.
- Example: A pressure sensor in a factory that slowly drifts due to heat or age and needs constant recalibration.
The "Ever-Expanding Library" (Continual Learning Regime):
Imagine you are a librarian who keeps getting new books every week, but you have a tiny shelf. You need to learn about the new books without throwing away the old ones you already know. If you forget the old books to make room for the new ones, that's called "catastrophic forgetting." The robot must keep adding new knowledge without losing the old.
- Example: A brain-computer interface that learns new commands from a user over time while remembering the old ones.

The Hardware Reality Check
The paper also looks at the "brains" (hardware) these robots live on. These aren't powerful laptops; they are tiny microchips with very limited memory and battery.

Tiny Chips (Ultra-Constrained): These are like pocket calculators. They can only handle the "One-Time Move" scenario. They are too weak to keep learning constantly.
Standard Chips (Standard MCUs): These are like basic smartphones. They can handle all three scenarios, but they struggle with the "Ever-Expanding Library" because they run out of memory trying to remember everything.
Super Chips (PULP Platforms): These are like mini-supercomputers. They are the only ones powerful enough to handle the "Ever-Expanding Library" scenario effectively, but they are expensive and hard to program.

The Big Disconnect
The authors found a strange gap in the research world:

The "Lab Scientists" mostly test their robots on perfect, messy image datasets (like photos of cats and dogs) that are too heavy for real tiny devices. They focus on making the learning algorithm perfect in a vacuum.
The "Real-World Builders" are working on actual problems like fixing sensors or recognizing gestures, but they often don't have standard tests to compare their work.

Because of this, the "Lab Scientists" are building fancy engines that don't fit in the "Real-World Builders'" cars. The paper suggests we need better ways to test these robots together, considering the whole system (the brain, the memory, and the learning method) rather than just one part.

In Summary
This paper is a map of the current landscape of "learning robots" on tiny devices. It tells us that to make these robots useful in the real world, we need to stop treating them as static machines. We need to design them to handle three specific types of changes (one-time, constant, or growing), match them with the right size of "brain" (hardware), and stop testing them in isolation. We need to build better tools to help engineers design the whole robot at once, not just piece by piece.

Technical Summary: What changes after deployment? A survey on On-device Learning in TinyML

1. Problem Statement

Machine learning models deployed on microcontroller-class devices (TinyML) face a fundamental challenge: post-deployment distribution change. Models trained offline on curated datasets often underperform when exposed to real-world conditions where data distributions shift after deployment. While On-Device Learning (ODL) addresses this by executing the learning process directly on the device, existing literature lacks a unified characterization of how distribution changes occur. Prior surveys have not explicitly distinguished between different types of distribution change regimes, leading to a conflation of solutions that operate under fundamentally different constraints and assumptions. Furthermore, a persistent gap exists between methodological benchmarks (often image-centric) and real-world deployment scenarios (often sensor-centric), hindering the transfer of algorithmic advances to practical applications.

2. Methodology

This paper presents a comprehensive survey of approximately 70 ODL works in the TinyML domain. The survey is structured around a primary organizing principle: the distribution change regime. The authors categorize the literature into three distinct regimes based on the nature of the data-generating process ( $P$ ) after deployment time $T$ :

Single-Change Regime: The distribution shifts exactly once at deployment ( $p_{T-1} \neq p_T$ ) and remains stable thereafter. The challenge is rapid, efficient adaptation to new conditions for the device's remaining life.
Concept Drift Regime: The distribution changes multiple times continuously and unpredictably ( $p_{t-1} \neq p_t$ ). The focus is on adapting quickly to new conditions while discarding or downweighting irrelevant past knowledge.
Continual Learning Regime: New concepts are introduced incrementally over time. The solution must accumulate new knowledge without forgetting previously learned information (addressing catastrophic forgetting). While formally similar to concept drift in terms of distribution change, the assumption regarding the relevance of past knowledge and evaluation criteria differs significantly.

For each regime, the survey analyzes the literature through three complementary lenses:

Application Lens: The real-world contexts and tasks addressed (e.g., keyword spotting, sensor calibration, brain-machine interfaces).
Hardware Lens: The target devices, ranging from ultra-constrained processors (e.g., Cortex-M0, ISPU) to standard MCUs (Cortex-M4/M7) and parallel ultra-low-power (PULP) platforms.
Solution Lens: The technical composition of ODL solutions, decomposed into four components:
- $R(\cdot)$ : Dimensionality reduction (e.g., frozen CNNs, statistical feature extractors).
- $B$ : Data management (buffering strategies, admission/trigger conditions).
- $f_t(\cdot)$ : The ML algorithm (e.g., DNNs, SVMs, KNN).
- $L$ : The learning mechanism (e.g., backpropagation, forward-only updates, prototype generation).

The survey also formalizes the ODL problem, defining constraints on memory ( $M_D$ ), clock frequency ( $F_D$ ), power, and execution time ( $E$ ), and establishes evaluation protocols specific to each regime (e.g., prequential evaluation for concept drift, longitudinal evaluation for continual learning).

3. Key Contributions

Regime-Centric Taxonomy: This is the first survey to organize the entire ODL literature around the distribution change regime (Single-Change, Concept Drift, Continual Learning), arguing that solutions designed for one regime are not directly comparable to those in another due to differing assumptions about past knowledge and adaptation goals.
Comprehensive Scope: The survey covers ~70 works, significantly expanding upon prior efforts (which surveyed ~10 works) and capturing the substantial body of research published post-2023.
Multi-Lens Analysis: By analyzing works through application, hardware, and solution lenses simultaneously, the survey reveals cross-cutting patterns, such as the correlation between hardware capability and regime complexity, and the misalignment between method-oriented and application-oriented research.
Identification of Structural Gaps: The paper identifies a disconnect between methodological contributions (often focused on backpropagation optimization on standard benchmarks) and application-oriented works (which prioritize label efficiency and real-world constraints). It highlights that methodological works rarely address the scarcity of labeled data or the full pipeline constraints (memory, latency, power) simultaneously.

4. Key Results and Findings

Application Findings

Misalignment of Tasks: Method-oriented works predominantly use image classification benchmarks (CIFAR, MNIST) which are often too computationally demanding for real-world TinyML. Conversely, application-oriented works focus on tasks like Keyword Spotting (KWS), Human Activity Recognition (HAR), and sensor calibration, which lack standardized benchmarks.
Label Acquisition Strategies: Successful application-oriented works treat label acquisition as a first-class design requirement. Strategies include few-shot learning, self-supervised learning (using auxiliary sensors), unsupervised learning (for anomaly detection), and implicit labeling via structured stimuli.
Regime Distribution: Application-oriented works are heavily concentrated in the Single-Change regime (e.g., KWS personalization, sensor calibration). The Concept Drift and Continual Learning regimes are dominated by method-oriented works, with very few real-world application deployments reported.

Hardware Findings

Hardware-Regime Correlation: There is a monotonic relationship between hardware capability and regime complexity.
- Ultra-constrained processors (Cortex-M0, ISPU) appear almost exclusively in the Single-Change regime due to tight memory/compute limits.
- Standard MCUs (Cortex-M4/M7) span all three regimes but are underrepresented in Continual Learning due to memory constraints for replay buffers.
- PULP platforms (multi-core, e.g., GAP9) dominate the Continual Learning regime, as they are the only class currently capable of supporting the high memory and computational demands of continual learning algorithms.
Reporting Gaps: Hardware reporting is inconsistent. Metrics like peak RAM and energy are often reported per-instance, per-batch, or per-epoch without standardization, making cross-work comparison difficult.

Solution Findings

Component Patterns:
- Dimensionality Reduction ( $R$ ): Frozen pre-trained CNNs are the dominant choice for image/audio inputs, while identity functions or statistical extractors are used for low-dimensional sensor data.
- Data Management ( $B$ ): Fully online operation ( $|\Delta|=1$ ) dominates Single-Change and Concept Drift regimes. Explicit buffering is structurally necessary in Continual Learning to prevent catastrophic forgetting.
- Learning Mechanism ( $L$ ): Full backpropagation is the standard for Continual Learning (supported by replay buffers). In Single-Change and Concept Drift, lighter alternatives (forward-only updates, prototype-based classifiers, KNN) are prevalent.
Optimization Gap: Methodological works heavily optimize backpropagation (sparsification, quantization), yet these optimizations are rarely adopted in application-oriented deployments. This suggests a disconnect where algorithmic improvements are not validated against the full pipeline constraints (e.g., labeling strategy, data buffering costs).

5. Significance and Claims

The paper claims that the distribution change regime is the critical, yet previously overlooked, factor that determines the feasibility and design of ODL solutions. By explicitly distinguishing between Single-Change, Concept Drift, and Continual Learning, the survey provides a framework for meaningful comparison that prior literature lacked.

The authors modestly claim that their work highlights a structural gap in the field: algorithmic progress and application development are advancing in parallel with limited cross-fertilization. They argue that the field requires:

Standardized Evaluation Frameworks: Benchmarks that fix hardware budgets, solution components, and deployment constraints (including label scarcity) to enable fair comparison.
End-to-End Design Tools: Software frameworks that treat the entire ODL pipeline ( $R, B, f, L$ ) as jointly optimizable objects, rather than optimizing components in isolation.
Hardware for ODL: A shift from adapting cloud-training architectures to designing hardware specifically for ODL constraints (scarce labels, no on-device validation, tight power budgets).

The survey concludes that while ODL widens the scope of TinyML applications, realizing this vision requires addressing the misalignment between methodological benchmarks and real-world deployment scenarios, particularly regarding label efficiency and the specific demands of non-stationary learning regimes.