⚡ electrical engineering

TinyDéjàVu: Smaller RAM and Faster Inference with Neural Networks on MCUs for Sensor Data Streams

This paper introduces TinyDéjàVu, an open-source framework and set of algorithms that optimize data flows for neural network inference on resource-constrained microcontrollers, achieving up to a 90% reduction in RAM usage with comparable latency to prior methods for processing sensor data streams.

Original authors: Zhaolan Huang, Emmanuel Baccelli

Published 2026-05-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhaolan Huang, Emmanuel Baccelli

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a tiny, battery-powered robot (a microcontroller) that lives in a forest. Its job is to listen to birds singing and instantly figure out what kind of bird it is. The problem? This robot has a very small brain (limited memory) and a tiny battery. If it tries to listen to a long song by memorizing the entire song every time a new note arrives, it will run out of memory and die (the battery will drain).

This paper introduces a new software framework called TinyDéjàVu (a play on the feeling of having experienced something before) that helps these tiny robots listen to long streams of data without getting overwhelmed.

Here is how it works, using simple analogies:

1. The Problem: The "Re-Reading" Trap

Normally, when these robots analyze sound, they look at the data in chunks called "sliding windows." Imagine reading a book by looking at a small window that moves one word at a time.

The Old Way: Every time the window moves one word to the right, the robot stops, throws away the old view, and re-reads the entire new window from scratch. If the window is 100 words long and the robot moves it 90 times, it ends up re-reading the same 90 words over and over again. This wastes massive amounts of energy and memory.

2. The Solution: The "State-Space" Memory Trick

TinyDéjàVu changes the rules. Instead of re-reading the whole window, it uses a mathematical trick called a State-Space Model (SSM).

The Analogy: Imagine you are watching a movie. Instead of rewinding the whole film every time a new scene starts, you just remember the current state of the story. When a new frame arrives, you only update your memory with that one new piece of information.
How it helps: TinyDéjàVu turns complex math operations (like convolution) into this "update only" style. It realizes that 90% of the data in a new window is identical to the previous one. So, it doesn't store the whole window; it only stores the tiny "state" needed to update the result.

3. The "Déjà Vu" Effect

The name comes from the fact that the system recognizes that it has "seen" most of this data before.

The Metaphor: Think of a conveyor belt carrying boxes. The old method required a worker to stop the belt, unpack every box in the current view, and then move the view forward. TinyDéjàVu is like a worker who only picks up the new box that just arrived on the belt and updates their mental list. They ignore the boxes that haven't changed.

4. The Results: Smaller Brains, Faster Speed

The authors tested this on real microcontrollers (like the ones found in smart sensors) and compared it to previous methods.

Memory Savings: They found that TinyDéjàVu could reduce the memory needed by up to 90%. In their bird-singing example, a model that needed 75 kB of memory (too big for some tiny chips) was shrunk down to just 1 kB.
Speed: Because it stops doing the same math over and over, it became up to 20 times faster when processing overlapping data streams.
Accuracy: Despite doing less work, it didn't make mistakes. The answers were just as accurate as the old, heavy methods.

5. Real-World Proof

The team didn't just test this in a computer simulation. They actually deployed it on sensors in the UK to listen for Corn Bunting birds.

They successfully ran complex AI models on tiny, battery-powered devices that previously couldn't handle the task because they ran out of memory.
This allowed the devices to stay in the wild longer without needing battery changes.

Summary

TinyDéjàVu is like giving a tiny, low-power robot a "short-term memory" that is incredibly efficient. Instead of hoarding every piece of data it sees, it only keeps what is strictly necessary to update its understanding. This allows small devices to listen to long, continuous streams of data (like bird songs or heartbeats) without running out of battery or memory.

Technical Summary: TinyD´ej`aVu

Problem Statement

The deployment of neural networks for continuous inference on sensor data time-series in Internet of Things (IoT) devices faces severe resource constraints. These devices, typically powered by batteries and running on microcontrollers (MCUs) with limited RAM (e.g., 128 kB), must perform real-time processing on data streams. A common approach for time-series modeling involves using overlapping sliding windows to capture temporal context. However, standard inference methods recompute the entire model for every new window, leading to significant redundancy. This redundancy results in excessive Random Access Memory (RAM) usage for buffering intermediate activations and high computational latency, often rendering complex models infeasible for deployment on resource-constrained hardware.

Methodology

The paper introduces TinyD´ej`aVu, a framework that optimizes neural network inference for sensor data streams by transforming temporal operators into State-Space Models (SSMs). The core methodology involves three main components:

SSM Equivalence and Graph Transformation:
The authors formalize the equivalence between common temporal operators (such as 1-D convolutions, pooling, and dense layers with temporal mixing) and SSMs. By representing these operators as recursive state updates ( $h_t = A h_{t-1} + B x_t$ ), the framework eliminates the need to store large input windows or intermediate feature maps. Instead, only a small, fixed-size hidden state vector is maintained. The framework partitions the neural network compute graph into two regions:
- SSM-subgraph: Contains local or causal temporal operators, which are converted into efficient SSM equivalents.
- GTA-subgraph (Global Temporal Aggregator): Begins at the first operator with a global receptive field (e.g., a dense layer flattening time steps or an attention mechanism). This section handles the aggregation of the processed stream.
Deep Sliding Window Optimization:
For overlapping sliding windows, TinyD´ej`aVu eliminates redundant computation. Instead of reprocessing the entire window, the system updates only the hidden states corresponding to the new data points introduced by the stride. The Global Temporal Aggregator (GTA) is also transformed to accept these updated hidden states, ensuring that computation is triggered only when necessary new features are available.
Implementation and Optimization:
- Circular Buffers: The SSMs are implemented using circular buffers to manage hidden states, avoiding dynamic memory allocation.
- Global Pooling: A specialized algorithm converts global pooling operations into a cascade of smaller SSMs, reducing RAM complexity from $O(N)$ to $O(\lceil N/s \rceil)$ , where $N$ is the window size and $s$ is the stride.
- Mixed Precision: The framework supports BF16 (Brain Floating Point) precision for storing hidden states, offering a 2x reduction in RAM usage compared to FP32 with minimal impact on accuracy for many tasks.
- Toolchain: The system is built on top of PyTorch for analysis and microTVM for graph rewriting and C-code generation, targeting MCUs like the Arm Cortex-M series.

Key Contributions

Novel Framework: Introduction of TinyD´ej`aVu, which automates the conversion of temporal operators in neural networks into SSMs for streaming inference.
Algorithmic Innovation: Development of algorithms to handle overlapping sliding windows by eliminating redundant computations and a specific optimization for global pooling to drastically reduce RAM footprints.
Open Source Implementation: Publication of the full implementation, including the graph transformation logic and benchmarks, to ensure reproducibility.
Comprehensive Benchmarking: Evaluation across diverse model architectures (CNNs, TCNs, Transformers, WaveNet variants) on common 32-bit MCU hardware (Arm Cortex-M).

Results

The authors present extensive experimental results comparing TinyD´ej`aVu against vanilla models and prior work (StreamiNNC):

RAM Reduction: TinyD´ej`aVu achieves up to a 90% reduction in peak RAM usage compared to vanilla models. In specific cases like TinyWaveNet, the reduction is substantial enough to allow models that previously required ~180 MB of RAM to run on devices with <350 kB RAM.
Latency Improvements: On streaming inputs with overlapping windows, the framework demonstrates significant latency reductions. On a host PC, it achieves up to 6x speedup in the streaming stage compared to untransformed models. On an STM32F7 MCU, latency decreases linearly as the overlap rate increases, with speedups of up to 20x observed in specific configurations.
Accuracy: The framework maintains numerical correctness, with deviations between the original and transformed models below $10^{-8}$ . Task-level accuracy remains comparable to the vanilla models in FP32. When using BF16, a slight accuracy drop is observed (e.g., 1-3% RMSE increase), which the authors suggest can be mitigated by retraining the model under BF16 precision.
Real-World Validation: A use case involving bird sound detection (bioacoustic monitoring) on nRF52840 sensors demonstrated a 75x reduction in RAM usage (from ~75 kB to ~1 kB) and a 60% reduction in inference latency compared to the previous state-of-the-art (TinyChirp).

Significance

The paper claims that TinyD´ej`aVu addresses a critical bottleneck in TinyML: the ability to run complex, long-sequence time-series models on extremely resource-constrained devices. By decoupling RAM usage from input sequence length through SSM-based streaming, the framework enables the deployment of "always-on" intelligence on battery-powered sensors without the need for specialized hardware accelerators. The authors emphasize that their approach is orthogonal to other optimization techniques (like quantization or pruning), meaning it can be combined with them for even greater efficiency. The work provides a practical, open-source pathway to making long-sequence sensor data processing feasible on standard, low-cost microcontrollers.