🤖 machine learning

Split CNN Inference on Networked Microcontrollers

This paper presents a fine-grained split inference system for networked microcontrollers that overcomes memory constraints by partitioning CNN models at the kernel and neuron level across multiple devices, enabling the execution of otherwise infeasible models while maintaining practical latency.

Original authors: Junyu Lu, Shashwath Suresh, Hao Liu, Qi Hong, Qing Wang

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Junyu Lu, Shashwath Suresh, Hao Liu, Qi Hong, Qing Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very difficult puzzle to solve, but you only have a tiny desk to work on. The puzzle is a "Deep Neural Network" (a complex AI brain), and your desk is a Microcontroller (MCU)—a small, cheap computer chip found in everything from toasters to smart sensors.

The problem is that the puzzle is too big for your tiny desk. Even if the puzzle pieces (the model's "weights") fit in your pocket, the act of solving it requires you to spread out a massive amount of temporary notes (called "intermediate activations") on the desk. Your desk is too small; the notes spill over, and the task fails.

This paper proposes a clever solution: Don't try to solve the whole puzzle on one desk. Instead, get a team of friends, each with their own tiny desk, and solve it together.

Here is how the authors made this work, explained simply:

1. The Old Way vs. The New Way

The Old Way (Coarse Splitting): Imagine trying to split the puzzle by giving one friend the top half and another friend the bottom half. But even the "top half" is still too big for one tiny desk. This approach fails because the pieces are still too chunky.
The New Way (Fine-Grained Splitting): The authors realized they needed to cut the puzzle into much smaller, bite-sized pieces. Instead of splitting by "layers" (big sections), they split by individual neurons (the tiny dots inside the network).
- Analogy: Imagine a massive mural. Instead of giving one artist a whole wall to paint, you give them just a single square inch of the wall. You give another artist a different square inch. Together, they paint the whole mural, but no single artist ever has to hold the entire painting at once.

2. The "Team Captain" (The Coordinator)

Since the friends (MCUs) are working on tiny fragments, they need a manager to keep things organized.

The Coordinator: This is a central device (like a PC or a dedicated chip) that acts as the team captain.
What it does: It doesn't do the heavy lifting. Instead, it holds the map. It tells Friend A, "You work on these specific pixels," and "Send your result to Friend B." It routes the temporary notes between the friends so everyone has exactly what they need to do their tiny part of the job.

3. The "Smart Assignment" (Resource-Aware)

Not all friends are the same. Some have faster desks (faster processors), some have bigger pockets (more memory), and some are slower to pass notes (slower network).

The Problem: If you give the same amount of work to a slow friend and a fast friend, the fast friend will just sit there waiting, wasting time.
The Solution: The authors created a "Rating System." They measure how fast and capable each MCU is.
- The Analogy: Think of a relay race. You don't give the same distance to a sprinter and a walker. You give the sprinter a longer leg of the race and the walker a shorter one, so everyone finishes at roughly the same time. The system automatically calculates who should do how much work to keep the team moving efficiently.

4. The Results

The team tested this on a real setup using 8 small computers (Teensy 4.1 MCUs) and a popular AI model called MobileNetV2.

The Outcome: A single computer couldn't run this AI model at all because it ran out of memory. But when they split the work across 8 computers, the model ran successfully.
The Trade-off: The total time to solve the puzzle got a little slower because the friends had to spend time passing notes back and forth. However, the memory usage on each individual computer dropped drastically, making the impossible task possible.

Summary

This paper shows that by breaking a complex AI task into microscopic pieces and distributing them across a network of small, cheap computers, we can run powerful AI on devices that are too weak to handle it alone. It's like turning a single, overloaded elephant into a team of ants that can carry a crumb together.

Technical Summary: Split CNN Inference on Networked Microcontrollers

Problem Statement

Deploying Deep Learning (DL) models on Microcontroller Units (MCUs) is severely constrained by limited memory resources, specifically Random Access Memory (RAM). While TinyML techniques such as quantization, pruning, and neural architecture search effectively reduce model parameters and computation, they often fail in practice due to peak RAM usage during inference. This peak usage is dominated by intermediate activations and temporary buffers, which can exceed the available RAM of a single MCU even when model weights fit within flash memory. Consequently, many Convolutional Neural Network (CNN) models remain infeasible for standalone MCUs.

Existing split inference strategies, designed for GPUs or edge-cloud scenarios, rely on coarse-grained partitioning at layer boundaries. These approaches fail on MCUs because even a single layer's intermediate feature maps or kernel weights can exceed the memory capacity of an individual device. Furthermore, naive distribution across heterogeneous MCUs leads to load imbalance and excessive communication overhead, degrading end-to-end latency.

Methodology

The authors propose a fine-grained split inference system that enables collaborative CNN execution across a network of MCUs. The system operates under a coordinator-worker architecture and employs three core mechanisms:

1. Fine-Grained Model Partitioning

Unlike traditional layer-wise splitting, this system partitions computation at sub-layer granularity:

Convolutional Layers: The output feature map is split into fine-grained patches. Convolutional kernels are assigned to specific workers based on the output neurons they compute. This allows different MCUs to process different subsets of the output channels or spatial regions.
Linear Layers: The weight matrix is partitioned column-wise, where each column corresponds to an output neuron. Workers are assigned specific columns, enabling independent computation of output neurons.
Model Reinterpretation: A pre-processing pipeline reinterprets pre-trained models to explicitly extract neuron-level dependencies, receptive fields, and tensor dimensions. This metadata is serialized and used to generate weight fragments and routing maps.

2. Resource-Aware Workload Allocation

To address hardware heterogeneity (variations in clock frequency, memory size, and communication bandwidth), the system assigns a capability rating ( $R_i$ ) to each MCU. This rating is derived from a model of execution time that accounts for:

Computation time (based on clock frequency and workload).
Communication time (based on data volume, bandwidth, and latency).

Workloads are allocated proportionally to these ratings. The system includes an iterative adjustment mechanism to handle hard storage constraints: if a worker's assigned weight fragment exceeds its flash capacity, the overflowed rating is redistributed to other workers with available space.

3. Cross-Layer Activation Mapping and Execution

A coordinator orchestrates the inference flow:

Mapping: The system constructs an Assignment Mapping (which worker computes which output neurons) and an Activation Routing Mapping (which worker must send which input activations to which downstream worker).
Execution: Inference proceeds layer-by-layer. The coordinator transmits required input activations to workers based on the routing map. Workers compute their assigned neurons using local weight fragments and return partial results. The coordinator aggregates these results to form the input for the next layer.
Optimizations: The system employs layer fusion (combining Conv-BN-ReLU), quantization (32-bit float to 8-bit int), and asynchronous communication to overlap computation and data transfer, minimizing idle time.

Key Contributions

Fine-Grained Split Inference: A novel approach that partitions CNN models at the sub-layer (neuron/kernel) level, enabling the execution of models that cannot fit on a single MCU by distributing both parameters and intermediate activations.
Resource-Aware Coordination: A rating-based workload allocation mechanism that dynamically balances computation and communication loads across heterogeneous MCUs, preventing bottlenecks caused by slower devices or high-latency links.
System Implementation and Evaluation: A practical implementation on a real testbed of up to 8 Teensy 4.1 MCUs running MobileNetV2, demonstrating feasibility and scalability.

Experimental Results

The system was evaluated using MobileNetV2 on a testbed of 8 Teensy 4.1 MCUs (600 MHz, 1 MB RAM) and a simulator scaling up to 120 MCUs.

Feasibility: The system successfully executed MobileNetV2, a model that causes out-of-memory failures on a single MCU. Peak RAM usage per MCU was reduced to fit within the 1 MB budget.
Latency and Efficiency:
- Heterogeneity: In heterogeneous scenarios (varying frequencies and communication delays), the rating-based allocation significantly outperformed uniform splitting and frequency-only splitting. For example, in a specific heterogeneous case, the optimized scheme reduced total latency by 18.9% compared to a baseline.
- Scalability: As the number of MCUs increased from 3 to 8, computation time decreased monotonically, while communication time increased. The total inference time showed a trade-off where communication overhead eventually became the dominant cost.
- Memory Scaling: Simulation results showed that peak per-MCU memory usage drops significantly as more devices are added, with the most substantial gains occurring within the first 10–20 devices. Beyond this point, diminishing returns are observed.

Significance and Claims

The paper claims that fine-grained split inference is a viable alternative to aggressive model compression for deploying deep learning on resource-constrained MCUs. By shifting the execution paradigm from a single powerful node to a collaborative network of inexpensive MCUs, the system overcomes the peak RAM bottleneck that currently limits TinyML.

The authors emphasize that their approach does not require heavyweight operating systems and operates on bare-metal or lightweight runtimes. They conclude that while communication overhead is a limiting factor for large-scale deployments, the proposed system successfully enables the inference of complex CNNs on networks of MCUs that would otherwise be infeasible, providing a foundation for future work in efficient communication and broader model classes.