Optimizing Split Learning Latency in TinyML-Based IoT Systems

This paper presents the first experimental latency benchmark of TinyML-based Split Learning on ESP32-S3 boards, demonstrating that a proposed Beam Search-based algorithm using the ESP-NOW protocol effectively minimizes end-to-end inference latency by optimizing split points across various models and communication protocols.

The Gist

Imagine you have a very smart but tiny robot (like a smart camera on a doorbell) that needs to solve a complex puzzle, such as recognizing a face. The problem is, this robot is small, has a tiny battery, and a weak brain. If you ask it to solve the whole puzzle alone, it will take forever, or it might run out of battery before finishing.

This paper explores a clever workaround called Split Learning. Instead of asking the tiny robot to do everything, you split the job in half. The robot does the first, easy part of the puzzle, then shouts the "clues" it found to a bigger, stronger robot nearby (like a smart speaker or a local server). The bigger robot finishes the hard part of the puzzle and shouts the answer back.

The authors of this paper wanted to figure out the fastest way to do this shouting and listening game using real, low-power hardware (specifically ESP32-S3 boards, which are cheap, open-source microcontrollers).

Here is a breakdown of their findings using simple analogies:

1. The "Shouting" Problem: Choosing the Right Protocol

When the tiny robot sends its clues to the big robot, it has to choose a "language" or a "delivery method" to send the data. The researchers tested four different methods, like choosing between different types of mail services:

• UDP: Like sending a postcard. It's very fast because you don't wait for a receipt, but if the card gets lost, you don't know.
• TCP: Like sending a registered letter. It's very reliable (you get a receipt), but it takes longer because of all the "handshake" paperwork before the letter is sent.
• BLE (Bluetooth): Like a slow, chatty walkie-talkie. It connects well but takes a long time to set up the conversation and sends data in very small, fragmented chunks.
• ESP-NOW: Like a specialized, high-speed walkie-talkie that doesn't need to set up a formal connection first. It just blasts the message out.

The Winner: Surprisingly, ESP-NOW was the fastest overall. Even though it has a small "envelope" size limit (it can't carry huge chunks of data at once), it saves so much time by skipping the formal connection setup that it beat the others. It finished the round-trip (sending clues and getting an answer back) in about 3.6 seconds, while Bluetooth took over 10 seconds.

2. The "Cutting" Problem: Where to Split the Job?

The researchers also had to decide exactly where to cut the puzzle.

• Cutting too early: The tiny robot does almost nothing, but it has to send a huge pile of clues to the big robot. This clogs the network.
• Cutting too late: The tiny robot does almost everything, which takes too long for its weak brain.

They tested different "cut points" in two popular AI models (MobileNet-V2 and ResNet50). They found that the best spot to cut depends on the model and the network, but generally, they wanted to find the "Goldilocks" zone where the tiny robot does just enough work without overwhelming the network.

3. The "Smart Planner": Beam Search

Finding the perfect cut point is like trying to find the best route through a maze.

• Brute Force: Trying every single possible path. This guarantees the best route, but it takes forever (days) to calculate.
• Greedy Search: Taking the first path that looks good. It's fast, but you might get stuck in a dead end later.
• Beam Search (The Winner): Imagine you are exploring the maze, but instead of checking every path, you only keep track of the top 3 most promising paths at any given moment. If a path looks bad, you drop it. If a path looks good, you keep it and explore further.

The researchers created an algorithm using this Beam Search method.

• The Result: It found a near-perfect route almost instantly (in about 0.1 seconds for a group of 5 devices).
• Why it matters: It's fast enough to be used in real-time systems, unlike the "Brute Force" method which would take hours or days to calculate the same thing.

Summary of the "Recipe"

The paper concludes with a simple recipe for making these tiny IoT devices work together efficiently:

1. Use ESP-NOW for communication because it skips the boring setup steps and is the fastest for round-trips.
2. Use the Beam Search algorithm to automatically decide where to split the AI model. This ensures the tiny robot and the big robot share the work in the most time-efficient way possible.

By combining the right "shouting method" (ESP-NOW) with a smart "planner" (Beam Search), they managed to make these tiny, low-power devices solve complex AI puzzles much faster than before, without needing to upgrade the hardware.

Technical Summary

Technical Summary: Optimizing Split Learning Latency in TinyML-Based IoT Systems

Problem Statement

The rapid evolution of Artificial Intelligence faces a significant bottleneck in deploying deep learning (DL) inference on ultra-low-power, resource-constrained edge and IoT devices. While TinyML offers a solution through lightweight models, many applications still exceed the memory and processing capabilities of individual microcontrollers. Split Learning (SL) addresses this by partitioning a model across devices, executing early layers on the sensor and offloading the remainder to a companion device. However, the performance of SL in this context remains under-explored. Specifically, there is a lack of empirical evidence regarding:

1. The end-to-end inference latency of SL on constrained hardware under realistic low-power wireless protocols.
2. The impact of different wireless communication protocols (WiFi, ESP-NOW, BLE) on split latency, including network setup, transmission of intermediate activations, and prediction feedback.
3. The optimal selection of "split points" (where the model is divided) to minimize total latency, considering both computation and communication overheads.

Existing studies have largely focused on smartphones or single-board computers, often assuming ideal transmission conditions or using heuristic split selection methods that do not account for protocol-specific overheads like packet loss or connection handshakes.

Methodology

The authors propose an experimental framework and an optimization algorithm to address these gaps.

1. Experimental Testbed

• Hardware: The system utilizes ESP32-S3-WROOM-1 boards (240 MHz, 16 MB Flash) as IoT nodes and a Desktop PC (Intel Core i9-14900) as an edge server.
• Models: Two Convolutional Neural Networks (CNNs) were used: MobileNet-V2 (lightweight) and ResNet50 (larger).
• Framework: The models were prepared, partitioned, and quantized using TensorFlow Lite (TFLite) on the edge server. Firmware was deployed to IoT devices via Over-the-Air (OTA) updates.
• Protocol Comparison: Four wireless communication protocols were benchmarked for transmitting intermediate activations:
  • UDP (over WiFi)
  • TCP (over WiFi)
  • ESP-NOW (low-power, peer-to-peer)
  • BLE (Bluetooth Low Energy)
• Measurement: Latency was measured using high-resolution timers on the ESP32-S3, capturing Round-Trip Time (RTT) components including protocol setup, model loading, tensor allocation, inference, buffering, transmission, and feedback.

2. Optimization Framework

The paper formulates the split point selection as an optimization problem to minimize total inference latency ($T_{inference}$), defined as the sum of device-local processing latency ($T_d$) and transmission latency ($T_{tr}$).

• Transmission Model: The transmission latency accounts for packet size, Maximum Transmission Unit (MTU) limits, propagation delay, and packet loss probability.
• Search Algorithms: To solve the optimization problem (finding the optimal set of split points $s^*$), the authors compare four strategies:
  1. Brute Force: Exhaustive search (computationally infeasible for large $L$).
  2. Random-Fit: Random selection of split points.
  3. First-Fit: Selects the first split point satisfying a latency threshold.
  4. Greedy Search: Sequentially selects split points to minimize immediate segment cost.
  5. Beam Search: A novel approach for this context that expands only the top-$B$ most promising partial solutions at each step, balancing search accuracy with computational efficiency.

Key Results

Protocol Performance

• ESP-NOW: Achieved the best overall Round-Trip Time (RTT) of 3.6 seconds in the two-device setup. Despite a smaller packet limit (250 bytes) compared to UDP/TCP, its lack of connection-handshake overhead and efficient MAC-layer broadcast mechanism resulted in the lowest total latency.
• UDP: Provided the lowest raw transmission latency (e.g., 1.4 ms for small payloads) due to a large MTU (1472 bytes) and lack of acknowledgment overhead. However, protocol setup times were significant (>2 seconds).
• TCP: Suffered from high latency due to connection setup and retransmission overheads, particularly when handling large intermediate activation tensors (e.g., >100 packets), leading to buffer stalls on the ESP32.
• BLE: Resulted in the highest latency (10.4 s RTT) due to excessive fragmentation (512-byte MTU) and high setup/feedback delays.

Split Point Optimization

• Algorithm Efficiency: The Beam Search algorithm demonstrated near-optimal latency performance comparable to Brute Force but with drastically reduced processing time. For a scenario with 5 devices, Beam Search required only 0.1 seconds of processing time, whereas Brute Force would take exponentially longer (projected ~7857 seconds for 6 devices).
• Latency Reduction: Beam Search reduced latency by over 600% compared to Random-Fit for 6 devices.
• Model Specifics:
  • For MobileNet-V2, Beam Search consistently achieved the lowest latency across varying device counts.
  • For ResNet50, while Beam Search remained the most efficient, latency fluctuations were observed at higher device counts due to some nodes lacking the capacity to run specific model segments.

Specific Findings on Split Points

Manual benchmarking identified the block_16_project_BN layer in MobileNet-V2 as a highly effective split point when using ESP-NOW, balancing the computational load and data transmission size effectively.

Significance and Claims

The paper claims to provide the first experimental latency benchmark of TinyML-based Split Learning on low-power ESP32-S3 boards. Its primary contributions are:

1. Empirical Evidence: It fills a gap in the literature by providing real-world measurements of SL latency across different wireless protocols, moving beyond theoretical simulations or smartphone-based studies.
2. Protocol Selection: It establishes that while UDP offers low transmission latency, ESP-NOW is the superior protocol for end-to-end SL RTT in constrained IoT environments due to negligible setup overhead.
3. Optimization Algorithm: It introduces and validates a Beam Search-based algorithm for automatic split point selection. The authors claim this method offers a practical, scalable solution for real-time deployments, providing near-optimal latency with minimal computational cost, unlike exhaustive search methods.
4. Reproducibility: The source code and experimental setup are made publicly available to serve as a reproducible baseline for future research in TinyML and Split Learning.

The authors conclude that while their current work focuses on static split points and fixed protocols, future work aims to develop a dynamic framework that adapts split points, chunk sizes, and protocols in real-time based on network conditions and device resources.