💻 computer science

Latency-Optimal Adaptive Split Inference for Privacy-Preserving Cloud-Edge-End Collaboration

This paper proposes a latency-optimal adaptive split inference framework for privacy-preserving cloud-edge-end collaboration, where end devices execute plaintext model prefixes and encrypt activations using fully homomorphic encryption (FHE) before offloading encrypted segments to edge and cloud servers, achieving significant speedups over full-cloud FHE while maintaining model accuracy.

Original authors: Yi Li, Peng Zhang, Man Ho Au

Published 2026-08-04

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Yi Li, Peng Zhang, Man Ho Au

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot brain (a neural network) that can look at a photo and tell you exactly what's in it. Now, imagine that robot lives far away in a giant, powerful computer center called the "Cloud." If you want to use this brain, you have to send your photo to the Cloud. But what if your photo is a secret? Maybe it's a picture of your medical records or a private diary entry. Sending it to the Cloud feels risky because the Cloud might peek at it.

To solve this, scientists invented a magical lockbox called Fully Homomorphic Encryption (FHE). Think of it like a special glove that lets you do math on a locked box without ever opening it. You can put your secret photo inside, lock it, and send it to the Cloud. The Cloud can "think" about the photo while it's still locked, crunching the numbers inside the box, and send back a locked answer. The Cloud never sees the photo, but it still figures out the answer! The problem is, this magic lockbox is incredibly heavy and slow to carry. It takes a long time to do the math on the locked box, and if you have to carry the whole heavy box all the way to the Cloud, it gets even slower.

This is where the story gets interesting. What if you didn't have to carry the entire heavy box? What if you could do the easy part of the thinking yourself, lock up the middle part, and then share the rest of the work with a helper nearby? This is the big question researchers are asking: How can we split up the work so it's fast, but still keeps our secrets safe?

The Smart Split: A Teamwork Solution

In this paper, a team of researchers named Yi Li, Peng Zhang, and Man Ho Au propose a clever new way to handle this "heavy box" problem. They call their idea a Latency-Optimal Adaptive Split Inference framework. Let's break down what that means using a simple story.

Imagine you are trying to solve a giant, 1,000-piece puzzle (the AI model).

The Old Way (Full Cloud): You take the whole puzzle, lock it in a heavy safe, and mail it to a super-fast factory (the Cloud). The factory unlocks it, solves it, and mails it back. But because the safe is so heavy and the factory is far away, it takes forever.
The New Way (Split Inference): You keep the first few easy pieces of the puzzle (the "prefix") and solve them yourself at home. Then, you take the middle section, lock it in a smaller, lighter safe, and send it to a helper who lives just down the street (the Edge). The helper does some more work on the locked section. If the puzzle is still too big for the helper, they pass the locked box to the big factory (the Cloud) to finish the last few pieces. Finally, the answer comes back to you, and you unlock it.

The researchers built a "smart planner" (a computer program) that figures out the perfect place to cut the puzzle. It asks: "Should I stop after piece 50? Piece 100? Or should I send the whole thing to the Cloud?" The planner looks at how fast your computer is, how fast the helper is, how fast the Cloud is, and how much data needs to be carried. It picks the combination that gets the answer back to you the fastest, without ever letting the helper or the Cloud see the actual picture.

How They Did It

The team tested their idea on two different types of puzzles: one with pictures of animals (called CIFAR-10) and one with pictures of medical slides (called PathMNIST). They used a specific type of magic lock (called CKKS) that is good at handling the math needed for these pictures.

Here is what they found:

Speed is King: Their "split" method was a game-changer for speed. When they compared their method to sending the whole locked box to the Cloud, their method was about 12.9 times faster for the animal pictures and 12.8 times faster for the medical pictures.
The "Fine-Tuning" Trick: They discovered that splitting the work at a very detailed level (cutting between individual layers of the puzzle, which they call "convolution-level") was much better than cutting it only at big chunks (called "block-level"). The detailed split was about 3.9 times faster than the chunky split.
No Secrets Lost: Even though they were splitting the work and using the magic lock, the answers were just as accurate as if they had done the whole thing on a normal computer. The accuracy didn't drop at all.
The Cost: The trade-off is that the "heavy box" (the data) has to be sent back and forth a few more times. The researchers measured that for every single picture, the total time to get the answer was about 1,033 seconds (roughly 17 minutes) for the animal pictures and 1,023 seconds for the medical pictures. While that sounds like a long time, it is a massive improvement over the alternative, which would have taken over 13,000 seconds (more than 3 hours) if they sent everything to the Cloud!

Why This Matters

The researchers are very clear about what they did and what they didn't do. They didn't invent a new type of magic lock; they used the existing one (CKKS) but figured out how to use it much more efficiently by splitting the work. They also didn't solve the problem of making the math instant; the process is still slow because the magic lock is heavy. However, they proved that by being smart about where you do the work—using your own device for the start, a local helper for the middle, and the big factory for the end—you can make privacy-preserving AI actually usable.

They tested this on a simulated setup (a computer pretending to be a local helper and a big factory) and found that their "smart planner" consistently picked the fastest route. They showed that even if the internet connection changes, their plan stays stable. The only time the plan changes is if the local helper becomes almost as fast as the big factory, in which case the planner decides to stop sending the box to the factory at all.

In short, this paper shows us that we don't have to choose between privacy and speed. By breaking the work into tiny, smart pieces and passing them along a chain of helpers, we can keep our secrets safe in a locked box while still getting our answers much, much faster than before. It's like realizing you don't need to run the whole marathon alone; you can run the first mile, hand your baton to a friend, and let them run the rest, all while keeping your secret message safe in the baton.

Technical Summary: Latency-Optimal Adaptive Split Inference for Privacy-Preserving Cloud-Edge-End Collaboration

1. Problem Statement

Internet of Things (IoT) end devices face a critical trade-off when processing privacy-sensitive data (e.g., medical images): they lack the computational resources to run full Convolutional Neural Networks (CNNs) locally, yet offloading raw data or intermediate features to cloud or edge servers exposes sensitive information. While Fully Homomorphic Encryption (FHE) allows computation on encrypted data without decryption, deploying FHE entirely on the cloud is prohibitively slow due to ciphertext expansion and homomorphic arithmetic overhead. Conversely, relying solely on edge servers is insufficient due to their limited capacity compared to the cloud.

The core challenge addressed by this paper is determining where to split a CNN and how to distribute the encrypted suffix between edge and cloud tiers to minimize end-to-end latency while strictly preserving privacy. Specifically, the paper seeks to solve the problem of selecting an optimal split pair (an end-side split point and an edge-side termination point) that balances local plaintext execution, encryption costs, communication delays, and remote FHE execution costs.

2. Methodology

System Architecture and Threat Model

The proposed framework operates on a three-tier architecture:

End Device: Acts as the trust anchor. It executes the initial plaintext prefix of the CNN, encrypts the intermediate activation using FHE (specifically the CKKS scheme), and retains the secret key locally. It never transmits raw inputs or plaintext features.
Edge Server: The first external tier. It receives encrypted activations and executes a continuous segment of the model on ciphertexts.
Cloud Server: The high-capacity tier. It completes the remaining encrypted suffix if the edge does not terminate the inference.

Threat Model: Edge and cloud servers are modeled as "honest-but-curious." They follow the protocol faithfully but may attempt to infer private information from the ciphertexts they process. The framework assumes no collusion between tiers; even if edge and cloud combine observations, they cannot decrypt data without the secret key held solely by the end device.

FHE-Compatible Model Design

To enable efficient homomorphic evaluation, the paper replaces standard ReLU activations with a quadratic residual activation function: $\sigma(z) = (\alpha \cdot z)^2 + z$ . This preserves a linear residual pathway while using a degree-2 polynomial, which is compatible with the CKKS scheme's support for addition and multiplication. The model is trained from scratch with this activation to ensure accuracy is maintained in the encrypted domain.

Latency-Optimal Planning Algorithm

The authors formulate the deployment as a constrained optimization problem to minimize total latency $T_{tot}(q, e)$ , where $q$ is the end-side split point and $e$ is the edge-side termination point. The total latency is the sum of:

Local Plaintext Execution ( $T_{end}$ ): Computed based on the end device's GFLOPS.
Encryption ( $T_{enc}$ ): Cost to encrypt the split activation.
Communication ( $T_{comm}$ ): Transmission delays for ciphertexts between tiers, calculated based on payload size (determined by CKKS parameters and SIMD packing) and configured link bandwidths.
Remote FHE Execution ( $T_{edge}, T_{cloud}$ ): Estimated using empirically calibrated cost densities (seconds per GFLOP) for specific layer groups (e.g., convolution+pooling, fully connected layers) on the respective hardware.

The planner supports two granularities:

Convolution-level: Allows split points at individual convolution layers.
Block-level: Restricts split points to the boundaries of complete network blocks or fully connected layers.

The system evaluates all feasible pairs $(q, e)$ under constraints such as a maximum model exposure policy (limiting the fraction of model parameters on the end device) and structural consistency, selecting the pair that minimizes latency.

3. Key Contributions

Privacy-Preserving Split Framework: A novel architecture where the end device remains the sole holder of the secret key and plaintext data, while edge and cloud tiers perform computation strictly on FHE ciphertexts.
Latency-Aware Planner: A unified cost model and optimization algorithm that jointly considers plaintext execution, encryption, communication, and heterogeneous FHE execution costs across edge and cloud tiers.
Granularity-Aware Partitioning: The framework supports both convolution-level and block-level splitting, demonstrating that finer-grained splits can significantly reduce latency by better utilizing edge resources.
Prototype Implementation: A working prototype using CKKS/OpenFHE on a hybrid testbed (Raspberry Pi 4B for end, Docker containers on a Xeon server for edge/cloud) with VGG16-based models.

4. Experimental Results

Experiments were conducted on CIFAR-10 and PathMNIST datasets using a modified VGG16 (SquareVGG16) architecture.

Latency Performance:
- The proposed convolution-level split achieved an amortized end-to-end latency of 1033.279 s/sample on CIFAR-10 and 1023.429 s/sample on PathMNIST.
- This represents a speedup of approximately 12.9× compared to a full-cloud FHE baseline and 3.9× compared to a block-level split baseline.
- The full-cloud FHE baseline suffered from high latency (approx. 13,357 s/sample on CIFAR-10) due to executing the entire model remotely.
Accuracy:
- The collaborative scheme achieved 88.53% accuracy on CIFAR-10 and 91.43% on PathMNIST.
- These results match the plaintext SquareVGG16 model exactly (or within 0.01 percentage points), confirming that the FHE-compatible activation and split inference do not degrade model performance.
Communication Overhead:
- The convolution-level split incurred higher communication volume (approx. 276 MB/sample) compared to the block-level split (141 MB/sample) because the handoff to the cloud occurred earlier (after Conv. III-1 vs. after Block III). However, the reduction in edge-side FHE computation time outweighed the communication cost, resulting in lower total latency.
Sensitivity Analysis:
- The optimal split plan (Block II on the end, Conv. III-1 on the edge) remained stable across various bandwidth configurations. The plan only shifted deeper into the edge tier when the edge's FHE execution rate approached that of the cloud.

5. Significance and Claims

The paper claims that its primary significance lies in demonstrating that privacy-preserving split inference can be made latency-optimal through adaptive partitioning. By treating the split point selection as a joint optimization problem involving encryption, communication, and heterogeneous FHE execution costs, the framework bridges the gap between the high security of FHE and the practical latency requirements of IoT applications.

The authors emphasize that their work provides a systems-level solution that:

Preserves the cryptographic privacy boundary of full-cloud FHE (no plaintext leaves the end device) while drastically reducing latency.
Validates that fine-grained (convolution-level) splitting is superior to coarse-grained (block-level) splitting in heterogeneous cloud-edge environments, even when communication costs increase.
Offers a reproducible, benchmark-calibrated cost model for future research in encrypted distributed inference.

The paper concludes modestly, noting that the current implementation is a CPU-based research prototype for non-interactive batch inference, not real-time decision making, and that future work will focus on hardware acceleration and broader model families.