🤖 machine learning

Towards Deep Encrypted Training: Low-Latency, Memory-Efficient, and High-Throughput Inference for Privacy-Preserving Neural Networks

This paper presents optimized algorithms and a pipeline architecture for batched homomorphic encryption inference that significantly reduce runtime and memory usage compared to state-of-the-art designs, enabling efficient privacy-preserving training and high-throughput processing of encrypted neural network workloads.

Original authors: Nges Brian Njungle, Eric Jahns, Michel A. Kinsy

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nges Brian Njungle, Eric Jahns, Michel A. Kinsy

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very sensitive secret, like your medical records or your bank account details. You want to ask a super-smart AI a question about this data (e.g., "Do I have a disease?" or "Is this transaction fraudulent?"), but you don't want to show the AI your actual data. You also don't want the AI to see your answer until it's done.

This is the problem Homomorphic Encryption (HE) solves. It's like putting your data in a magic, unbreakable glass box. You can hand this box to a worker (the AI), and they can perform calculations inside the box without ever opening it or seeing what's inside. When they are done, they hand the box back, and you open it to see the result.

However, there's a big catch: It's incredibly slow and heavy.

Think of the "magic box" as being made of lead. If you ask the AI to look at just one photo, it takes a long time. If you ask it to look at 512 photos at once (which is how normal AI works to be fast), the old methods would require the AI to carry 512 separate lead boxes. The computer would get so heavy and tired (running out of memory and time) that it would crash or take forever.

This paper is about building a "Magic Conveyor Belt" that solves this problem.

Here is a simple breakdown of what the authors did:

1. The Problem: The "One-by-One" Bottleneck

Previously, if you wanted to process a batch of 512 encrypted images, the AI had to handle them one by one, or in a very clumsy way that wasted space.

Analogy: Imagine a post office where a worker has to open a heavy, locked safe, take out one letter, stamp it, put it back, lock the safe, and then move to the next safe. Doing this for 512 letters takes forever.
The Old Way: The previous best method was like having a worker who could open the safe, grab 512 letters, but the safe was so heavy (370 GB of memory) that it required a massive, expensive crane (super-computer) just to lift it.

2. The Solution: The "Smart Packing" Trick

The authors realized that instead of treating every image as a separate heavy box, they could pack them tightly together like Tetris blocks inside a single, larger magic box.

The Analogy: Instead of 512 separate safes, they figured out how to stack 512 letters neatly inside one giant, reinforced safe.
The Result: The worker only has to lift one safe instead of 512. This makes the process much faster and requires much less muscle (memory).

3. The "Accumulator" (The Traffic Controller)

As the AI processes the images, the "boxes" get smaller (because the AI zooms out to see the whole picture). This leaves empty space inside the magic box.

The Old Way: The worker would leave the empty space empty, wasting room.
The New Way: The authors built a device called an Accumulator. Think of it like a smart conveyor belt that automatically grabs the empty space left behind by the first batch of images and immediately fills it with the next batch.
Why it matters: It keeps the "magic box" 100% full at all times. No wasted space, no waiting around. This allows the system to process huge batches of data without slowing down.

4. The Results: Faster, Lighter, Cheaper

The team tested this on two famous AI models (ResNet-20 and ResNet-34) using standard image datasets.

Speed: They processed 512 images in about 8.86 seconds per image (on average). The previous best method took about 15.76 seconds. That's nearly 2x faster.
Memory: The old method needed a massive 370 GB of RAM (like a super-computer). Their new method only needed 98 GB (like a high-end gaming PC or a standard server). That's almost 4x less memory.
Hardware: They did this on a standard, affordable server, whereas the previous record holders needed a million-dollar super-computer setup.

Why Should You Care?

This isn't just about math; it's about privacy in the real world.

Hospitals: A hospital could send thousands of encrypted X-rays to a cloud AI to check for tumors without ever revealing the patients' identities.
Banks: A bank could check thousands of encrypted transactions for fraud instantly without exposing customer data.
You: You could use a smart AI assistant to analyze your private financial or health data without the AI ever "seeing" your raw numbers.

In a nutshell: The authors figured out how to pack more encrypted data into less space and move it faster, turning a slow, heavy, expensive process into something that can run on standard computers. They turned a "heavy, slow lead box" into a "lightning-fast, efficient conveyor belt."

1. Problem Statement

Privacy-Preserving Machine Learning (PPML) using Homomorphic Encryption (HE) allows neural network inference on encrypted data without revealing raw inputs or model weights. However, current state-of-the-art HE systems face two critical bottlenecks:

Single-Input Limitation: Most existing works focus on processing one encrypted image at a time. This approach is computationally expensive and lacks scalability for high-throughput scenarios (e.g., processing hundreds of medical scans or financial transactions simultaneously).
Resource Inefficiency in Batching: While batch processing exists, prior methods (e.g., Cheon et al., 2024) suffer from massive memory footprints (often >370 GB RAM) and high latency. They struggle to efficiently utilize SIMD (Single Instruction, Multiple Data) slots in ciphertexts, leading to wasted resources and poor amortized latency.
Hardware Constraints: Existing high-performance solutions often rely on specialized, high-end hardware (e.g., 1 TB RAM, 40+ cores), making them inaccessible for practical deployment on standard mid-range servers.

2. Methodology

The authors propose a comprehensive framework built on the OpenFHE library and the FHEON framework, utilizing the CKKS scheme (approximate HE for real numbers). Their approach focuses on algorithmic optimization and architectural pipeline design.

A. Optimized Layer Algorithms

Channel-By-Channel (CBC) Packing: Instead of packing entire images into a single ciphertext, the input is reorganized so that each ciphertext contains the same channel across multiple images in a batch. This preserves batch-level parallelism while enabling efficient channel-wise operations.
Optimized Convolution:
- Rotation Reduction: The authors introduce a strategy to reduce the number of required rotation keys for a $k \times k$ kernel from $k^2$ to just 4 keys (using base rotations of $\pm 1$ and $\pm W$ ).
- Elimination of Intermediate Rotations: By aligning data via CBC packing, the need to re-align intermediate results across channels before summation is eliminated, reducing rotation overhead.
Optimized Fully Connected (FC) Layers:
- Batched Inner Products: The FC layer is reformulated to process multiple inputs simultaneously using slot-wise summation.
- Rotation Key Decomposition: To handle large batch sizes without generating excessive rotation keys, the rotation offset is decomposed into base_idx and mod_idx, allowing the reuse of rotation keys and limiting the number of keys needed per operation to two.

B. Multi-Threaded Execution

The system exploits thread-level parallelism by decoupling computation steps:

Convolution: Input alignment (rotation generation) and convolution evaluation are split into independent submodules. Threads process disjoint subsets of ciphertexts or output channels without data dependencies until the final aggregation.
FC Layers: Per-output-channel evaluation and batch-wise result placement are parallelized.
Result: This design minimizes synchronization overhead and maximizes CPU core utilization, though it requires careful memory management to handle thread-local intermediate ciphertexts.

C. The Accumulator Pipeline Architecture

A novel Accumulator mechanism is introduced to manage ciphertext slots dynamically:

Slot Reclamation: Downsampling layers (strided convolutions/pooling) reduce spatial dimensions, leaving "empty" slots in the ciphertext.
Dynamic Merging: The accumulator merges multiple intermediate ciphertexts from different block groups into a single, densely packed ciphertext.
Batch Size Control: This allows the system to increase the effective batch size as the data flows deeper into the network (e.g., starting with 16 images and merging up to 512), ensuring the ciphertext remains fully utilized throughout the inference pipeline.
Key Reuse Strategy: Encryption keys are loaded and offloaded dynamically based on the active block group, significantly reducing the peak memory footprint.

3. Key Contributions

Algorithmic Efficiency: Development of optimized algorithms for batched HE-friendly layers that reduce rotation keys, streamline operations, and improve ciphertext packing efficiency.
Pipeline Architecture: Introduction of an Accumulator-based pipeline that maintains high resource utilization by dynamically merging ciphertexts, enabling flexible batch size control and maximizing throughput.
Scalable Evaluation: Implementation and evaluation of ResNet-20 and ResNet-34 models on CIFAR-10 and CIFAR-100 datasets across batch sizes ranging from 16 to 512.
Hardware Accessibility: Demonstration of high performance on a standard mid-range server (32 CPU cores, 256 GB RAM), avoiding the need for the 1 TB RAM systems required by previous state-of-the-art works.

4. Experimental Results

The system was evaluated on an Intel Xeon Gold 5418Y (32 cores, 256 GB RAM).

Performance (ResNet-20, Batch 512):
- Amortized Inference Time: 8.86 seconds per image.
- Memory Usage: 98.96 GB RAM.
- Comparison: Compared to the state-of-the-art Cheon et al. (2024), this represents a 1.78× speedup in runtime and a 3.74× reduction in memory usage (from 370.28 GB to 98.96 GB), despite using significantly less powerful hardware.
Performance (ResNet-34, Batch 256):
- Amortized Inference Time: 28.14 seconds per image.
- Memory Usage: 246.78 GB RAM.
Accuracy: The encrypted models achieved accuracy comparable to their plaintext counterparts (e.g., 92.2% vs 92.8% for ResNet-20 on CIFAR-10), confirming the correctness of the optimizations.
Scalability: The amortized latency decreases significantly as batch size increases, proving the effectiveness of the slot-reclamation and merging strategy.

5. Significance

Bridging the Gap: This work bridges the gap between theoretical HE capabilities and practical, high-throughput deployment. It moves PPML from single-query scenarios to batch-processing workloads essential for real-world applications like healthcare and finance.
Cost-Effectiveness: By reducing memory requirements from ~370 GB to <100 GB for high-throughput tasks, the authors make encrypted inference feasible on standard enterprise servers, lowering the barrier to entry for PPML adoption.
Foundation for Training: While focused on inference, the techniques (slot reclamation, multi-threading, efficient batching) lay the necessary groundwork for the next frontier: encrypted deep neural network training, which is currently computationally prohibitive.
Open Source: The authors intend to release the first open-source codebase for high-throughput encrypted neural network inference, fostering further research and reproducibility.