💻 computer science

Homomorphic Encryptions for Privacy Preserving Vision

This paper presents a privacy-preserving computer vision framework using Fully Homomorphic Encryption (specifically Microsoft SEAL and TenSEAL) to enable accurate CNN inference on encrypted data, demonstrating minimal accuracy loss across diverse datasets from MNIST to CIFAR-10 while supporting complex operations like multi-channel processing and multiple convolutional layers.

Original authors: Preey Shah, Rohan Virani, Sanjari Srivastava

Published 2026-06-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Preey Shah, Rohan Virani, Sanjari Srivastava

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very valuable, secret recipe for a cake (your private data, like a medical scan or a financial record). You want to ask a famous, high-tech bakery (a cloud-based AI service) to tell you if the cake will turn out well, but you are legally forbidden from showing them the actual recipe. You can't even send them a copy of the recipe to look at later.

This paper is about a clever trick that lets you send the recipe in a locked, unbreakable box to the bakery. The bakers can mix, bake, and taste the cake inside the box without ever opening it or seeing the ingredients. When they send the box back, it contains the answer ("It will be delicious!"), but the box remains locked the whole time.

Here is how the authors achieved this, explained simply:

The Problem: The "Glass Box" Dilemma

Usually, to use powerful AI tools in the cloud, you have to give them your data in plain sight. But for sensitive things like health records, laws say you can't do that. This stops people from using the best AI tools available.

The Solution: Homomorphic Encryption

The authors used a technology called Homomorphic Encryption. Think of this as a magical lockbox.

Normal Encryption: You lock a box. The baker can't touch the cake inside.
Homomorphic Encryption: You lock a box, but the baker has a special pair of gloves that lets them knead, mix, and bake the cake through the box. They can perform math on the cake without ever seeing what it is.

The Challenge: The "Square" Problem

Standard AI (specifically Convolutional Neural Networks, or CNNs) uses a lot of "non-linear" math. Imagine a step in the recipe that says, "If the temperature is above 100, turn the heat off." This is a decision based on a maximum value or a curve.

The Issue: The magic lockbox only understands simple addition and multiplication. It doesn't understand "maximums" or complex curves.
The Fix: The authors had to rewrite the AI's recipe. Instead of asking for a "maximum," they used a simple math trick (like squaring a number) that mimics the curve but only uses multiplication. They also replaced "picking the biggest number" with "averaging the numbers," which the lockbox can handle.

What They Did (The Experiments)

The team built three different "lockbox-ready" AI models to test this idea:

The Simple Model: A basic AI that looks at black-and-white images (like handwritten numbers).
The Layered Model: A slightly more complex AI that looks at images through multiple layers of filters and uses "averaging" to shrink the image size.
The Color Model: An AI capable of looking at colored images (like the CIFAR-10 dataset), which is much harder because it has to process three different color channels (Red, Green, Blue) at once.

They tested these on famous image datasets:

MNIST: Handwritten numbers (the "Hello World" of AI).
Fashion-MNIST: Pictures of clothes.
Kuzushiji MNIST: Ancient Japanese characters.
CIFAR-10: Small, colorful photos of everyday objects (cats, cars, birds).

The Results: Did it Work?

Accuracy: Surprisingly, yes!

For the simpler models, the AI guessed correctly on the locked-box data just as well as, or sometimes even better than, the unlocked version. The authors suggest the "noise" inside the lockbox might have accidentally helped the AI avoid overthinking (overfitting).
For the more complex models, the accuracy dropped only slightly (about 1%), which is a very small price to pay for total privacy.

Speed: This is the catch.

Doing math inside a magic lockbox is slow. The authors found that processing encrypted images took significantly longer than normal.
For the complex color images, it took about 7.5 seconds per image just to get an answer.
They noted that their software didn't use powerful graphics cards (GPUs) yet, which is why it was so slow. If they added those, it would be much faster.

The Trade-Off: Security vs. Speed

The authors played with the "settings" of the lockbox (called context parameters).

Stronger Locks: If you make the lockbox harder to break (higher security), the math takes longer.
Precision: If you want the answer to be super precise, you need more "bits" of data, which also slows things down.
They found a "sweet spot" where the lockbox was secure enough and the accuracy was high, even if it was a bit slow.

The Bottom Line

The paper proves that you can run complex image recognition tasks on secret, encrypted data without decrypting it first. You don't have to choose between privacy and using powerful AI.

However, the current technology is like a slow, manual car compared to a high-speed race car. It works perfectly for privacy, but it's not fast enough for real-time applications yet. The authors hope that in the future, with better hardware (like GPUs) and faster encryption methods, this "magic lockbox" will become fast enough to use everywhere.

Technical Summary: Homomorphic Encryptions for Privacy Preserving Vision

Problem Statement
The paper addresses the conflict between the growing demand for cloud-based Machine Learning-as-a-Service (MLaaS) and legal requirements that prevent organizations from sharing sensitive data (e.g., medical or financial records) in plaintext. While deep learning is increasingly applied to sensitive domains like healthcare (tumor detection, medical imaging), current cloud solutions require data decryption, posing privacy risks. The authors aim to enable Computer Vision inference tasks—specifically Convolutional Neural Networks (CNNs)—to operate directly on encrypted data without decryption, thereby preserving data privacy while maintaining competitive accuracy and inference speeds.

Methodology
The project utilizes Fully Homomorphic Encryption (FHE), specifically the Cheon-Kim-Kim-Song (CKKS) scheme, which supports approximate arithmetic on encrypted data. The implementation relies on Microsoft SEAL and its Python wrapper, TenSEAL.

To adapt standard CNNs for homomorphic encryption, the authors implemented several critical modifications to handle the constraints of HE (which supports only additive and multiplicative operations over commutative rings):

Data Representation: Real numbers are replaced by fixed-precision floating-point numbers encoded into integers.
Non-Linearity Approximation: Standard non-polynomial activation functions like ReLU and Sigmoid are replaced with low-degree polynomials (specifically $z^2$ ) to introduce non-linearity.
Pooling Layers: Max-pooling is replaced by scaled average pooling, which can be expressed as a convolution with a static kernel.
Convolution Implementation: Convolution is transformed into matrix multiplication using the "image-to-columns" method. Since rearranging ciphertext elements is difficult, this preprocessing is done on plaintext before encryption.
Multi-Layer Handling: To support stacked convolutional layers and multi-channel inputs (colored images), the system decrypts and re-encrypts data between layers. This is necessary because TenSEAL lacks native documentation for handling 2-D matrix rearrangements on encrypted 1-D vectors. The authors note this does not compromise privacy, as only kernel sizes and strides (not model parameters) are required for the rearrangement.

Experimental Setup
The authors evaluated three distinct CNN architectures across four datasets: MNIST, Kuzushiji MNIST, Fashion-MNIST, and CIFAR-10.

Architecture 1: A simple CNN with one convolutional layer and two fully connected (FC) layers (single-channel input).
Architecture 2: A deeper model with two convolutional layers, average pooling, and two FC layers (single-channel input).
Architecture 3: A model supporting multi-channel (RGB) inputs, where channel outputs are computed separately and combined via ciphertext addition.

Models were first trained on unencrypted data. The resulting weights were then used to initialize the "encrypted" variants for inference. The study focused on a test split of 5,000 samples per dataset to manage the significant computational overhead of FHE.

Key Results
The experiments demonstrated that classification on encrypted data is feasible with minimal accuracy loss:

Accuracy: For Architecture 1, encrypted inference often matched or slightly exceeded unencrypted baseline accuracy (e.g., 98.90% vs. 97.14% on MNIST). The authors attribute potential improvements to the natural noise introduced by CKKS's approximate arithmetic, which may reduce overfitting. For Architecture 2, accuracy dropped by approximately 1% on complex datasets (KMNIST, Fashion-MNIST), likely due to compounded errors from multiple decrypt/re-encrypt cycles.
Complexity: The system successfully extended support to multi-channel inputs (CIFAR-10), achieving 54.1% accuracy on encrypted data compared to 56.12% on unencrypted data.
Performance Trade-offs: Inference time is significantly higher than unencrypted inference (orders of magnitude slower). The study analyzed the trade-off between context parameters (polynomial modulus degree and coefficient modulus) and performance:
- Increasing the bit scale (precision) did not significantly increase inference time but was necessary to maintain accuracy.
- Increasing the polynomial bit modulus (security) linearly increased inference time (e.g., doubling the modulus from 8192 to 16384 doubled the inference time) while maintaining similar accuracy.

Significance and Contributions
The paper claims modest but significant contributions to the field of privacy-preserving vision:

Feasibility Demonstration: It proves that CNNs can perform inference on encrypted images with accuracy comparable to unencrypted baselines across various datasets, including complex ones like CIFAR-10.
Library Extensions: The authors added support for complex operations on top of TenSEAL, including stacked convolutional layers, average pooling, and multi-channel (colored) image processing.
Parameter Analysis: The work provides an empirical analysis of the time-accuracy-security trade-offs inherent in CKKS encryption, offering guidance on selecting context parameters.

The authors conclude that while current performance is limited by the lack of GPU support in FHE libraries and the computational cost of encryption, the ability to run Computer Vision tasks directly on ciphertexts offers a viable solution for privacy-sensitive applications. They suggest that future improvements in hardware acceleration and faster HE schemes could further enhance the practicality of this approach.