💻 computer science

Privacy-Preserving Object Detection for Vision Transformer-Based Models

This paper introduces a novel privacy-preserving object detection method for Vision Transformer-based models that utilizes perceptual encryption and domain adaptation with keys to protect sensitive visual information while maintaining accuracy comparable to unprotected models.

Original authors: Homare Sueyoshi, Kiyoshi Nishikawa, Hitoshi Kiya

Published 2026-08-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Homare Sueyoshi, Kiyoshi Nishikawa, Hitoshi Kiya

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern world, computers have learned to see. They can scan a photograph and instantly identify a dog, a car, or a person, a capability that powers everything from self-driving cars to security systems. This ability relies on a type of artificial intelligence called a Vision Transformer, which breaks an image into small pieces and analyzes how they fit together to understand the whole scene. However, this power comes with a risk. To use these systems, people often have to send their private photos to a remote server in the cloud. If that server is compromised, or if the people running it are untrustworthy, sensitive visual information could be exposed. The challenge for scientists is to find a way to let the computer see enough to do its job without ever revealing the actual image to anyone but the owner.

A team of researchers at Tokyo Metropolitan University has tackled this problem by creating a new method for protecting images while they are being analyzed for object detection. Their work focuses on a specific type of artificial intelligence model that is particularly good at spotting things in pictures. Instead of trying to hide the image with complex, slow encryption that would confuse the computer, they devised a clever trick involving the way the computer processes the image. They take the original photo and scramble its visual details using a secret key, turning it into a jumbled, unrecognizable mess to the human eye. At the same time, they apply a matching scramble to the computer's own internal instructions. When these two scrambled versions meet, the computer can still perform its task with high accuracy, but the server holding the data sees only noise.

The researchers tested this idea using a powerful model known as ViTdet, which is designed to find and locate objects within an image. In a standard setup, if you send a scrambled image to a computer that hasn't been adjusted for it, the system fails completely. In their experiments, when the researchers encrypted the photos but left the computer model unchanged, the system's ability to detect objects dropped to near zero. The computer simply could not make sense of the scrambled data. This proved that simply hiding the image was not enough; the tool used to read the image had to be changed to match.

To solve this, the team created a system where the model itself is encrypted before it is ever sent to the cloud. They generated a random pattern, like a unique shuffling code, and used it to rearrange the way the computer understands the small pieces of an image. They then applied the exact same shuffling pattern to the photos before they were sent for analysis. Because the shuffling was done in a specific mathematical way, the two scrambled elements canceled each other out inside the computer's brain. The computer processed the data as if it were looking at the original, clear photo, even though it never actually saw one.

The results of their tests were striking. When they used this dual-encryption method on a large collection of images containing eighty different categories of objects, the system performed almost as well as if no encryption had been used at all. In one test using a standard dataset, the system correctly identified objects with an accuracy score of 50.118, which is very close to the 51.412 score achieved when using unencrypted images. Even when they tested the system on a much larger and more complex set of over a thousand object categories, the performance remained high, with the encrypted method achieving scores only slightly lower than the unencrypted baseline. Crucially, the server running the analysis never saw the original image, nor did it possess the secret key needed to unscramble it.

This approach offers a significant step forward for privacy. It demonstrates that it is possible to outsource sensitive visual tasks to the cloud without sacrificing accuracy or security. The researchers found that the method works effectively across different sizes of objects, from small details to large scenes, and holds up even when the images are compressed or resized. By ensuring that the computer's internal logic and the input data are locked together with a secret key, they have created a system where the cloud can do the thinking without ever knowing what it is thinking about. This means that in the future, users could potentially share their private photos for analysis with the confidence that the visual content remains hidden, protected by a mathematical shield that only the user and their specific model can unlock.

Technical Summary: Privacy-Preserving Object Detection for Vision Transformer-Based Models

Problem Statement

While Vision Transformers (ViTs) have achieved significant success in privacy-preserving learning, access control, and model protection, existing research utilizing perceptual encryption has been largely confined to image classification tasks. There is a notable gap in applying these privacy-preserving techniques to object detection, a more complex task requiring not only classification but also localization. Furthermore, standard encryption methods applied directly to input images typically degrade detection performance to near-zero levels when the model itself remains unencrypted, as the encryption disrupts the feature extraction process.

Methodology

The paper proposes a novel object detection framework that integrates perceptual encryption with the embedding structure of Vision Transformers (ViT). The method is designed to protect sensitive visual information in test images while maintaining detection accuracy comparable to non-encrypted models.

Core Framework

The proposed system operates on a client-server architecture involving a model creator, a user, and a cloud server:

Key Generation: A model creator generates a random permutation matrix ( $E_{enc}$ ) using a secret key ( $k$ ).
Model Encryption: The patch embedding matrix ( $E$ ) of a pre-trained ViT-based object detection model (specifically ViTdet) is encrypted using $E_{enc}$ to create an encrypted model ( $\hat{E} = E_{enc}E$ ). This encrypted model is uploaded to the cloud server.
Image Encryption: The user encrypts their test image using the same $E_{enc}$ . The image is divided into non-overlapping blocks ( $p \times p$ ), flattened into vectors, and transformed via the permutation matrix ( $\hat{b}_i = b_i E_{enc}^T$ ).
Inference: The encrypted image is sent to the cloud server, which performs inference using the encrypted model. The server never accesses the plain image or the secret key.

Theoretical Mechanism

The method relies on the mathematical properties of the ViT architecture and orthogonal matrices:

Orthogonality: The random permutation matrix $E_{enc}$ is orthogonal, satisfying $E_{enc}^T = E_{enc}^{-1}$ .
Cancellation Effect: In the ViT embedding layer, the input patch $x_p$ $x_{p}$ is projected by the embedding matrix $E$ $E$ . In the proposed method, the encrypted patch $\hat{x}_p$ $\overset{x}{^}_{p}$ is projected by the encrypted matrix $\hat{E}$ $\hat{E}$ .
- The projection becomes: $\hat{x}_p \hat{E} = (x_p E_{enc}^T) (E_{enc} E) = x_p (E_{enc}^T E_{enc}) E = x_p E$ .
Result: The encryption applied to the image and the encryption applied to the model cancel each other out at the embedding layer. Consequently, the input sequence fed into the Transformer Encoder ( $\hat{z}_0$ ) is mathematically identical to the sequence in the non-encrypted scenario ( $z_0$ ), ensuring the detection logic remains unaffected.

Key Contributions

First Application to Object Detection: This work presents the first method to utilize perceptual encryption specifically for object detection tasks, expanding the scope beyond image classification.
ViT-Based Architecture: The method leverages the specific embedding structure of ViT-based models (ViTdet), utilizing a plain ViT backbone with Window Attention and a simple feature pyramid.
Domain Adaptation with Keys: The study successfully adapts domain adaptation techniques involving secret keys, previously used for classification, to the more complex domain of object detection.
Dual Encryption Strategy: The paper demonstrates that encrypting both the model's embedding layer and the input image is necessary to preserve accuracy, as encrypting only the image results in catastrophic performance loss.

Experimental Results

The method was evaluated using ViTdet models with three different backbones (ViT-B, ViT-L, and ViT-H) on the COCO and LVIS datasets.

Accuracy Preservation: The proposed method achieved detection accuracy (Average Precision, AP) nearly identical to the non-encrypted baseline.
- On COCO, the ViT-H backbone achieved an AP of 55.843 (Proposed) compared to 56.594 (Baseline).
- On LVIS (which features 1,203 categories), the ViT-H backbone achieved an AP of 47.811 (Proposed) compared to 49.123 (Baseline).
Failure of Partial Encryption: When only the images were encrypted without encrypting the model, accuracy dropped to near zero (e.g., ViT-H on COCO dropped from 56.594 to 0.406 AP), confirming the necessity of the dual-encryption approach.
Visual Protection: The encrypted images appeared as scrambled blocks, effectively hiding sensitive visual information from the cloud server, while the model successfully detected objects.
Minor Discrepancies: The authors note that the slight reduction in accuracy compared to the baseline (e.g., ~1-2% drop) is attributed to the influence of JPEG compression inherent in the COCO dataset, rather than the encryption method itself.

Significance and Claims

The paper claims that by exploiting the embedding structure of Vision Transformers, it is possible to achieve privacy-preserving object detection without compromising detection performance. The significance lies in:

Enabling the deployment of object detection models on untrusted cloud servers where visual data privacy is paramount.
Demonstrating that the theoretical cancellation of encryption effects in the embedding layer holds true in practice for complex detection tasks.
Providing a practical framework where the cloud server can perform inference on encrypted data without ever possessing the decryption key or the original image.

The authors conclude that the method effectively protects visual information while maintaining high detection accuracy, validating the feasibility of privacy-preserving object detection using ViT-based architectures.