💻 computer science

Privacy-Preserving Clothing Classification using Vision Transformer for Thermal Comfort Estimation

This paper proposes a privacy-preserving clothing classification method using Vision Transformers (ViT) that enables secure occupant-centric thermal comfort control by maintaining high accuracy on encrypted images, thereby overcoming the severe accuracy degradation typically associated with conventional privacy-preserving image classification schemes.

Original authors: Tatsuya Chuman, Yousuke Udagawa, Hitoshi Kiya

Published 2026-04-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tatsuya Chuman, Yousuke Udagawa, Hitoshi Kiya

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want your home's air conditioning to be perfectly comfortable, but you don't want to show your family photos to the company that manages the system. That's the problem this paper solves.

Here is the story of how they did it, explained simply:

The Problem: The "Glass House" Dilemma

To make a room feel just right, heating and cooling systems need to know what people are wearing. A heavy coat means the room needs to be warmer; a t-shirt means it can be cooler.

Usually, cameras take pictures of people to guess what they are wearing. But sending raw photos to a cloud server is like handing your diary to a stranger to read. It's a privacy nightmare.

Scientists have tried to fix this by scrambling the photos (encrypting them) before sending them. However, there's a catch: scrambling usually breaks the "brain" (the AI) trying to read the photo. It's like trying to solve a jigsaw puzzle when someone has chopped the pieces into tiny, unrecognizable bits. The old methods were so bad at reading scrambled photos that the AI would guess wrong almost half the time.

The Solution: The "Lego Block" Brain

The authors of this paper found a new way to scramble the photos that doesn't break the AI's brain. They used a special type of AI called a Vision Transformer (ViT).

Think of a normal AI (like a CNN) as a detective who looks at a photo pixel-by-pixel, like reading a book one letter at a time. If you shuffle the letters, the detective can't read the story.

The Vision Transformer, however, is like a detective who looks at the photo in big chunks (like Lego blocks). It doesn't care about the exact order of the tiny pixels inside the block; it just cares about the shape and color of the whole block.

The Magic Trick: The Secret Shuffle

Here is the clever part of their method:

The Setup: The AI developer takes a pre-trained "brain" and locks it with a secret key.
The Scramble: When a user wants to check their outfit, they take their photo and cut it into those same big "Lego blocks." Then, using the secret key, they shuffle the blocks around and mix up the pixels inside each block.
- Analogy: Imagine a photo of a cat. The old method would turn it into static noise. This new method turns it into a photo where the cat's head is on the left, the tail is on the right, and the fur is a different color, but the shape of the cat is still there.
The Result: Because the Vision Transformer only looks at the big chunks, it can still recognize the "cat" (or the "coat") even though the picture looks like a scrambled mess to a human eye.

What They Found

The researchers tested this on a huge database of clothing photos, sorting them into four groups: sleeveless, short-sleeve, long-sleeve, and heavy outerwear.

The Old Way (Pixel-based): When they scrambled the photos, the AI's accuracy dropped significantly. For heavy coats, it went from being right 73% of the time to only 65%. It was struggling to tell the difference between a sweater and a jacket.
The New Way (ViT-based): When they used their new "Lego block" shuffle, the AI got 95.65% right.
- The Miracle: The accuracy on the scrambled photos was exactly the same as the accuracy on the original, unscrambled photos. The scrambling didn't hurt the AI at all.

The Bottom Line

This paper shows that we can now send "scrambled" photos to the cloud to figure out what people are wearing for temperature control, without the AI losing its ability to guess correctly.

It's like sending a locked, shuffled puzzle to a friend who has the special skill to solve it instantly, while keeping the picture hidden from everyone else. This makes it possible to have smart, comfortable homes without sacrificing your privacy.

1. Problem Statement

The paper addresses a critical conflict in modern building management systems: the need for Occupant-Centric Control (OCC) to optimize thermal comfort versus the imperative of user privacy.

Context: HVAC (Heating, Ventilation, and Air Conditioning) systems increasingly use camera images and deep learning to estimate occupant thermal comfort, often relying on the Predicted Mean Vote (PMV) index.
Key Parameter: Clothing insulation (clo-value) is a vital input for PMV calculations.
The Challenge:
- Transmitting raw images containing Personally Identifiable Information (PII) to untrusted cloud servers for processing raises significant privacy concerns.
- Existing privacy-preserving methods, which typically encrypt images before sending them to a model (e.g., using Convolutional Neural Networks or CNNs), suffer from severe accuracy degradation. This is because pixel-based encryption disrupts the spatial correlations that CNNs rely on for feature extraction.

2. Methodology

The authors propose a novel scheme that leverages the Vision Transformer (ViT) architecture combined with a specific key-based image encryption technique.

A. System Architecture

The workflow involves two parties: a Model Developer and a User.

Model Development:
- A pre-trained ViT model (specifically ViT-S/16) is fine-tuned on a clothing dataset.
- The model parameters are encrypted using a secret key $K$ to create an encrypted model $M$ .
- $M$ is uploaded to a cloud server.
User Inference:
- The user encrypts their input image $x$ using the shared secret key $K$ to generate an encrypted image $x'$ .
- $x'$ is transmitted to the cloud server.
- The server performs inference using the encrypted model $M$ on the encrypted image $x'$ and returns the classification result.

B. Encryption Mechanism (ViT-Based)

Unlike traditional pixel-based encryption, this method exploits the specific processing nature of ViT:

Patch Division: The input image is divided into non-overlapping blocks of size $M \times M$ , matching the ViT patch size ( $P=16$ ).
Two-Step Encryption:
1. Pixel Shuffling: Pixels within each block are permuted using a pseudo-random sequence generated by the key.
2. Block Scrambling: The blocks themselves are spatially permuted (shuffled) across the image.
Theoretical Advantage: Since ViT processes images as a sequence of patches and utilizes a self-attention mechanism that is largely independent of the absolute spatial arrangement of these patches, the model can extract features from the scrambled image $x'$ without needing decryption. This preserves the semantic information required for classification.

3. Key Contributions

Novel Application of ViT for Privacy: The paper is the first to demonstrate that ViT architectures can maintain high classification accuracy on encrypted images, overcoming the accuracy drop seen in CNN-based privacy schemes.
End-to-End Privacy-Preserving HVAC Control: It provides a practical framework for estimating clothing insulation (a critical PMV parameter) without exposing raw occupant images to cloud providers.
Zero Accuracy Degradation: The proposed scheme achieves identical accuracy on encrypted images as it does on plain (unencrypted) images, effectively eliminating the privacy-utility trade-off for this specific task.

4. Experimental Results

The authors evaluated the method using the DeepFashion dataset (26,887 images), categorized into four insulation levels: Sleeveless, Short-Sleeve Shirt, Long-Sleeve Shirt, and Outerwear.

Baseline Comparison: They compared the proposed ViT-based scheme against a conventional Pixel-based encryption method applied to a ResNet-18 model.
Performance Metrics:
- Conventional Method (ResNet-18 + Pixel Encryption): Suffered a significant accuracy drop from 93.51% (plain) to 83.34% (encrypted). The drop was severe for complex categories like Long-Sleeve Shirt (dropping to 67.40%) and Outerwear (65.64%).
- Proposed Method (ViT-S/16 + ViT-based Encryption): Maintained an overall accuracy of 95.65% on encrypted images, showing 0% degradation compared to the plain image baseline (95.65%).
- Category Consistency: The ViT-based scheme outperformed the conventional method across all four clothing categories.

5. Significance and Impact

Reliable Thermal Comfort Assessment: By maintaining high accuracy on encrypted data, the system ensures that PMV calculations remain reliable, preventing misclassification of clothing insulation which could lead to inefficient HVAC control or occupant discomfort.
Scalability and Edge Deployment: The model used (ViT-S/16) has 22.0 million parameters, making it suitable for implementation on edge devices in consumer electronics, further enhancing the practicality of the solution.
Paradigm Shift: This work suggests a shift in privacy-preserving computer vision, moving away from CNNs (which struggle with encrypted spatial data) toward Transformer-based architectures that are inherently more robust to the spatial permutations introduced by encryption.

In conclusion, the paper successfully demonstrates that Vision Transformers offer a viable solution for privacy-preserving image classification in smart building applications, solving the long-standing issue of accuracy loss in encrypted domains.