💻 computer science

Privacy-Preserving Person Re-Identification from Temporal Sequences with Transformer and Hungarian Optimization

This paper proposes a privacy-preserving person re-identification framework that utilizes depth images and a Transformer architecture with temporal sequences, optimized via the Hungarian algorithm and batch hard triplet loss, to achieve competitive performance on top-view datasets while protecting individual identity.

Original authors: Raphaël Delécluse, Hazem Wannous, Laurent Guimas

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Raphaël Delécluse, Hazem Wannous, Laurent Guimas

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find a specific person in a busy train station. Usually, security cameras take photos of people's faces and clothes (RGB images) to do this. But that feels a bit like spying; it captures too much personal detail, and if the lighting changes or someone wears a hat, the system might get confused.

This paper proposes a smarter, more private way to solve the "Who is who?" problem using depth images (which see the world in 3D shapes rather than colors) and a bit of mathematical magic.

Here is the breakdown of their solution using simple analogies:

1. The "Shadow" Camera (Privacy First)

Instead of taking a photo of a person's face, the system uses a special camera that only sees silhouettes and 3D shapes.

The Analogy: Think of it like looking at a person's shadow on a wall. You can see how tall they are, how wide their shoulders are, and how they walk, but you can't see their eyes, nose, or what they are wearing.
Why it matters: Because it doesn't capture faces, it protects people's privacy. It's perfect for public places like train stations where you want to track movement without spying on individuals.

2. The "Movie Clip" vs. The "Snapshot" (Temporal Sequences)

Old systems tried to identify people by looking at a single frozen picture (a snapshot). This paper says, "That's not enough!" Instead, they feed the computer a short movie clip of the person walking.

The Analogy: Imagine trying to recognize a friend. If you only see a still photo of them standing still, it might be hard if they are wearing a coat. But if you see a 5-second video of them walking, you recognize them by their unique gait (how they walk).
The Tech: They use a Transformer (a type of AI brain) to watch these "movie clips" and learn the rhythm of a person's movement, not just what they look like.

3. The "Matchmaker" (Hungarian Optimization)

Once the system has watched people enter a building and later exit, it has two lists of "shadows": one list of people coming in, and one list of people going out. The goal is to match the person who entered to the person who left.

The Problem: If you just look for the closest match for each person individually, you might make mistakes. For example, if two people look very similar, the system might accidentally swap them.
The Solution: They use the Hungarian Algorithm.
The Analogy: Imagine a dance hall where everyone needs a partner. If you just let people pick the person closest to them, you might end up with two people fighting over the same partner, or two people left alone. The Hungarian Algorithm is like a super-smart dance instructor who looks at everyone in the room at once and assigns partners in a way that minimizes the total "awkwardness" for the whole group. It ensures no one is matched twice and the overall pairing is the best possible.

4. The "Hard Trainer" (Batch Hard Triplet Loss)

To teach the AI to be good at this, they use a special training method called Batch Hard Triplet Loss.

The Analogy: Imagine a coach training an athlete. Instead of giving them easy practice drills, the coach picks the toughest opponents to fight against.
- The "Anchor" is the athlete.
- The "Hard Positive" is a teammate who looks almost exactly like the athlete (hard to tell apart).
- The "Hard Negative" is an opponent who looks very similar but is actually different.
By forcing the AI to struggle with these hardest examples, it learns to spot the tiny differences that matter, making it much smarter.

What Did They Find?

The researchers tested this system on three different datasets (collections of video data) from top-down cameras.

The Result: Even though they only used the "shadow" (depth) images and ignored colors/faces, their system was almost as good as systems that use full-color photos.
The Magic Boost: When they added the "Matchmaker" (Hungarian Algorithm) to the mix, the accuracy jumped significantly. In some tests, they achieved 100% precision, meaning they matched every single person correctly without any swaps.

The Bottom Line

This paper shows that you don't need to invade people's privacy (by filming their faces) to track them in public spaces. By using 3D shapes, watching how they move over time, and using a smart matching algorithm to pair them up, you can get highly accurate results while keeping everyone's identity safe. It's like recognizing a friend by their walk and silhouette rather than their face.

Technical Summary: Privacy-Preserving Person Re-Identification from Temporal Sequences with Transformer and Hungarian Optimization

Problem Statement

Person Re-Identification (Re-ID) is a critical task in surveillance and human behavior analysis, traditionally relying on RGB images. However, RGB-based methods face significant challenges: they raise privacy concerns by capturing identifiable features like faces, and they are highly sensitive to lighting variations, occlusions, and viewpoint changes. While depth imagery offers a privacy-preserving alternative by obscuring facial features and providing lighting-invariant 3D structural data, Re-ID using only depth data remains under-explored, particularly in top-view configurations where distinguishing features are limited. Furthermore, existing approaches often struggle to associate individuals across different temporal sequences (e.g., entry vs. exit) without relying on local nearest-neighbor matching, which can be suboptimal in noisy or crowded scenarios.

Methodology

The authors propose a novel Re-ID framework that prioritizes privacy by utilizing depth images, specifically in a top-view configuration. The methodology integrates three core components:

Temporal Sequence Processing with Transformers:
Instead of processing single frames, the model accepts temporal sequences of Regions of Interest (RoIs) extracted from depth images (and RGB images for comparison).
- Architecture: The system employs a dual-encoder approach (for RGB-D) or a single encoder (for Depth-only). The RGB stream uses a ResNet-50 pre-trained on ImageNet, while the depth stream uses a smaller, non-pretrained ResNet variation with a single input filter.
- Feature Fusion: Embeddings from both modalities are concatenated to form a joint representation ( $N \times 2D$ ).
- Temporal Modeling: A Transformer encoder processes these concatenated sequences to capture dynamic movement patterns and temporal dependencies. The model utilizes self-attention to weigh the significance of each frame without the complexity of cross-attention mechanisms.
- Pooling: A mean pooling operation is applied to the sequence embeddings to generate a final robust representation, enhancing resilience to noise and occlusions.
Discriminative Feature Learning:
To improve feature separation, the authors employ Batch Hard Triplet Loss. Unlike standard triplet loss, this strategy selects the hardest positive (most similar same-identity sample) and hardest negative (most dissimilar different-identity sample) within each mini-batch.
- In the RGB-D configuration, the total loss is a sum of three components: Transformer Loss (sequence level), Depth Loss, and RGB Loss. This ensures balanced learning across modalities.
Global Optimization via Hungarian Algorithm:
The Re-ID task is formulated as an assignment problem where incoming sequences (query) must be matched to outgoing sequences (gallery).
- A distance matrix is constructed using Euclidean distances between embeddings.
- The Hungarian algorithm is applied to minimize the global cost of matching, ensuring a one-to-one correspondence that avoids conflicts (e.g., multiple queries assigned to the same gallery item). This global optimization replaces local nearest-neighbor matching to improve robustness against noise and similar body shapes.

Key Contributions

The paper outlines three primary contributions:

Depth-Only Re-Identification: The authors demonstrate that person Re-ID can be successfully achieved using only depth images. This modality inherently preserves privacy by omitting facial and texture details, making it suitable for public spaces. The study shows that depth-based models can achieve competitive performance when combined with appropriate feature extraction and matching techniques.
Temporal Sequence Utilization: By processing sequences of frames rather than individual snapshots, the approach captures temporal dynamics and movement patterns (gait, posture changes). This sequential processing provides more robust feature representations, significantly improving accuracy, especially when individuals have similar static appearances or when occlusions occur.
Hungarian Algorithm Integration: The integration of the Hungarian algorithm optimizes the association process across multiple views. By solving the global assignment problem, the method ensures precise matching even in complex scenarios with noise or similar body shapes, outperforming standard local matching approaches.

Experimental Results

The proposed method was evaluated on three top-view and frontal-view datasets: TVPR2, GODPR, and BIWI RGBD-ID.

Performance Metrics: Evaluation was conducted using Cumulative Matching Characteristics (CMC), Mean Average Precision (mAP), and Precision.
Depth vs. RGB-D:
- On TVPR2, the Depth-only model achieved a Rank-1 accuracy of 88.4% (Precision 94.6%), while the RGB-D model reached 99.5% (Precision 100.0%).
- On GODPR, Depth-only achieved 78.6% Rank-1, compared to 100.0% for IR-D.
- On BIWI RGBD-ID (frontal view), Depth-only achieved 60.7% Rank-1, while RGB-D reached 96.4%.
Impact of Hungarian Optimization: The application of the Hungarian algorithm significantly improved Precision across all datasets. Notably, for the GODPR and BIWI datasets, the algorithm enabled 100.0% Precision for both Depth-only and RGB-D models, despite lower Rank-1 scores in the depth-only configurations. This highlights the algorithm's ability to resolve global matching conflicts effectively.
State-of-the-Art Comparison: The proposed RGB-D model outperformed existing methods (e.g., CMOT, SeSAME, HRN) on all three datasets. For instance, on TVPR2, the proposed method achieved 99.5% Rank-1 (100.0% with Hungarian), surpassing the previous best of 98.8%. On BIWI, it improved Rank-1 from 77.8% (CMOT) to 96.4%.
Cross-Dataset Evaluation: When training on TVPR2 and testing on unseen datasets (GODPR, BIWI), the Depth-only model showed robustness, particularly on GODPR where it outperformed the RGB-D model (likely due to modality mismatch with IR data). The Hungarian algorithm consistently boosted performance in these cross-dataset scenarios.

Significance and Claims

The paper claims that its approach offers a practical, privacy-preserving solution for person Re-ID in applications where individual anonymity is paramount, such as transport hubs and public squares. By leveraging depth data, the system avoids capturing sensitive facial information while still extracting sufficient geometric and anthropometric features for identification.

The authors emphasize that while depth-only models face inherent limitations in distinguishing individuals with similar body shapes compared to RGB models, these limitations can be effectively mitigated through temporal sequence processing and global optimization via the Hungarian algorithm. The results demonstrate that depth-only Re-ID can achieve competitive performance, particularly when the goal is to correlate entry and exit events in a privacy-sensitive manner. The work suggests that this combination of modalities and optimization techniques provides a viable path for deploying Re-ID systems in real-world environments without compromising individual privacy.

Future work is identified as focusing on real-time deployment, scaling to larger datasets, and integrating additional modalities such as thermal imaging or multi-camera data.