💻 computer science

Thermal-Only Crowd Counting with Deployment-Time Privacy Protection

This paper proposes a privacy-preserving thermal-only crowd counting framework that leverages depth-to-RGB diffusion models to extract discriminative features, achieving competitive performance against RGB-Thermal fusion methods while eliminating the need for continuous RGB capture during inference.

Original authors: Yifei Qian, Zhongliang Guo, Chun Tong Lei, Bowen Deng, Chun Pong Lau, Xiaopeng Hong, Michael P. Pound

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yifei Qian, Zhongliang Guo, Chun Tong Lei, Bowen Deng, Chun Pong Lau, Xiaopeng Hong, Michael P. Pound

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Counting People Without Getting Caught

Imagine you are trying to count how many people are in a crowded park. Usually, security cameras use RGB cameras (the kind that take normal color photos, like your phone). But there are two big problems with this:

Privacy: If you take a color photo, you can see people's faces, clothes, and identities. In public spaces, people don't always want to be filmed in high definition.
The "Mismatch" Problem: Some systems try to use two cameras at once: one color (RGB) and one heat-sensing (Thermal). They hope to combine the best of both worlds. However, the two cameras are rarely perfectly aligned. It's like trying to stitch together two maps of the same city where one is slightly shifted to the left and the other is tilted. This "misalignment" makes the computer confused and less accurate.

The Solution: The "Thermal-Only" Detective

The authors of this paper, TDCount, propose a clever new way to count people. They say: "Let's forget the color camera entirely during the actual counting."

Instead, they use a Thermal camera (which sees heat signatures, like a ghostly outline of a person) as the only input when the system is running. This solves the privacy issue because thermal images look like glowing blobs; you can't recognize faces or specific people.

How It Works: The "Ghost Translator"

Thermal images are tricky. A person looks like a blob of heat, but so does a hot car engine or a warm bench. It's hard for a computer to tell them apart just by looking at the heat.

To fix this, the authors use a Digital Translator (a type of AI called a Diffusion Model). Here is the analogy:

The Input: The system takes the "blobby" thermal image.
The Bridge: It first guesses what the "depth" (how far away things are) looks like based on the heat.
The Translator: It feeds this depth map into a pre-trained AI that knows how to turn a "depth sketch" into a "color photo."
- Think of it like this: Imagine you have a rough pencil sketch of a crowd. You feed this sketch into a magical artist who knows how to paint a realistic crowd scene.
The Secret Sauce: The system doesn't actually show you the painted color photo. It only uses the mathematical "understanding" the artist gained while painting to help the thermal camera understand the crowd better.

By using this "translator," the thermal camera learns to distinguish between a person and a hot car, even though it never actually sees the color photo.

The "One-Step" Trick

Usually, these AI translators take many steps to refine an image (like sketching, then shading, then coloring). The authors discovered something surprising: The first step is actually the best.

The Analogy: Imagine you are trying to guess the shape of a hidden object by feeling it.
- Step 1: You feel the general outline. It's rough, but it tells you exactly where the object is.
- Step 2 & 3: You start trying to guess the texture and color. In doing so, you might start "hallucinating" details that aren't there, or you might get distracted by the texture and lose track of the shape.
The Result: The authors found that stopping after the very first step gave the most accurate count. Waiting for the AI to "refine" the image actually made the counting worse because it introduced errors.

The Results: Better than the Two-Camera Team

The team tested their "Thermal-Only" system on two famous datasets (RGBT-CC and DroneRGBT).

The Competition: They compared their system against the best existing methods that use both color and thermal cameras.
The Outcome: Their thermal-only system performed just as well, and sometimes better, than the systems using two cameras.
The Win: They achieved high accuracy without ever capturing a single color photo of a person's face during the actual counting process.

Why This Matters (According to the Paper)

The paper emphasizes that this is a privacy-first approach.

Training vs. Deployment: They did use color photos once to teach the "Translator" AI how to understand crowds. But once the AI is trained, the system is deployed in the real world using only thermal cameras.
No Continuous Surveillance: In real-world surveillance, the biggest privacy risk is the camera constantly recording high-definition video of people's faces. This system eliminates that risk entirely by never recording color video in the field.

Summary

The paper introduces TDCount, a system that counts crowds using only heat signatures. It uses a clever AI trick to "imagine" what the scene looks like in color to help the heat camera understand the scene better, but it never actually records or stores color images. This makes it highly accurate while protecting people's privacy, solving the problem of "misaligned" cameras, and proving you don't need a color lens to count a crowd effectively.

Technical Summary: Thermal-Only Crowd Counting with Deployment-Time Privacy Protection

Problem Statement
Crowd counting is essential for public safety and urban planning but faces significant challenges in adverse lighting conditions and privacy preservation. While RGB-Thermal (RGB-T) fusion methods have improved robustness over single-modal RGB approaches, they suffer from two critical limitations. First, they rely on continuous RGB capture during deployment, which raises severe privacy concerns by recording identifiable features (e.g., faces) in public spaces. Second, existing fusion strategies implicitly assume perfect alignment between RGB and thermal sensors; in practice, spatial and temporal misalignments due to sensor placement and acquisition timing degrade fusion performance. Current attempts to solve alignment issues yield only incremental gains, suggesting that circumventing the need for RGB data entirely may be a more tractable path.

Methodology: TDCount
The authors propose TDCount, the first framework designed for thermal-only inference in RGB-T crowd counting scenarios. The method eliminates RGB dependency at inference time while leveraging cross-modal priors learned during a controlled training phase. The core components are:

Depth-to-RGB Diffusion Bridge: To address the inherent ambiguity of thermal signatures (e.g., difficulty distinguishing humans from other heat sources), the framework employs a pre-trained Latent Consistency Model (LCM) adapted via ControlNet. A depth estimation model generates a depth map ( $D_T$ ) from the input thermal image ( $T$ ). This depth map conditions the ControlNet to extract discriminative features ( $F_{TD}$ ) that bridge the thermal domain to the semantic richness of RGB data without requiring actual RGB input.
Deterministic Latent Input: Unlike standard diffusion models that sample noise stochastically, TDCount fixes the initial noisy latent ( $z_T$ ) as a shared constant across all training and inference iterations. This ensures that the feature extraction process is fully deterministic and reproducible, preventing the "many-to-one" mapping issue where identical inputs yield inconsistent features, which would otherwise render the regression loss landscape pathological.
Single-Step Denoising Strategy: The authors investigate the number of denoising steps required for feature extraction. They demonstrate that single-step LCM denoising yields features most faithful to the structural content of the depth conditioning signal. Multi-step denoising progressively decouples features from the conditioning input, prioritizing perceptual refinement over structural fidelity and accumulating errors that degrade counting accuracy.
Prototype Alignment Loss ( $L_{PA}$ ): To align the heterogeneous thermal features ( $F_T$ ) with the diffusion-derived semantic space ( $F_{TD}$ ), the framework introduces a text-guided prototype alignment loss. This loss aligns thermal features with CLIP-encoded textual count prototypes (e.g., "There are $c$ people in the area"), facilitating effective integration of the two feature streams.

Key Contributions

Privacy-Centric Framework: TDCount is the first framework to enable thermal-only inference in RGB-T crowd counting, eliminating the primary privacy exposure vector (continuous RGB capture) in real-world surveillance deployments.
LCM-Based Feature Learning: The paper introduces an LCM-based feature learning framework and reveals a counter-intuitive finding: the initial single-step denoising step provides more discriminative features for counting than multi-step approaches, which accumulate structural errors.
Competitive Performance without RGB: The method achieves competitive performance against state-of-the-art dual-modality methods using only thermal input at inference, challenging the assumption that RGB data is indispensable for high-accuracy crowd counting.

Experimental Results
The framework was evaluated on two widely used datasets: RGBT-CC and DroneRGBT.

Performance: On the RGBT-CC dataset, TDCount outperformed previous RGB-T methods, improving GAME(0) by 0.85 and RMSE by 1.02 compared to the MC3Net baseline. On the DroneRGBT dataset, it established new state-of-the-art performance across all metrics, reducing GAME(0) by 0.26 and RMSE by 0.39 compared to the previous best method (RGBT-Booster).
Ablation Studies:
- Depth Processing: Replacing the baseline thermal-only approach with the ControlNet-based depth processing significantly reduced errors (GAME(0) dropped from 17.05 to 10.62), confirming the value of diffusion-derived priors.
- Deterministic Input: Using a stochastic latent input caused the model to fail to converge (GAME(0) > 50), validating the necessity of the fixed latent design.
- Denoising Steps: Performance degraded monotonically as the number of denoising steps increased from 1 to 4, confirming that single-step extraction is optimal.
- Prototype Alignment: The proposed $L_{PA}$ outperformed standard cross-entropy loss, further reducing RMSE from 25.01 to 19.57.

Significance and Claims
The authors claim that TDCount demonstrates that a single, well-utilized modality (thermal) can match complex dual-modality systems through effective feature utilization, thereby avoiding the alignment challenges and privacy risks inherent to RGB-T fusion.

Crucially, the paper addresses privacy concerns regarding the use of RGB data during the training phase. The authors argue that the fine-tuned ControlNet does not constitute a privacy leakage channel because:

Architectural Limits: The VAE encoder in Stable Diffusion (8x downsampling) compresses fine-grained identity features (like faces in distant crowd shots) into latent tokens, making them unrecoverable.
Memorization Conditions: Diffusion models typically memorize training data only when samples are heavily duplicated; crowd counting datasets feature unique individuals, and the training uses a generic prompt ("a photo of a crowd of people"), preventing identity-specific conditioning.

Thus, the framework successfully separates the need for RGB data (used once offline for learning cross-modal priors) from the deployment phase (where only thermal data is captured), substantially reducing privacy exposure in public surveillance.