💻 computer science

Coarse-to-Fine Hierarchical Alignment for UAV-based Human Detection using Diffusion Models

This paper introduces Coarse-to-Fine Hierarchical Alignment (CFHA), a three-stage diffusion-based framework that bridges the synthetic-to-real domain gap in UAV human detection by decoupling and aligning global style and local content while removing hallucinations, thereby significantly improving detection accuracy without requiring extensive real-world annotations.

Original authors: Wenda Li, Meng Wu, Liangzhao Chen, Sungmin Eum, Heesung Kwon, Qing Qu

Published 2026-03-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wenda Li, Meng Wu, Liangzhao Chen, Sungmin Eum, Heesung Kwon, Qing Qu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Video Game" vs. The "Real World"

Imagine you are teaching a robot to find people in a crowd using a drone. To teach it, you need thousands of pictures of people. But taking real photos from a drone is expensive, dangerous, and hard to label (you have to draw boxes around every person manually).

So, scientists use synthetic data. They create a "video game world" (like Grand Theft Auto or The Sims) where they can generate millions of perfect, labeled pictures of people for free.

The Catch: The robot learns to find people in the video game perfectly. But when you fly the drone in the real world, the robot fails miserably. Why? Because the video game looks too "plastic." The lighting is wrong, the textures are too smooth, and the shadows don't look real. This is called the Sim2Real Gap (Simulation to Reality gap).

The Old Way vs. The New Way

The Old Way (The "Filter" Approach):
Previous methods tried to fix this by slapping a "filter" on the video game images to make them look like real photos.

Analogy: Imagine taking a photo of a plastic toy car and running it through a photo editor to make it look like a real car.
The Problem: The filter makes the background look real, but it often smears the toy car. The wheels get blurry, the shape gets distorted, or the robot accidentally invents a "ghost car" that wasn't there. Since the robot needs to see tiny details (like a person's head from high up), these smears ruin its ability to detect them.

The New Way (CFHA):
The authors of this paper, Wenda Li and his team, built a three-step "repair shop" called CFHA (Coarse-to-Fine Hierarchical Alignment). Instead of just slapping a filter on, they fix the image in layers, like a master chef refining a dish.

The Three-Step "Repair Shop" (CFHA)

Think of the synthetic image as a rough sketch of a real scene. CFHA turns it into a masterpiece in three stages:

Stage 1: The "Global Style Transfer" (The Makeover)

What it does: It looks at the whole picture and changes the "vibe." It takes the lighting, color, and atmosphere from a few real drone photos and applies them to the video game image.
The Analogy: Imagine you have a black-and-white drawing of a forest. You take a photo of a real forest at sunset and use it to paint the sky, the grass, and the shadows on your drawing. Now your drawing looks like it was taken at sunset.
The Catch: While the background looks great, the tiny people in the drawing might still look a bit blurry or weird because the "paint" covered up their fine details.

Stage 2: The "Local Refinement" (The Zoom-In)

What it does: This is the secret sauce. The system zooms in on the tiny people (who are often just a few pixels big in drone shots). It uses a powerful AI (a Diffusion Model) to "hallucinate" (imagine) realistic details like skin texture, clothing folds, and hair, based on a text description of what a person should look like.
The Analogy: Remember the blurry people from Stage 1? Imagine a master artist taking a magnifying glass to those blurry blobs. They don't just paint over them; they carefully redraw the eyes, the shirt collar, and the shoes to make them look hyper-realistic, while leaving the background alone.
Why it matters: This ensures the robot can actually see the person, not just a blurry blob.

Stage 3: The "Hallucination Removal" (The Bouncer)

What it does: Sometimes, the AI gets too creative. In Stage 2, it might accidentally invent a person who isn't there, or make a person look like a monster. This stage acts as a strict bouncer. It checks every person in the image against a "mental list" of what real humans look like. If a person looks too weird or fake, it deletes them.
The Analogy: Imagine a bouncer at a club. He checks everyone's ID. If someone looks like a cardboard cutout or a weird alien, he kicks them out. Only the people who look 100% real get to stay in the final photo.

Why This Matters

The team tested this on four different real-world drone datasets. The results were impressive:

Before: The robot was confused and missed many people.
After: The robot became much sharper, finding people it previously missed.
The Result: They saw a 7.3% to 14.1% improvement in accuracy. In the world of AI, that's a huge jump.

The Takeaway

The paper teaches us that to make AI work in the real world, you can't just make things look real globally. You have to fix the big picture (lighting and color) AND the tiny details (the people themselves) separately.

If you try to fix everything at once, you mess up the details. But if you fix the background first, then zoom in to fix the people, and finally kick out the fake ones, you get a perfect training set. This allows robots to learn from video games and then successfully find people in real life, which is crucial for search-and-rescue missions, disaster response, and keeping our cities safe.

1. Problem Statement

The paper addresses the critical bottleneck in Unmanned Aerial Vehicle (UAV) human detection: the scarcity of annotated real-world data.

The Challenge: Training deep learning detectors on real UAV imagery is difficult due to the high cost of annotation and extreme variability in deployment conditions (altitude, viewpoint, lighting).
The Sim2Real Gap: While synthetic data offers a scalable alternative, models trained on it fail in the real world due to the "Sim2Real" domain gap. This gap consists of two distinct issues:
1. Global Style Gap: Differences in color, illumination, texture, and atmospheric conditions.
2. Local Content Gap: Discrepancies in object-level characteristics, such as instance density, spatial arrangement, and fine-grained details.
Limitations of Existing Methods: Current diffusion-based adaptation methods often require abundant real data for fine-tuning (impractical for few-shot scenarios) or fail to preserve the semantic structure of small objects (like distant humans), leading to "hallucinations" (artifacts) that degrade detector performance.

2. Methodology: Coarse-to-Fine Hierarchical Alignment (CFHA)

The authors propose CFHA, a three-stage diffusion-based framework designed to translate synthetic images into the target real domain while strictly preserving geometric labels. The method operates in a few-shot setting (using only ~20 real reference images).

Stage I: Global Style Transfer (GST)

Goal: Align macroscopic appearance (color, lighting, haze) between synthetic and real images without altering spatial structure.
Mechanism:
- Utilizes DDIM inversion to map synthetic images into the latent space of a pre-trained Stable Diffusion model.
- Employs a structure-aware cross-attention mechanism to extract style features from a few-shot real reference set.
- Uses Adaptive Instance Normalization (AdaIN) within the latent space to fuse style statistics with the original content features.
- Result: A globally styled image that matches the target domain's photorealism but retains the original bounding box annotations.

Stage II: Local Refinement (LR)

Goal: Restore fine-grained details to small human instances that are often blurred or distorted by global style transfer.
Mechanism:
- Crops object-centric patches from the GST output.
- Generates text prompts via automated image captioning (DAPE) to describe the content.
- Applies a super-resolution diffusion model (fine-tuned via LoRA) conditioned on these prompts.
- Performs a one-step reverse diffusion process to up-scale and refine the texture of the human instances while keeping the background frozen.
- Result: High-fidelity, photorealistic details for small human instances.

Stage III: Hallucination Removal (HR)

Goal: Filter out generated artifacts or structurally unrealistic human instances introduced during the generation process.
Mechanism:
- Leverages a pre-trained CLIP model to create a semantic filter.
- Distills the distribution of real human embeddings into a single prototypical reference vector ( $\mathbf{t}^*$ ), anchored by a text prompt (e.g., "a photo of a person taken from a drone").
- Calculates the cosine similarity between generated human crops and this reference vector.
- Stochastic Retention: Instances with low similarity scores are masked out and removed from the training set and annotations.
- Result: A clean, data-consistent training dataset free of spurious objects.

3. Key Contributions

Unified Global-Local Alignment: CFHA is the first framework to explicitly decouple and address both the global style gap and the local content gap in a hierarchical manner, recognizing that local inconsistencies (e.g., unnatural human density) are as critical as style shifts.
Few-Shot Diffusion Adaptation: The method operates effectively in a few-shot regime, requiring only a small set of real reference images, making it practical for data-scarce UAV scenarios.
Instance-Aware Refinement: By integrating super-resolution and semantic filtering, the framework specifically targets the preservation and enhancement of small, distant human objects, which are the primary failure point in UAV detection.

4. Experimental Results

The framework was evaluated on four public UAV benchmarks: Okutama-Action, Semantic Drone, VisDrone2018, and Manipal-UAV.

Performance Gains:
- CFHA consistently outperformed non-transformed baselines and existing state-of-the-art methods (CUT, IP-Adapter, PTL).
- Average Improvements: +7.3% improvement in mAP@50 and +4.1% in mAP@95 across the four datasets.
- Notable Achievement: On the Semantic Drone dataset, CFHA boosted mAP@50 from 18.6 (baseline) to 32.7 (+14.1).
Ablation Studies:
- GST alone actually degraded performance because global translation corrupted small object structures.
- Local Refinement (LR) was crucial for recovering performance by injecting structural realism.
- Hallucination Removal (HR) further refined accuracy by removing anomalous instances.
- Key Insight: The study revealed that patch-level realism (quality around the object) is a far better predictor of detection performance than global visual quality (measured by FID). In some cases, improving local realism slightly increased the global FID score but significantly improved detection accuracy.

5. Significance

This paper provides a robust solution to the data scarcity problem in UAV-based human detection. By moving beyond simple style transfer to a hierarchical, instance-aware alignment, CFHA ensures that synthetic data is not only visually realistic but also semantically consistent with real-world distributions. This approach enables the training of highly robust detectors in scenarios where collecting real-world annotated data is prohibitively expensive or impossible, with direct applications in search and rescue, disaster response, and urban surveillance.