Leveraging Prior Knowledge of Diffusion Model for Person Search

Technical Summary: DiffPS (Diffusion Prior Knowledge for Person Search)

Problem Statement
Person search involves the joint tasks of detecting pedestrians in uncropped scene images and re-identifying (re-ID) them against a query. Existing state-of-the-art methods predominantly rely on ImageNet pre-trained backbones (e.g., ResNet50, ConvNeXt) and share a single backbone feature stream for both detection and re-ID. This approach presents two critical limitations:

1. Suboptimal Feature Representation: ImageNet backbones are trained for category-level recognition on images with single dominant objects and simple backgrounds. Consequently, they struggle to capture the precise localization cues and instance-level discriminative features required for person search in complex, cluttered scenes with overlapping individuals.
2. Optimization Conflict: Sharing a backbone for two fundamentally different objectives—detection (distinguishing humans from background) and re-ID (distinguishing specific identities)—creates conflicting gradient updates. This forces the shared parameters to learn contradictory representations, hindering convergence and degrading performance for both sub-tasks.

Methodology
The authors propose DiffPS, a framework that leverages the prior knowledge of a pre-trained diffusion model (specifically Stable Diffusion v2-1) to resolve these issues. Instead of fine-tuning the backbone, DiffPS freezes the pre-trained diffusion model and extracts task-specific features, thereby eliminating the optimization conflict. The framework is built upon an analysis of four key properties of diffusion priors: text-conditioning, timestep sensitivity, hierarchical structure, and shape bias.

Based on this analysis, DiffPS integrates three specialized modules:

1. Diffusion-Guided Region Proposal Network (DGRPN):

   • Goal: Enhance person localization.
   • Mechanism: Leverages the strong alignment between diffusion features and text embeddings. It extracts cross-attention maps associated with the "person" token from the UNet's up-stage. These maps are thresholded to identify high-confidence regions, converted into Gaussian distributions, and used to modulate the detection-specific feature map. This guides the Region Proposal Network (RPN) to focus on relevant person regions while suppressing background noise.

2. Multi-Scale Frequency Refinement Network (MSFRN):

   • Goal: Mitigate the "shape bias" inherent in diffusion models, where low-frequency (global structure) components often dominate over fine-grained textures.
   • Mechanism: Extracts multi-scale feature maps from the up-stage of the UNet. It employs a hierarchical frequency decomposition strategy using Discrete Wavelet Transform (DWT) to separate features into low-frequency and high-frequency sub-bands. A Sub-band Refinement Block (SRB) specifically enhances the high-frequency components (crucial for identity discrimination) before reconstructing the features via Inverse DWT. This ensures the retention of fine-grained details necessary for re-ID.

3. Semantic-Adaptive Feature Aggregation Network (SFAN):

   • Goal: Leverage the semantic alignment between diffusion features and text to create robust person representations.
   • Mechanism: Computes cosine similarity between the refined feature map and pre-defined text embeddings (e.g., "head," "shirts," "pants," "shoes") derived from the CLIP text encoder. These similarities generate spatial semantic maps that highlight relevant body parts. The network aggregates these part-based features using learnable weights, effectively suppressing background noise and occlusions while emphasizing informative body regions.

Key Contributions

• Novel Framework: The paper introduces DiffPS, the first work to leverage the prior knowledge of pre-trained diffusion models specifically for the person search task.
• Resolution of Optimization Conflict: By freezing the diffusion backbone and utilizing task-specific branches, DiffPS completely decouples the optimization of detection and re-ID, preventing the gradient interference that plagues existing shared-backbone methods.
• Specialized Modules: The proposal of DGRPN, MSFRN, and SFAN effectively adapts diffusion priors to the specific demands of localization and fine-grained identity recognition.
• State-of-the-Art Performance: The framework achieves new state-of-the-art results on standard benchmarks without requiring the backbone to be fine-tuned for the specific task.

Experimental Results
The authors evaluated DiffPS on the CUHK-SYSU and PRW datasets.

• CUHK-SYSU: DiffPS achieved 97.8% mAP and 98.4% Top-1 accuracy, surpassing previous methods (e.g., SeqNeXt+GFN, SEAS) by significant margins.
• PRW: The model achieved 62.0% mAP and 91.0% Top-1 accuracy, outperforming the second-best method by 1.5% mAP.
• Ablation Studies:
  • Conflict Resolution: Unlike decoupled methods like DMRNet that still suffer performance drops when tasks are jointly optimized, DiffPS maintained identical performance whether tasks were optimized jointly or separately, confirming the elimination of gradient conflict.
  • Module Efficacy: Removing MSFRN or SFAN resulted in measurable drops in mAP and Top-1, validating the necessity of frequency refinement and semantic aggregation.
  • Timestep Analysis: The study confirmed that features extracted at earlier timesteps (or t=0) preserve the fine-grained details and semantic structure most beneficial for person search, unlike later timesteps where noise dominates.

Significance
The paper claims that DiffPS demonstrates the potential of pre-trained diffusion models as powerful, general-purpose backbones for vision tasks beyond image generation. By fully exploiting the rich spatial context and fine-grained details embedded in diffusion priors, the framework addresses the limitations of traditional ImageNet backbones. Furthermore, it establishes a new paradigm for multi-task learning where task-specific features can be extracted from a frozen, pre-trained model, completely resolving the inherent optimization conflicts found in shared-backbone architectures. This approach sets a new benchmark for person search performance while offering a more efficient training strategy that does not require extensive backbone fine-tuning.