An Empirical Study of Validating Synthetic Data for Text-Based Person Retrieval

1. Problem Statement

Text-Based Person Retrieval (TBPR) aims to identify a specific individual from a gallery of images using a textual description. While research in this field is advancing, it faces two critical bottlenecks:

• Privacy Concerns: Mainstream approaches rely on real-world surveillance images, raising significant privacy issues.
• Annotation Burden: Training requires massive datasets with manual, high-quality textual annotations, which are labor-intensive and expensive to produce.

Existing attempts to use synthetic data often rely on real data as a foundation (e.g., using real images to generate text or vice versa), inheriting the same privacy and dependency limitations. There is currently no systematic study on the feasibility of using purely synthetic data (without any real data access) or understanding the effectiveness boundaries of synthetic data across different data availability scenarios.

2. Methodology

The authors propose a unified data synthesis pipeline capable of operating entirely without real person data. The framework consists of four main stages:

A. Inter-class Image Generation (Creating New Identities)

• Automatic Prompt Construction: Instead of using existing annotations, the authors design a strategy to generate diverse prompts automatically.
  • Plain Template: A structured template (e.g., {age} {gender} person with {hair}...) populated with elements from pre-defined descriptor lists.
  • LLM-Extended Prompts: Rough templates (covering appearance, profession, location, state) are fed into a Large Language Model (Qwen2) to expand them into diverse, detailed descriptions.
• Image Synthesis:
  • Scenario 1 (No Real Data): Uses Stable Diffusion (SD) with offline prompts and random seeds.
  • Scenario 2 (Limited Data): Fine-tunes SD using a specific loss function (reconstruction loss + class-specific prior preservation loss) on a few real pairs (e.g., 8 pairs) to learn the domain style while preserving generation diversity.
  • Scenario 3 (Abundant Data): Fine-tunes SD with LoRA on large real datasets.

B. Intra-class Image Augmentation (Creating Variations)

To increase data diversity for the same identity, the authors employ text-driven image editing on the generated images using cross-attention manipulation in Stable Diffusion:

• Local Editing: Modifies specific regions (e.g., changing background from "street" to "airport") by swapping local prompt tokens.
• Global Editing: Alters the overall style (e.g., "oil painting") by adding global descriptors.
• Non-rigid Editing: Changes posture (e.g., "running") using mutual cross-attention mechanisms to preserve structure while altering pose.
• Dimensions: Background, Weather, Style, and Posture.

C. Automatic Text Generation

• Multimodal Large Language Models (MLLMs): Models like InternVL and MiniCPM are used to generate textual descriptions for the synthetic images.
• Variable Instruction: To avoid structural homogeneity, the system uses a pool of 121 templates. The instruction randomly selects a template style for each image, ensuring the generated text varies in structure and phrasing, mimicking real-world annotation diversity.

D. Data Denoising & Robust Learning

• Denoising: Filters out synthetic images with incomplete body parts (using YOLOv8 keypoint detection) and removes irrelevant text noise.
• Robust Training: The baseline model (TBPS-CLIP with EVA-CLIP backbone) employs noise-robust strategies: label smoothing, Gaussian Mixture Models (GMM) to identify noisy data, and channel masking.

3. Key Contributions

1. First Purely Synthetic TBPR Pipeline: The paper presents the first pipeline capable of generating large-scale TBPR datasets (images + text) without any access to real person data, addressing privacy and annotation constraints.
2. Comprehensive Empirical Study: The authors systematically evaluate synthetic data effectiveness across three distinct scenarios:
   • S1 (No Data): Purely synthetic training.
   • S2 (Limited Data): Training with minimal real data (e.g., 8 pairs) augmented by synthetic data.
   • S3 (Abundant Data): Using synthetic data as a complement to large real datasets.
3. Open-Source Release: The code and the generated large-scale synthetic datasets (SynTBPR-S1, S2, S3) are released to the community.

4. Experimental Results

The study was conducted on three standard benchmarks: CUHK-PEDES, ICFG-PEDES, and RSTPReid.

• Performance in Data-Scarce Scenarios (S1 & S2):
  • In the No Data (S1) scenario, training solely on 1.5M synthetic samples achieved a Rank-1 of ~54% on CUHK-PEDES, a massive improvement over the baseline (22%) and comparable to methods trained on real data.
  • In the Limited Data (S2) scenario (8 real pairs), the pipeline successfully bootstrapped performance, achieving results comparable to or better than models trained on full real datasets.
• Performance in Data-Rich Scenarios (S3):
  • Adding synthetic data to abundant real data provided marginal but consistent gains (e.g., +3% Rank-1 on average), acting as effective regularization.
• Comparison with SOTA:
  • The proposed method (SynTBPR) outperformed or matched state-of-the-art methods that rely on pre-training with real-data-derived synthetic datasets (like MALS, LUPerson).
  • It demonstrated strong cross-domain capabilities, successfully transferring a model trained on synthetic target data to real target environments.
• Ablation Studies:
  • Prompt Quality: Combining plain and LLM-extended prompts yielded the best results.
  • Editing: Background and weather editing provided the most significant performance boosts; posture editing was less effective due to quality issues.
  • Text Generation: Variable instructions produced higher-quality, more diverse text than constant instructions.

5. Significance and Impact

• Privacy Preservation: By proving that high-performance TBPR models can be trained without real surveillance images, this work offers a viable solution for privacy-sensitive applications (e.g., missing person tracking in restricted areas).
• Cost Reduction: It eliminates the need for expensive manual annotation, replacing it with automated generation pipelines.
• Scalability: The pipeline allows for the generation of infinite variations of person data, enabling training in rare environments (e.g., heavy snow, specific lighting) where real data is scarce.
• Paradigm Shift: The study challenges the assumption that real data is a prerequisite for TBPR, establishing a new paradigm where synthetic data can serve as a standalone replacement or a powerful augmentation tool.

Conclusion: The paper validates that high-quality synthetic data, generated via a unified pipeline of automatic prompting, image editing, and MLLM-based text generation, is a feasible and effective alternative to real data for Text-Based Person Retrieval, particularly in privacy-constrained and data-scarce environments.