Unsafe2Safe: Controllable Image Anonymization for Downstream Utility

1. Problem Statement

Large-scale image datasets used for training computer vision models often contain sensitive information (e.g., faces, ID badges, personal documents, license plates, or demographic cues). Deep learning models have a tendency to memorize training data, making them vulnerable to privacy attacks such as model inversion and membership inference.

Existing anonymization methods suffer from significant limitations:

• Partial Protection: Most rely on detectors for specific regions (like faces) and apply blurring or masking, which often fails to catch privacy cues outside predefined regions (e.g., text on a poster or a specific uniform).
• Utility Loss: Aggressive redaction or inpainting often destroys the semantic structure and global layout of the image, degrading the performance of downstream tasks (classification, captioning, VQA).
• Lack of Control: Current generative approaches often lack fine-grained control over what to edit and how to rewrite it without compromising non-private semantics.

The core challenge is to create an anonymization pipeline that rewrites sensitive content to neutralize identity while preserving the global structure and task-relevant semantics necessary for model training.

2. Methodology: The Unsafe2Safe Pipeline

Unsafe2Safe is a fully automated, two-stage pipeline that combines Vision-Language Models (VLMs), Large Language Models (LLMs), and instruction-driven Diffusion editors.

Stage 1: Privacy Inspection and Textual Guidance Generation

This stage converts raw images into structured textual instructions for editing.

1. Privacy Inspection: A VLM (InternVL2.5) inspects images against a predefined set of privacy criteria (derived from VISPR). It flags images as "unsafe" if they contain sensitive content. The system is tuned for high recall to minimize false negatives (leakage).
2. Dual Captioning: For unsafe images, the VLM generates two captions:
   • Private Caption ($c_{priv}$): A full description including all sensitive attributes.
   • Public Caption ($c_{pub}$): A sanitized description that omits identity-specific details (e.g., names, specific demographics, text) but preserves the scene layout and non-sensitive objects.
3. Edit Instruction Generation: An LLM (Qwen3) takes the $c_{pub}$ and generates a structured Edit Instruction ($c_{edit}$). This instruction proposes specific, identity-neutral substitutions for sensitive elements (e.g., "change the person's hair to short black hair," "remove the license plate") while ensuring the scene's logic remains intact.

Stage 2: Safe Image Generation

This stage applies the textual guidance to generate the anonymized image.

1. Diffusion Editing: The pipeline uses text-guided diffusion editors (e.g., InstructPix2Pix, FlowEdit).
2. Dual-Conditioning: The editor is conditioned on both the Public Caption (to anchor global semantics and layout) and the Edit Instruction (to guide specific modifications).
3. Safe Cross Attention (Novelty): To prevent the model from overwriting non-sensitive regions (a common issue where instruction-following models change the whole image), the authors introduce a Safe Cross Attention module.
   • This module operates alongside standard cross-attention in the UNet transformer.
   • It processes embeddings from both $c_{pub}$ and $c_{edit}$ simultaneously.
   • Function: It reinforces the preservation of non-private semantics (via $c_{pub}$) while applying targeted edits to sensitive regions (via $c_{edit}$), ensuring structural fidelity.

3. Key Contributions

1. Unsafe2Safe Framework: A scalable, automated pipeline that integrates VLM-based privacy reasoning, LLM-generated edit instructions, and diffusion editing to remove identity-sensitive content without manual annotation.
2. Safe Cross Attention: A novel architectural module that enables fine-grained control, allowing the model to preserve global scene structure while performing localized, identity-neutral rewriting.
3. Unified Evaluation Suite: The authors propose a comprehensive evaluation framework covering four dimensions:
   • Quality: Visual realism (CLIP similarity).
   • Cheating: Unintended copying of pixel-level cues (SSIM, LPIPS).
   • Privacy: Face similarity, text similarity, demographic predictability (Race Entropy), and VLM-based privacy scores.
   • Utility: Downstream task performance (classification accuracy, captioning, VQA).
4. Dataset Construction: The creation of a large-scale, privacy-safe dataset (triplets of private caption, public caption, edit instruction) and the release of tools for generating such datasets.

4. Experimental Results

The method was evaluated on MS-COCO, Caltech101, MIT Indoor67, and OK-VQA.

• Privacy Protection:
  • Face & Text Similarity: Unsafe2Safe significantly reduced face similarity (FaceSim) and text similarity (TextSim) compared to baselines like DeepPrivacy2 and FaceAnon.
  • Demographic Diversity: It increased Race Entropy, indicating a more uniform and less identity-specific demographic distribution in the generated images.
  • VLM Score: A VLM judge assigned significantly higher scores (indicating better privacy removal) to Unsafe2Safe outputs compared to baselines.
• Downstream Utility:
  • Models trained on Unsafe2Safe data achieved accuracy comparable to (and in some cases exceeding) models trained on raw data.
  • For example, on Caltech101, the accuracy remained near 94-95%, matching raw data performance.
  • In generative tasks (Image Captioning and VQA), BLEU-4, CIDEr, and VQA accuracy were preserved, demonstrating that task-relevant semantics were not lost.
• Fine-Tuning Benefits: Fine-tuning the diffusion editor (InstructPix2Pix) on the automatically generated triplets further improved both privacy metrics (lower FaceSim) and utility (higher accuracy).
• Comparison to Baselines: Unlike face-only anonymizers that leave background privacy cues (e.g., text on signs) intact, Unsafe2Safe successfully anonymized non-human elements and background details while maintaining the overall scene composition.

5. Significance

• Scalability: Unsafe2Safe provides a principled, automated solution for constructing large, privacy-safe datasets without the prohibitive cost of manual auditing.
• Balancing Privacy and Utility: It addresses the critical trade-off in privacy-preserving AI, proving that high-fidelity anonymization does not require sacrificing the utility of the data for training downstream models.
• Generalizability: By moving beyond fixed detectors to multimodal reasoning, the framework can handle open-domain images and diverse privacy cues (beyond just faces), making it applicable to a wider range of real-world data scenarios.
• Community Impact: The release of the pipeline and generated datasets supports reproducibility and encourages the adoption of privacy-safe practices in computer vision research.

In conclusion, Unsafe2Safe represents a shift from "removing" sensitive data to "rewriting" it intelligently, ensuring that datasets remain useful for AI training while effectively neutralizing privacy risks.