Technical Summary: Personalized Generative Models for Contextual Debiasing

Problem Statement

Computer vision models often fail to generalize to uncommon scenarios due to contextual biases present in training datasets. In real-world data, objects frequently co-occur with specific contexts (e.g., skis appear with people, wine glasses on dining tables). Consequently, models learn to rely on these contextual cues rather than the object features themselves. This leads to poor performance when objects appear in rare or "out-of-context" scenarios (e.g., skis without a person).

While recent general-purpose vision-language models show some robustness, many industrial applications rely on compact, specialized classifiers that struggle with these biases. Existing solutions face significant limitations:

• Training Objective Modifications: Methods like reweighting or over-sampling are limited to the original dataset distribution and often degrade performance on common (in-context) samples.
• Generative Data Augmentation: Standard text-to-image (Txt2Img) models often generate images with viewpoints and backgrounds that diverge significantly from the target dataset, making them irrelevant for downstream tasks.
• Local Image Editing: Inpainting or local editing methods fail to handle complex object-scene interactions. For instance, removing a person from an image via inpainting often leaves objects (like a skateboard) floating unnaturally, as these methods do not adjust related elements to maintain physical plausibility.

Methodology: DecoupleGen

The authors propose Decoupling Contextual Patterns with Generations (DecoupleGen), a pipeline that leverages personalized text-to-image diffusion models to synthesize training data with rare contexts while preserving the visual fidelity of the original dataset.

The approach consists of three main stages:

1. Personalized Fine-Tuning:

   • Inspired by subject-driven generation methods (e.g., Textual Inversion, DreamBooth), the method fine-tunes a pre-trained diffusion model on individual images from the original dataset.
   • Instead of learning a token for a specific subject, the model learns new word tokens (e.g., [Vbackground], [Vhandbag]) that capture the visual details of specific image regions (background, biased object, contextual object).
   • The training objective combines a masked reconstruction loss and a cross-attention loss to ensure each token attends only to its corresponding region, faithfully reconstructing the visual details of the original image components.

2. Contextual Manipulation via Token Composition:

   • During inference, the model generates new images by composing the learned tokens with standard text prompts.
   • To create "debiased" samples, the pipeline removes common contextual tokens from the prompt. For example, to generate a handbag without a person, the prompt changes from "a photo of [Vhandbag] and [Vperson] and [Vbackground]" to "a photo of handbag at [Vbackground]."
   • This allows the model to reposition objects and adjust the scene to maintain physical plausibility (e.g., placing a wine glass on a table rather than letting it float) while retaining the original image's style and viewpoint.

3. Verification and Selection:

   • Not all generations are successful. A verification step uses a pre-trained semantic segmentation model to annotate generated images.
   • Samples are filtered to ensure they contain the target object but exclude the common contextual class (e.g., keeping the handbag but ensuring the person is absent). Only these successful, semantically meaningful samples are added to the training set.

Key Contributions

• Novel Framework: Introduction of DecoupleGen, which adapts diffusion model personalization to decouple objects from their common contexts, addressing the limitations of both local editing and standard Txt2Img generation.
• Visual Coherence: The method generates images that are visually aligned with the original dataset distribution, unlike standard generative models that often produce out-of-distribution samples.
• Physical Plausibility: By learning region-specific tokens, the method handles complex object-scene interactions (e.g., repositioning objects when a context is removed) better than local editing techniques.
• Verification Mechanism: A filtering pipeline ensures that synthesized data is relevant and useful for downstream recognition tasks.

Experimental Results

The authors evaluated DecoupleGen on object classification (NICO dataset) and multi-object recognition (COCO-Stuff dataset).

• NICO Dataset (Single-Object Classification):

  • DecoupleGen achieved an overall accuracy of 85.8% and a worst-group accuracy (WGA) of 49.2%.
  • This outperformed the standard classifier (85.3% Acc, 35.0% WGA) and matched the state-of-the-art worst-group accuracy of Feedback Guidance, while also improving overall accuracy—a feat previous methods failed to achieve simultaneously.

• COCO Dataset (Multi-Object Recognition):

  • Using 18k generated samples, DecoupleGen improved the Exclusive mAP (performance on images where the object appears without its common context) by 1.6% over Feedback Guidance (27.20% vs. 25.55%).
  • It also improved Cooccur mAP (performance on common contexts) and All mAP, demonstrating consistent gains across all metrics.
  • Compared to loss-function-based methods (e.g., reweighting), DecoupleGen provided superior performance without sacrificing accuracy on well-represented groups.

• Qualitative Analysis:

  • UMAP visualizations confirmed that DecoupleGen generations lie closer to the real data distribution than those from pre-trained Txt2Img models.
  • Visual comparisons showed DecoupleGen successfully handled complex interactions (e.g., removing a person while adjusting a wine glass's position) where inpainting and Feedback Guidance failed.

Significance and Claims

The paper claims that DecoupleGen offers a significant advancement in contextual debiasing by bridging the gap between generative capabilities and dataset distribution alignment.

• Superior Generalization: The method effectively improves model performance on underrepresented groups (uncommon contexts) without degrading performance on common groups, a common trade-off in debiasing.
• Distribution Alignment: The authors emphasize that their personalized fine-tuning process is crucial for ensuring generated samples remain relevant to downstream tasks, a key differentiator from off-the-shelf generative models.
• Scalability and Limitations: The authors acknowledge that fine-tuning on every image is computationally expensive and may struggle with very small objects or highly complex scenes where visual details are difficult to capture. They suggest future work should focus on improving scalability and addressing secondary biases that may be inadvertently introduced by the generative process.

Ultimately, the paper argues that personalized generative models provide a robust mechanism for augmenting training data with semantically meaningful, physically plausible, and distribution-aligned samples, thereby mitigating contextual biases in compact vision models.