Learning from Synthetic Data via Provenance-Based Input Gradient Guidance

1. Problem Statement

While synthetic data (generated via data mixing, simulators, or generative models) is effective for increasing training data diversity and reducing collection costs, existing methods suffer from a critical limitation: they do not explicitly teach the model which input regions are relevant to the target object.

• Indirect Robustness: Current synthetic learning methods (e.g., Mixup, CutMix, Diffusion-based augmentation) improve robustness only indirectly by diversifying the input distribution.
• Spurious Correlations: Because the model is not explicitly guided to ignore non-target regions, it often learns spurious correlations caused by synthesis biases, artifacts, or background co-occurrences (e.g., associating a specific background with a bird class).
• The Gap: Although the synthesis process inherently knows the "provenance" of each pixel (i.e., which part of the image came from the target object and which came from the background or another object), this information is discarded during standard training. The model is left to infer target regions solely from the final classification label, leading to reliance on irrelevant features.

2. Methodology

The authors propose a learning framework that utilizes provenance information (metadata generated during the synthesis process indicating the origin of each input element) as an auxiliary supervisory signal. This is achieved through Input Gradient Guidance.

Core Components

1. Provenance Extraction:

   • During data synthesis, a mask $M$ is generated that distinguishes between target regions (originating from the object of interest) and non-target regions (background, other objects, or synthesized artifacts).
   • Image Mixing (e.g., CutMix): The mask $M$ is the binary mixing mask itself.
   • Skeleton Mixing: A spatio-temporal mask indicates which skeleton joints originate from which input sequence.
   • Image Generation (e.g., Diffusion): A difference image between the original and edited image is computed. Regions with significant differences (edited areas) are marked as non-target ($I=0$), while unchanged regions are marked as target ($I=1$).

2. Input Gradient Guidance (The Provenance Loss, $L_{PG}$):

   • The method decomposes the input gradients ($\nabla_{\tilde{x}} f_y(\tilde{x})$) based on the provenance mask.
   • Objective: Suppress gradients in non-target regions to prevent the model from relying on them for prediction.
   • Soft Labels (Mixing Methods): For a synthetic sample combining image A and B, the loss penalizes gradients for class A appearing in regions from image B, and vice versa.
     $$L_{PG} = \| (\mathbf{1}-M) \odot \nabla_{\tilde{x}} f_A(\tilde{x}) + M \odot \nabla_{\tilde{x}} f_B(\tilde{x}) \|_2^2$$
   • Hard Labels (Generation Methods): For an edited image with a single label, the loss penalizes gradients in the edited (non-target) regions.
     $$L_{PG} = \| (\mathbf{1}-M) \odot \nabla_{\tilde{x}} f_y(\tilde{x}) \|_2^2$$
   • Total Loss: $L_{total} = L_{cls} + \alpha L_{PG}$, where $\alpha$ controls the regularization strength.

3. Key Contributions

1. Provenance-Guided Framework: The paper introduces a novel framework that leverages the inherent "source" information of synthetic data to explicitly guide the model to focus on target regions and ignore non-target regions.
2. Input Gradient Guidance: Unlike prior work that relies on indirect regularization or attention mechanisms requiring manual annotations, this method uses a differentiable loss function to directly suppress input gradients in irrelevant regions.
3. Modality and Method Agnosticism: The approach is applicable to various synthetic data generation techniques, including:
   • Image mixing (CutMix, ResizeMix).
   • Skeleton sequence mixing.
   • Generative image editing (Diffusion models).
4. Annotation-Free: The provenance masks are automatically derived from the synthesis process, requiring no additional human labeling.

4. Experimental Results

The method was evaluated across three distinct tasks and modalities:

• Weakly Supervised Object Localization (CUB-200-2011):

  • Task: Predicting bounding boxes using only image-level labels.
  • Result: Integrating the method with CutMix improved mean MaxBoxAccV2 from 62.3% (baseline) to 65.1% (VGG16) and from 91.5% to 92.1% (SAT/Transformer).
  • Visualization: Grad-CAM visualizations showed that the proposed method successfully suppressed attention on background/synthetic regions, focusing strictly on the bird.

• Weakly Supervised Spatio-Temporal Action Localization (UCF101-24):

  • Task: Localizing actions in video using skeleton sequences.
  • Result: Using BatchMix on skeleton sequences, the method improved Average Precision (AP) from 38.0% (Mixing baseline) to 39.7%.
  • Significance: Demonstrated effectiveness in the spatio-temporal domain, ensuring the model attends to action-relevant skeleton parts rather than static background joints.

• Image Classification (CUB, iWildCam, Waterbirds):

  • Task: Fine-grained classification and robustness against domain shift/spurious correlations.
  • Result: When applied to ALIA (a diffusion-based augmentation method), the proposed method significantly boosted accuracy on the Waterbirds dataset (known for strong background correlations), improving accuracy from 71.4% to 80.7%. This confirms the method's ability to suppress spurious background correlations.

• Efficiency & Robustness:

  • Hyperparameter Tuning: The method did not degrade hyperparameter tuning efficiency.
  • Mask Precision: The method remained robust even when the provenance mask was slightly imprecise (e.g., via dilation/erosion), with performance drops staying below 1% for moderate errors.

5. Significance

This work addresses a fundamental flaw in synthetic data learning: the reliance on the model to implicitly learn what is relevant. By explicitly utilizing provenance information to constrain input gradients, the authors enable models to:

1. Directly learn discriminative features for target objects rather than relying on indirect data augmentation effects.
2. Mitigate spurious correlations caused by synthetic artifacts or background biases, which is crucial for deploying robust AI in real-world scenarios (e.g., autonomous driving, robotics).
3. Generalize across modalities, proving that the concept of "knowing where the data came from" is a universal supervisory signal for improving representation learning, regardless of whether the data is mixed, simulated, or generated.

In summary, the paper transforms synthetic data from a mere source of volume into a source of structural supervision, significantly enhancing model robustness and localization accuracy without additional annotation costs.