DiffInf: Influence-Guided Diffusion for Supervision Alignment in Facial Attribute Learning

The paper introduces DiffInf, a self-influence-guided diffusion framework that identifies and generatively corrects annotation-inconsistent facial images to align visual content with labels, thereby improving classification performance without discarding data or sacrificing distributional coverage.

The Gist

Imagine you are trying to teach a robot how to recognize human emotions and age. You show it thousands of photos, but there's a problem: some of the photos have the wrong labels attached to them.

For example, you might show the robot a picture of a grumpy-looking 60-year-old man, but the label says "Happy 20-year-old." Or you show a picture of a teenager, but the label says "Elderly."

In the world of AI, these mistakes are called "noisy labels." Usually, when an AI gets confused by a bad label, it gets frustrated. It tries to force the picture to fit the wrong label, which messes up its brain (its "learning").

The Old Way: The "Delete" Button

Traditionally, when researchers found these confusing, bad examples, their solution was simple: Delete them.

Think of it like a teacher throwing out a student's homework because the student got the answer wrong. The teacher thinks, "If I remove this bad homework, the class will learn better."

But here's the catch: Sometimes, that "bad" homework is actually a very unique and interesting piece of work! Maybe the student drew a picture of a cat that looks like a dog. If you throw it away, you lose that unique perspective. In AI, deleting these photos means the robot forgets about rare faces, weird lighting, or unusual expressions. It makes the robot less smart about the real world.

The New Way: DiffInf (The "AI Editor")

This paper introduces a new method called DiffInf. Instead of hitting the "Delete" button, DiffInf acts like a smart photo editor or a tutor.

Here is how it works, step-by-step:

1. Finding the "Troublemakers" (Influence Functions)

First, the system looks at all the photos and asks: "Which of these confusing photos are causing the most trouble for the robot's brain?"

In math terms, this is called calculating "self-influence." Imagine a classroom where one student keeps asking a question that confuses everyone else. That student has "high influence." DiffInf finds these specific photos that are disproportionately messing up the learning process.

2. The "Fix-It" Workshop (Generative Correction)

Instead of kicking the student out of the class, DiffInf invites them to a workshop. It uses a powerful tool called a Diffusion Model (think of it as a magical painter that can redraw things while keeping the original style).

The system says to the robot: "Okay, this photo is labeled 'Happy,' but the face looks 'Sad.' Let's keep the person's identity (their nose, eyes, and bone structure) exactly the same, but let's gently tweak their expression to actually look happy."

It's like taking a photo of a person frowning and using Photoshop to gently lift the corners of their mouth, without changing their face so much that it looks like a different person.

3. The Result: A Better Class

Now, the robot gets to study the fixed photo instead of the confusing one.

• The Identity is preserved: It's still the same person.
• The Label matches: The face now actually looks like the label says it does.
• The Data is saved: The robot hasn't lost any information; it just learned from a "corrected" version of the data.

Why is this a Big Deal?

The authors tested this on two tasks: guessing a person's age and guessing their emotion.

• The "Delete" method made the robot smarter, but it lost some data.
• The "DiffInf" method made the robot even smarter than the delete method.

It turns out that those "confusing" photos were actually valuable! They just needed a little help to make sense. By fixing the photos instead of throwing them away, the robot learns a more complete picture of the world.

The Analogy Summary

• Noisy Data: A student giving the wrong answer on a test.
• Old Method (Filtering): Expelling the student. You get a quieter class, but you lose their unique perspective.
• DiffInf (Influence-Guided Diffusion): The teacher sits down with the student, explains the mistake, and helps them rewrite the answer correctly. The student stays in the class, but now they contribute the right information.

The Bottom Line:
DiffInf teaches us that when data is messy, we shouldn't just throw it away. We should use AI to "clean up" the mess while keeping the valuable parts intact. This leads to AI systems that are not only more accurate but also fairer and more robust because they haven't forgotten the rare and unusual cases.

Technical Summary

1. Problem Statement

Facial attribute learning (e.g., age and expression classification) relies on large-scale datasets where attributes are often continuous and ambiguous but discretized into categorical labels. This process introduces annotation inconsistencies due to:

• Subjectivity: Human annotators may disagree on continuous traits like age.
• Visual Confounders: Factors like pose, illumination, ethnicity, and cosmetics create mismatches between the visual content of an image and its assigned label.

These inconsistencies act as noisy supervision, causing models to learn contradictory signals. This degrades representation learning, generalization, calibration, and fairness. Traditional approaches to handle noisy labels typically involve removing or reweighting problematic samples. However, the authors argue that discarding high-influence samples is suboptimal because these samples often contain rare but valid visual covariates (e.g., specific lighting or demographics) that are crucial for representing the full data manifold. Removing them reduces dataset diversity and coverage.

2. Methodology: DiffInf

The proposed framework, DiffInf, shifts the paradigm from exclusion to semantic conservation. Instead of discarding influential samples, it uses generative models to repair the visual content so it aligns with the assigned label while preserving the subject's identity.

The pipeline consists of four main stages:

A. Baseline Training & Influence Estimation

1. Baseline Classifier: A standard classifier is trained on the noisy dataset.
2. Self-Influence Scoring: The framework computes a self-influence score for each training sample using a first-order approximation (inspired by TracIn).
   • Concept: Self-influence measures how much upweighting a specific sample perturbs the model's parameters.
   • Identification: Samples with anomalously high self-influence are identified as "semantically disruptive." These are instances where the image-label mismatch causes the model to struggle significantly during optimization.
   • Approximation: To make this scalable, the method uses accumulated gradient energy across training checkpoints rather than exact Hessian inversion.

B. Learning an Influence Predictor

Directly computing influence during the generative correction phase is computationally prohibitive. Therefore, DiffInf trains a lightweight, differentiable predictor ($h_\omega$).

• This network takes image features and the assigned label as input and predicts the probability that a sample belongs to the high-influence set.
• It acts as a surrogate regularizer, allowing the system to identify which samples need correction during the optimization process without re-running expensive influence calculations.

C. Influence-Guided Generative Correction

For the identified high-influence samples, DiffInf employs a Latent Diffusion Autoencoder to perform targeted image editing. The goal is to generate a new image $\hat{x}_i$ that:

1. Preserves the original subject's identity.
2. Visually aligns with the assigned label (e.g., making an "Old" face look older if the original was ambiguous).
3. Reduces its destabilizing influence on the classifier.

This is achieved by optimizing the latent variables of the diffusion model using a composite loss function:

• Identity Preservation ($\mathcal{L}_{id}$): Minimizes cosine distance in a face recognition embedding space to ensure the person's identity remains unchanged.
• Regularization ($\mathcal{L}_{reg}$): Includes structural consistency (face parsing) and perceptual similarity (LPIPS) to ensure the edit looks realistic and maintains facial geometry.
• Influence Suppression ($\mathcal{L}_{si}$): Minimizes the output of the influence predictor, effectively pushing the corrected sample toward a region of feature space where it is less disruptive to training.

D. Refined Training Distribution

The original high-influence samples are replaced with the generated, corrected images. The dataset size remains unchanged, but the supervision noise is reduced, and the data distribution is preserved. A new classifier is then trained on this "influence-refined" dataset.

3. Key Contributions

1. DiffInf Framework: A novel self-influence-guided diffusion framework that aligns training images with assigned labels under noisy supervision without discarding data.
2. Targeted Generative Replacement: Instead of filtering, the method uses latent diffusion autoencoding to repair high-influence samples, preserving data coverage and rare visual modes.
3. Differentiable Influence Predictor: The introduction of a lightweight predictor enables scalable, end-to-end influence-guided correction during latent optimization.
4. Empirical Validation: Demonstrates that repairing influential mismatches is superior to simply removing them, particularly for multi-class age and expression classification.

4. Experimental Results

The method was evaluated on the FFHQ dataset for two tasks: 3-class Age Classification and 4-class Expression Recognition, under synthetic symmetric label noise (30% for age, 20% for expression).

• Performance Gains: DiffInf significantly outperformed standard noisy-label training, robust optimization baselines (e.g., ELR+, Small_loss), and influence-based filtering (removal).
  • Age Classification: Improved accuracy from 70.44% (noisy baseline) to 83.37% (DiffInf).
  • Expression Classification: Improved accuracy from 78.95% to 94.24%.
• Comparison with Removal: While removing high-influence samples improved performance, DiffInf consistently achieved higher accuracy, AUROC, and Cohen's $\kappa$. This confirms that the visual information in these "problematic" samples is valuable and should be repaired rather than discarded.
• Perceptual Fidelity: The generated corrections showed low LPIPS distances (0.196–0.244), indicating that edits were localized to attribute-relevant regions (e.g., skin texture for age, mouth curvature for expression) without altering identity or introducing artifacts.

5. Significance and Conclusion

The paper establishes a new paradigm for data-centric robustness.

• Conceptual Shift: It reinterprets high-influence samples not as harmful noise to be eliminated, but as semantically inconsistent supervision containing valuable visual data.
• Mechanism: By coupling causal data attribution (influence functions) with generative modeling (diffusion), DiffInf stabilizes learning dynamics by directly correcting the data level rather than just regularizing the loss.
• Impact: This approach is particularly effective for tasks with ambiguous boundaries (like age) and preserves the diversity of the training distribution, leading to better generalization and fairness compared to exclusion-based methods.

The authors note future work should focus on hyperparameter tuning, more accurate influence estimators, and analyzing demographic biases introduced by the generative correction process.