Random Erasing vs. Model Inversion: A Promising Defense or a False Hope?

Technical Summary: Model Inversion Defense via Random Erasing (MIDRE)

Problem Statement
Model Inversion (MI) attacks represent a critical privacy threat in machine learning, where adversaries exploit access to trained models to reconstruct private training data (e.g., facial images). While existing defenses primarily adopt model-centric strategies—such as modifying loss functions (e.g., BiDO, NLS), altering model architectures (e.g., TL-DMI, MI-RAD), or applying differential privacy—these approaches often suffer from a significant trade-off: they degrade the model's natural utility (accuracy) to achieve privacy protection. Furthermore, the impact of data augmentation techniques on MI robustness remains largely unexplored.

Methodology: MIDRE
The authors propose Model Inversion Defense via Random Erasing (MIDRE), a data-centric defense mechanism that leverages Random Erasing (RE), a technique traditionally used to improve model generalization against occlusion.

• Mechanism: During the training phase, the method applies a square region erasure strategy to private training samples. A random starting point $(x_e, y_e)$ is selected, and a square region of size $\sqrt{s_{RE}} \times \sqrt{s_{RE}}$ is masked (filled with ImageNet mean pixel values). The erasure ratio $a_e$ (the portion of the image erased) is controlled by a single hyperparameter.
• Training Objective: Unlike other defenses, MIDRE does not alter the training loss function or the model architecture. The training objective remains standard (e.g., cross-entropy), but the input data presented to the model is partially masked in every iteration.
• Key Properties: The method relies on two specific properties of RE:
  1. Partial Erasure: Prevents the model from observing the entire object during training, thereby reducing the amount of private information encoded in the model's weights.
  2. Random Location: Ensures the erased region varies across iterations, maintaining training data diversity and preventing the model from learning "shortcut" features based on fixed occlusions.

Feature Space Analysis
The paper provides a novel feature space analysis to explain the defense's effectiveness:

• Discrepancy Creation: Models trained with MIDRE learn feature representations biased toward the RE-private images (the masked training data) rather than the original private images.
• Attack Degradation: MI attacks aim to reconstruct images that maximize likelihood under the target model. Consequently, reconstructed images ($f_{recon}$) align closely with the features of the training data ($f_{RE}$).
• The Mismatch: Because the training data ($f_{RE}$) is a masked version of the private data ($f_{priv}$), there is a significant discrepancy between the features of the reconstructed images and the actual private images. This mismatch prevents the attack from accurately replicating the private data features.
• Utility Preservation: Despite the discrepancy between $f_{RE}$ and $f_{priv}$, the private images remain distinct from other classes and well-separated in the feature space, allowing the model to maintain high natural classification accuracy.

Key Contributions

1. Novel Defense Mechanism: MIDRE is the first work to utilize Random Erasing as a privacy protection mechanism against MI attacks, demonstrating that a simple data augmentation technique can effectively reduce attack accuracy.
2. Feature Space Insight: The authors provide a deep analysis showing that the defense works by creating a feature space discrepancy between reconstructed and private images, while maintaining class separability for utility.
3. Property Analysis: The study isolates and validates the importance of "Partial Erasure" (reducing private information exposure) and "Random Location" (preserving utility) as the drivers of the privacy-utility trade-off.
4. Comprehensive Evaluation: The method is evaluated across 37 experimental setups, including 9 state-of-the-art MI attacks (white-box, black-box, and label-only), 11 network architectures (including Vision Transformers), and 6 datasets with varying resolutions (64×64 to 224×224).

Experimental Results

• Privacy-Utility Trade-off: Extensive experiments demonstrate that MIDRE achieves State-of-the-Art (SOTA) performance in the privacy-utility trade-off.
• Attack Reduction: The method significantly degrades MI attack accuracy. For example, in high-resolution settings (224×224), MIDRE reduced attack accuracy by up to 69.39% (Setup 4) while simultaneously increasing natural model accuracy by 0.37%.
• Comparison with SOTA: Across various architectures (ResNet, DenseNet, MaxViT, Inception) and attacks (PPA, MIRROR, IF-GMI, LOMMA), MIDRE consistently outperformed existing defenses like BiDO, NLS, TL-DMI, and MI-RAD.
• Ablation Studies:
  • Partial vs. Entire Erasure: Reducing the number of training epochs (simulating "Entire Erasure") was less effective than partial erasure of images.
  • Random vs. Fixed Location: Random location erasure yielded higher model accuracy than fixed location erasure while maintaining similar attack resistance.
  • Combinability: MIDRE is complementary to existing model-centric defenses. Combining MIDRE with BiDO or NLS resulted in further reductions in attack accuracy (e.g., down to 3.75% in one setup) while preserving or improving utility.

Significance and Claims
The paper claims that MIDRE offers a simple yet effective defense that can be easily integrated with existing privacy-preserving techniques. Its primary significance lies in achieving significant degradation in MI attack accuracy without decreasing model utility in several configurations, and in some cases, even improving utility. The authors position this as a paradigm shift from complex model-centric modifications to a straightforward data-centric approach that effectively disrupts the information flow required for successful model inversion.