Technical Summary: Negative Unlabeled Dataset Usage Inference (NU-DUI)

Problem Statement

Dataset Usage Inference (DUI) aims to estimate the fraction of a specific dataset that was used to train a machine learning model. While existing methods like Dataset Inference (DI) can determine whether a dataset was used (binary classification), they fail to quantify how much of it was used. This quantification is critical for legal frameworks (e.g., fair-use analysis under the U.S. Copyright Act) and real-world data ownership disputes.

Current state-of-the-art approaches, such as DUCI (Tong et al., [46]), face two significant practical barriers:

1. Shadow Model Requirement: They require training expensive "shadow models" that mimic the target model's architecture and behavior. For large-scale generative models (e.g., 1.5B parameters), training a single shadow model can take over 1,500 A100 hours, making this computationally prohibitive.
2. Held-Out Data Assumption: They assume access to a trusted, in-distribution "held-out" set confirmed to be absent from the target model's training. In real auditing scenarios, such trusted non-member data is rarely available.

Furthermore, standard Membership Inference Attacks (MIAs) often degrade to near-random performance on large modern models due to weak per-sample signals and sensitivity to distribution shifts between member and non-member data.

Methodology: NU-DUI

The authors propose Negative Unlabeled Dataset Usage Inference (NU-DUI), a framework that eliminates the need for shadow models and trusted held-out data. The method reframes DUI as a Mixture Proportion Estimation (MPE) problem.

The pipeline consists of three main stages:

1. Synthetic Non-Member and Autoencoded Suspect Set Construction

To address the lack of a trusted held-out set, NU-DUI constructs reference data directly from the suspect set ($X_{real}$):

• Synthetic Non-Members ($X_{img2img}^{nonmember}$): The suspect images are paraphrased using an image-to-image generator ($G$) from a different generative family than the audited target model (e.g., using Stable Diffusion to paraphrase images for an autoregressive target, or vice versa).
  • Rationale: Using a different family prevents the synthetic samples from lying near a "fixed point" of the target model's loss function. If the same family were used, paraphrased images might exhibit artificially low loss (looking like members) due to the target model's specific reverse process dynamics, confounding the inference.
• Autoencoded Suspect Set ($X_{AE}$): The original suspect set is encoded and decoded using the same autoencoder used for paraphrasing.
  • Rationale: Generative models introduce specific high-frequency artifacts. By autoencoding the suspect set, the method injects these same artifacts into the "unlabeled" side of the comparison. This ensures that the distribution shift between the unlabeled mixture ($U$) and the synthetic negatives ($N$) is dominated by membership status rather than generator-specific artifacts.

2. MIA Feature Extraction

The framework queries the target model on both the autoencoded suspect set ($U$) and the synthetic non-member set ($N$). It extracts a vector of membership-inference features ($\Phi_{MIA}$) tailored to the target model family:

• For Image Autoregressive (IAR) models, it uses feature suites based on token-level losses and confidence scores.
• For Diffusion models, it uses noise-prediction errors and conditional likelihood discrepancies.
• Multiple signals are combined into a single feature vector to improve separation between member and non-member distributions, as single signals are often weak or noisy.

3. Mixture Proportion Estimation (MPE)

The problem is cast as estimating the proportion $\pi$ of members in the unlabeled set $U$, given that $U$ is a mixture of members (from the suspect set) and non-members (synthetic).

• The synthetic set $N$ serves as the labeled non-member reference.
• The autoencoded set $U$ serves as the unlabeled mixture.
• Standard MPE estimators (e.g., TIcE, AlphaMax, PUL variants) are applied to the feature vectors to recover the estimated member ratio $\hat{p}$.

Key Contributions

1. Reformulation as MPE: The authors recast Dataset Usage Inference as a Mixture Proportion Estimation problem, removing the dependency on expensive shadow models.
2. Synthetic Reference Generation: They propose a method to generate realistic synthetic non-members via cross-family image-to-image paraphrasing and to align the suspect set via autoencoding. This eliminates the reliance on trusted held-out non-member sets, which are typically unavailable in practice.
3. Empirical Validation: The method is demonstrated to produce accurate member-ratio estimates across five large-scale image generative models (both autoregressive and diffusion) with orders-of-magnitude lower computational cost than shadow-model-based approaches.

Experimental Results

The authors evaluated NU-DUI on large-scale models including VAR-24/30, RAR-XL/XXL, and DiT-RF-XL/2-8E2A, trained on ImageNet-1k.

• Accuracy: Across various ground-truth member ratios ($p$), NU-DUI (using the "Synth + AE" pipeline) closely tracks the identity line.
  • On RAR-XXL with a suspect set of 1,000 images, the Mean Absolute Error (MAE) was 0.058 (using PUL-based MPE), significantly closing the gap to the "Oracle" setting (which assumes access to real held-out non-members).
  • Without autoencoding ("Synthetic" only), the method systematically overestimates the member ratio at low $p$ due to artifact mismatches. Autoencoding corrects this bias.
• Cross-Family Necessity: Ablation studies showed that using a paraphraser from the same generative family as the target model significantly degraded estimation quality, confirming that cross-family generation is essential to avoid loss-function fixed points.
• Computational Efficiency:
  • DUCI (1 shadow model): ~1,500 A100 hours for a 1.5B parameter target.
  • NU-DUI: ~42.5 A100 minutes for the same target.
  • This represents a >2,000x speedup, making the method feasible for modern large models where shadow training is impossible.
• Scalability: Estimation accuracy improves rapidly as the suspect set size increases from 100 to 1,000 images, after which gains saturate. The method remains practical even with small suspect sets (e.g., 100 images).

Significance and Limitations

Significance:
The paper presents NU-DUI as a practical tool for data owners and auditors to obtain quantitative evidence of training data usage in modern generative models. By removing the need for shadow models and trusted held-out data, it makes dataset auditing feasible for large-scale, proprietary models where previous methods were computationally or data-prohibitive.

Limitations:
The authors note two primary constraints:

1. MIA Signal Strength: The accuracy of NU-DUI is bounded by the strength of available membership-inference signals. If a target model leaks little membership information, the estimates will be less stable.
2. Cross-Family Generator Availability: The method requires a sufficiently strong generative model from a different family to produce synthetic non-members. In domains where no such alternative generator exists, the method cannot be directly applied.

The paper concludes that NU-DUI unlocks dataset usage inference for modern generative models by leveraging synthetic data generation and mixture proportion estimation, offering a scalable alternative to shadow-model-based auditing.