💬 NLP

Boundary-targeted Membership Inference Attacks on Safety Classifiers

This paper introduces a boundary-targeted membership inference attack strategy that exploits low-confidence predictions to significantly improve the recovery of sensitive training data from safety classifiers, demonstrating that such models are vulnerable to privacy breaches despite content-based filtering, though existing noise strategies can effectively mitigate this risk.

Original authors: Anthony Hughes, Alexander Goldberg, Prince Jha, Adam Perer, Nikolaos Aletras, Niloofar Mireshghallah

Published 2026-05-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Anthony Hughes, Alexander Goldberg, Prince Jha, Adam Perer, Nikolaos Aletras, Niloofar Mireshghallah

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-meaning security guard at the entrance of a massive library. This guard's job is to spot people who might be in trouble or trying to cause harm, so they can get help or be stopped before they enter. To learn how to do this, the guard has studied thousands of real, private stories from people who were once in crisis, feeling depressed, or thinking about self-harm.

The paper by Anthony Hughes and his team asks a scary question: Can a stranger figure out which specific private stories the guard studied?

Here is the breakdown of their discovery, using simple analogies:

1. The "Confidence" Trap

Usually, when we think about a security guard (or an AI) making a mistake, we think about them being confused or unsure. But in this research, the authors found something counter-intuitive.

The Old Way: Attackers used to try to guess if a story was in the training data by looking at the stories the guard was 100% sure about. They thought, "If the guard knows this perfectly, it must be because they memorized it."
The New Discovery: The authors found that the guard is actually most "leaky" when they are least confident.
- The Analogy: Imagine the guard is looking at a blurry photo. If the photo is clearly a cat, the guard says, "That's a cat!" (High confidence). If the photo is a very strange, blurry mix of a cat and a dog, the guard hesitates.
- The researchers found that when the guard hesitates on a blurry, difficult story, it's often because they are recalling a specific, similar story from their memory rather than using general logic. This hesitation is a "tell." It's like the guard sweating when they see a face they recognize from a specific, private file, even if they aren't sure what to do with it.

2. The "Boundary" Strategy

The team invented a new way to attack the system called "Boundary-Targeted Membership Inference."

The Analogy: Think of the guard's decision-making as a tightrope. On one side is "Safe," and on the other is "Unsafe." Most people walk easily on the solid ground far from the rope. But the "boundary" is the wobbly part right in the middle.
The researchers realized that the people (or stories) standing right on that wobbly tightrope are the ones the guard has memorized the most. By focusing only on these wobbly, uncertain cases, the attacker can tell with much higher accuracy if a specific story was in the guard's private study files.
The Result: Using this "wobbly tightrope" strategy, the attackers could identify 19% of the specific distress conversations the guard had studied, whereas standard methods only caught about 5%. That's nearly 3.5 times more private information exposed.

3. Why Content Filters Don't Work

The team wondered: "Can we just remove the weird, blurry stories from the training data to fix this?"

The Analogy: They tried to find the "blurry" stories by looking at their text, like trying to find a specific grain of sand by looking at its color.
The Result: It didn't work. The "blurry" stories didn't look any different from the "safe" stories on the outside. They were just as normal-looking. You couldn't filter them out by reading them because the "danger" wasn't in the words; it was in how the AI's brain had memorized them.

4. The Solution: Adding "Static" to the Signal

Since they couldn't remove the dangerous memories, they tried to make the guard's answers fuzzier.

The Analogy: Imagine the guard is whispering their answers to you. If they whisper too clearly, you can hear exactly what they are thinking. The researchers suggested adding a little bit of "static" or "white noise" to the guard's voice right before they speak.
The Result: This "noise" (mathematically called Laplace perturbation) made the guard's hesitation less precise. It was enough to hide the specific memory of the private story without making the guard bad at their job. The guard could still say "Unsafe" or "Safe," but they couldn't accidentally reveal which specific private story they were remembering.

Summary

The paper shows that safety AIs, which are trained on sensitive human struggles, are vulnerable. If an attacker knows how to look at the moments where the AI is unsure, they can figure out exactly which private human stories the AI learned from.

The good news is that the authors found a fix: by adding a tiny bit of "noise" to the AI's final answer, we can protect those private stories without stopping the AI from doing its job of keeping people safe.

Technical Summary: Boundary-targeted Membership Inference Attacks on Safety Classifiers

Problem Statement

Safety classifiers are critical components in generative AI systems, deployed to filter harmful content and identify at-risk users (e.g., those exhibiting signs of self-harm or mental health crises). These models are trained on sensitive datasets containing real user interactions, including discussions of personal vulnerabilities. While the necessity of these classifiers is well-established, their privacy implications remain poorly understood. Specifically, there is a lack of systematic investigation into whether these classifiers leak information about the specific training examples they were exposed to.

This paper addresses the vulnerability of safety classifiers to Membership Inference Attacks (MIAs). In an MIA, an adversary attempts to determine whether a specific data point (e.g., a user conversation) was part of a model's training set. Conventional wisdom in MIA literature suggests that attacks are most effective against high-confidence examples, where the model has clearly memorized the data. However, this paper challenges that assumption for safety classifiers, hypothesizing that the most informative targets for inference lie at the decision boundary—examples where the classifier is least confident. These ambiguous cases are hypothesized to rely on memorization rather than generalization, making them uniquely vulnerable to privacy leakage.

Methodology

Threat Model

The study assumes a standard black-box threat model where an adversary has score-based access to a deployed safety classifier $f_\theta$ . The adversary can submit candidate conversations and observe the predicted class-probability vectors. To improve attack efficacy, the adversary is also equipped with a reference classifier $f_{ref}$ , trained on a disjoint dataset from a similar distribution. This assumption is grounded in reality, as model providers often release public safety classifier weights (e.g., Llama Guard, ShieldGemma) that can serve as reference models without access to the target's private training data.

Boundary-Targeted Selection Strategy

The core methodological contribution is boundary-targeted selection. Unlike standard MIAs that evaluate uniformly sampled examples, this strategy filters the candidate pool to isolate examples residing near the classifier's decision boundary.

Selection Classifier: A separate classifier $f_S$ is trained on a calibration set ( $D_{cal}$ ) disjoint from both the target's training data and the reference data. This ensures $f_S$ is independent of the target and reference models.
Filtering: The adversary scores all candidate examples based on the probability $f_S$ assigns to the ground-truth label. The $n$ candidates with the lowest probabilities (i.e., those where $f_S$ is least confident) are retained to form the "boundary set" $B$ .
Rationale: This approach isolates examples where the target classifier's confidence cannot be attributed to intrinsic separability (since the reference model is also uncertain). Any residual confidence in the target model is thus more plausibly attributed to memorization of the specific training instance.

Scoring Functions

The study evaluates two standard scoring functions on the boundary-selected pool:

Loss-based: Thresholds the target classifier's confidence on the true label ( $s = \log f_\theta(x)_y$ ).
LiRA (Likelihood Ratio Attack): Calibrates the target's confidence against the reference model ( $s = \log f_\theta(x)_y - \log f_{ref}(x)_y$ ).

Experimental Setup

The authors evaluated this strategy across:

Four Language Models: Llama-3.2-1B, Llama-3.2-3B, Llama-3.1-8B, and Gemma-3-12B.
Five Datasets: Spanning single-turn toxicity (BeaverTails), multi-turn jailbreaks (X-Guard-Train), multi-conversation emotional support (ESConv, Psychotherapy Eval), and a pooled configuration.
Training Regimes: Both full fine-tuning and parameter-efficient fine-tuning (LoRA).

Key Results

Amplification of Attack Effectiveness

The boundary-targeted strategy significantly outperforms standard global selection methods.

Performance Gain: On a classifier fine-tuned for detecting users requiring emotional support, the boundary-targeted attack recovered 19% of the flagged conversations at a 5% false-positive rate. This is 3.5 times higher than attacks using state-of-the-art MIA methods alone.
MI-AUC Improvements: Across models and datasets, boundary targeting yielded MI-AUC gains of +0.12 to +0.16 for safe classes and +0.04 to +0.18 for unsafe classes under LoRA. For multi-conversation data involving highly sensitive interactions, MI-AUC exceeded 0.75 for unsafe conversations.
Robustness to Training Capacity: While LoRA was found to reduce vulnerability for global attacks (particularly on unsafe content), the boundary-targeted attack closed this gap, extracting significant signal even in parameter-efficient settings. This suggests LoRA protects the "easy" population but leaves the boundary signal intact.

Characteristics of Vulnerable Examples

Confidence Correlation: There is a consistent monotonic relationship between classifier confidence and attack success. Examples in the lowest confidence decile (where the model struggles most) yield the highest MI-AUC, while high-confidence examples perform near random chance.
Harm Categories: Vulnerability varies by category. "Controversial Topics" and "Hate Speech" showed the strongest membership signals (MI-AUC 0.66–0.74), whereas categories with clearer definitions like "Financial Crime" were closer to random chance. This supports the hypothesis that ambiguous, subjective content drives memorization.
Generalizability: The effect was replicated in a depression screening task (using Reddit data), where boundary-targeted LiRA achieved an MI-AUC of 0.867, a 0.21–0.26 absolute improvement over the baseline.

Defense Evaluation

The paper investigates two mitigation strategies:

Content-Based Filtering: The authors tested whether boundary examples are semantically distinctive (geometric outliers) in embedding space. Using outlier analysis on Sentence-BERT and hidden states, they found no statistically significant evidence that boundary examples are distinctive. Consequently, filtering based on geometric properties is deemed ineffective.
Inference-Time Noise Injection: The authors propose adding Laplace noise to the classifier's logits before the final softmax.
- Results: This post-hoc defense effectively mitigates the attack. At moderate noise scales ( $\sigma \le 0.1$ ), the attack's MI-AUC decreased by 2–5 percentage points while classification accuracy degraded by less than 1 point.
- Trade-off: Higher noise scales ( $\sigma = 0.5$ ) further reduced attack success (e.g., dropping MI-AUC from 0.87 to 0.70 for Llama-3.2-1B) with manageable utility loss.

Significance and Claims

The paper claims to present the first systematic study of membership inference attacks specifically targeting safety classifiers. Its primary significance lies in:

Reframing the Attack Surface: It challenges the conventional MIA assumption that high-confidence examples are the primary leakage vector, demonstrating that for safety classifiers, the decision boundary is the critical vulnerability.
Privacy Risks in Welfare Monitoring: It highlights that classifiers designed to detect user distress (e.g., self-harm, suicidal ideation) are particularly susceptible to recovering the specific conversations of vulnerable users, posing a severe re-identification risk.
Practical Mitigation: It demonstrates that content-based filtering is insufficient and proposes a practical, post-hoc defense (noise injection) that can be deployed without retraining, offering a path to safer deployment of safety classifiers.

The authors conclude that while safety classifiers are essential, their current deployment without specific privacy safeguards (like noise injection) risks exposing the sensitive interactions of the very users they are intended to protect.