🤖 machine learning

In-Context Probing for Membership Inference in Fine-Tuned Language Models

This paper introduces ICP-MIA, a novel black-box membership inference attack framework that leverages the "optimization gap" via training-free in-context probing to significantly outperform existing methods in detecting whether sensitive data was used to fine-tune large language models.

Original authors: Zhexi Lu, Hongliang Chi, Nathalie Baracaldo, Swanand Ravindra Kadhe, Yuseok Jeon, Lei Yu

Published 2026-07-21

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhexi Lu, Hongliang Chi, Nathalie Baracaldo, Swanand Ravindra Kadhe, Yuseok Jeon, Lei Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot that has read almost everything on the internet. It's like a genius who knows a little bit about everything. But sometimes, you want to teach this robot a very specific, secret skill—like how to diagnose rare diseases using only private hospital records, or how to write legal contracts for a specific company. You give the robot a small, special set of practice problems to learn from. This process is called "fine-tuning."

Now, here's the scary part: after the robot learns these secrets, it might accidentally remember them too well. It's like a student who memorizes the answers to a practice test so perfectly that if you ask them the exact same question later, they sound suspiciously confident. This is a privacy risk. If someone can tell, just by asking the robot a question, whether that question was part of the secret practice set, they might be able to steal private information about the people whose data was used. This is called a "Membership Inference Attack." It's like trying to guess if a specific person was in a room just by listening to how the room sounds when they speak.

For a long time, experts tried to catch these leaks by looking at how confident the robot sounded. If the robot was super confident, they thought, "Aha! It must have seen this before!" But this is tricky. Sometimes the robot is just naturally good at answering easy questions, even if it's never seen them before. It's like a math whiz who gets an easy problem right instantly; that doesn't mean they memorized it, they just understood it. So, the old methods often got confused, mixing up "easy questions" with "secret questions."

The New Detective Tool: The "Optimization Gap"

In this paper, the researchers from Rensselaer Polytechnic Institute, IBM Research, and Korea University propose a clever new way to catch these privacy leaks. They call their method ICP-MIA. Instead of just listening to how confident the robot sounds, they look at something deeper: how much room the robot has left to learn.

Think of it like a sponge.

The Secret Sponges (Members): Imagine you soaked a sponge in water (the secret training data) until it was completely full. If you try to squeeze it or add more water, it can't hold much more. It's already "optimized."
The Dry Sponges (Non-Members): Now imagine a dry sponge that you've never touched. If you try to squeeze it or add water, it soaks up a lot! It has a huge "gap" between where it is and where it could be.

The researchers realized that for the secret data the robot learned, the "learning potential" is almost zero. For new, unseen data, there is still a lot of potential for the robot to improve its answer. They call this difference the Optimization Gap.

How They Test It Without Breaking the Robot

Here's the tricky part: The attackers (the privacy auditors) don't have access to the robot's brain. They can't see the sponge or squeeze it directly. They can only ask questions and get answers. So, how do they measure this "learning potential"?

They use a trick called In-Context Probing.

Imagine you are trying to guess if a student memorized a story. Instead of just asking, "What happened in the story?" you give them a hint. You say, "Remember that time we talked about dragons? Now, tell me the story again."

If the student memorized the story, the hint about dragons won't change their answer much. They already know it perfectly. Their confidence stays the same.
If the student didn't know the story, the hint might help them figure it out, or it might confuse them. Their answer changes significantly because they are trying to "learn" on the spot.

The researchers do this with the AI. They give the AI a sample question, but they also add a little "hint" or "context" before it (like a few similar examples). Then they see how much the AI's answer improves.

Big improvement? The AI didn't know it well. It was a "non-member" (not in the secret training set).
Tiny improvement? The AI already knew it perfectly. It was a "member" (part of the secret training set).

What They Found

The team tested this idea on several different AI models and datasets, including medical questions and news summaries. They found that their new method, ICP-MIA, was much better at spotting the secret data than the old methods.

It works better: On a medical dataset called HealthcareMagic, their method got a score of 0.942 (where 1.0 is perfect), beating the previous best methods which were around 0.847 and 0.837.
It's precise: In situations where you can't afford to make mistakes (like accusing an innocent person of being in the training set), their method was more than 2.6 times better at finding the real secrets without making false alarms compared to other methods.
It works without extra data: Some old methods needed a second, similar dataset to compare against. This new method can work just by using the question itself or by making up slight variations of the question, making it much easier to use in the real world.

They also discovered that the type of robot matters. Robots that were specifically taught to follow instructions (called "instruction-tuned" models) were actually easier to trick with this method. It seems that when a robot is good at following hints, it's also better at showing off whether it has memorized something or not.

Why This Matters

This isn't just a game of guessing. The researchers showed that even when the AI is protected by special privacy techniques (like adding noise to the learning process), their method can still find the secrets, though it gets harder. This suggests that as we use more private data to train these powerful robots, we need better ways to check if our secrets are safe.

The paper doesn't claim to have solved the privacy problem forever. Instead, it offers a new, sharper tool for privacy auditors. It's like giving security guards a better metal detector that can find hidden secrets even when they are buried under a pile of easy questions. By understanding the "Optimization Gap," we can finally see the difference between a robot that truly learned a secret and one that just got lucky.

Technical Summary: In-Context Probing for Membership Inference in Fine-Tuned Language Models

Problem Statement

Fine-tuned Large Language Models (LLMs) deployed in privacy-sensitive domains (e.g., healthcare, law, finance) are vulnerable to Membership Inference Attacks (MIAs). These attacks aim to determine whether a specific data sample was part of a model's training dataset, potentially revealing sensitive information.

Existing black-box MIA techniques for LLMs generally fall into two categories:

Reference-based attacks: These rely on training shadow models or reference models using auxiliary data from a distribution similar to the target's training set. This assumption often fails in real-world scenarios where training data is proprietary or private.
Reference-free attacks: Methods like Min-K% or ReCaLL rely on raw confidence scores, loss values, or token likelihoods. However, these signals are often entangled with a sample's intrinsic properties (e.g., difficulty or rarity), leading to poor generalization, low signal-to-noise ratios, and a lack of principled grounding regarding training dynamics.

The core challenge is identifying a robust signal for membership inference that is independent of sample difficulty and does not require access to auxiliary reference data or internal model parameters.

Methodology: ICP-MIA

The authors propose ICP-MIA, a novel framework grounded in the theory of training dynamics, specifically the phenomenon of diminishing returns during optimization.

1. The Optimization Gap

The authors posit that during fine-tuning, models adapt quickly to seen data (member samples) early in the process, followed by diminishing returns as training progresses. Consequently, at convergence:

Member samples have minimal remaining loss-reduction potential (they are already optimized).
Non-member samples retain significant potential for further optimization.

This disparity is defined as the Optimization Gap. The authors hypothesize that this gap serves as a fundamental, principled signal for membership inference.

2. In-Context Probing (ICP)

Since attackers operate in a black-box setting and cannot perform actual gradient updates to measure the Optimization Gap, the authors introduce In-Context Probing (ICP). This is a training-free method that simulates a fine-tuning step at inference time by strategically constructing input contexts.

The core mechanism relies on the interpretation of In-Context Learning (ICL) as an implicit optimization process. By prepending a probe context $C$ to an input $x$ , the model's log-likelihood (LL) for the target output $y$ changes. The ICP Score is defined as:
$\text{ICP}_{\text{score}}(s, C) = \text{LL}(y | x; M) - \text{LL}(y | C \oplus x; M)$

Non-members: Exhibit a large LL improvement (large negative ICP score) when probed, indicating significant remaining optimization potential.
Members: Exhibit minimal LL improvement (score near zero), as they have already been learned.

3. Probing Strategies

To estimate this gap, the paper proposes two complementary strategies for constructing probe contexts:

Reference-Data-Based (ICP-MIA-Ref): Uses an external auxiliary dataset to retrieve semantically similar samples as probe contexts. This does not require training a reference model but relies on semantic alignment.
Self-Perturbation (ICP-MIA-SP): Generates probes directly from the target sample without external data. This includes:
- Generation-based perturbation: Using an auxiliary LLM to generate variant responses for the same input.
- Masking-based perturbation: Randomly or strategically masking tokens in the target response to test the model's reliance on internalized knowledge versus context.

Key Contributions

Formalization of the Optimization Gap: The paper is the first to propose and formalize the "Optimization Gap"—the disparity in remaining loss-reduction potential between member and non-member samples—as a fundamental signal for membership inference.
ICP-MIA Framework: Introduction of a practical, training-free black-box method (ICP) that simulates fine-tuning behavior to estimate the Optimization Gap.
Dual Probing Strategies: Development of both reference-data-based and reference-free self-perturbation strategies, enhancing robustness across diverse data distributions.
Empirical Validation: Demonstration that ICP-induced loss changes correlate significantly with true single-step gradient-based loss reductions, validating ICP as a reliable proxy for the Optimization Gap.

Experimental Results

The framework was evaluated on three datasets (HealthcareMagic, CNN-DM, MedInstruct) and multiple LLMs (LLaMA-3.2-3B, LLaMA-3.2-3B-Instruct, Pythia-2.8B).

Performance: ICP-MIA significantly outperforms prior black-box MIAs. On the HealthcareMagic dataset with LLaMA-3.2-3B-Instruct, ICP-MIA-SP achieved an AUC of 0.942, surpassing ReCaLL (0.847) and Min-K% (0.837).
High-Precision Scenarios: The method excels in high-precision settings (low false positive rates). On CNN-DM, ICP-MIA-SP achieved a TPR@1%FPR of 0.518, more than 2.6 times higher than reference-free baselines like ReCaLL (0.195).
Robustness:
- Instruction Tuning: Attacks are more effective on instruction-tuned models, which exhibit stronger ICL capabilities, leading to a more pronounced Optimization Gap.
- PEFT: The attack remains effective across Parameter-Efficient Fine-Tuning (PEFT) configurations (LoRA, QLoRA), though higher adapter capacity increases vulnerability.
- Differential Privacy: While Differential Privacy (DP) suppresses attack success, ICP-MIA still outperforms most baselines under strong DP constraints.
Masking Analysis: Random masking was found to be more effective than deterministic LL-based masking, suggesting that diversity in probe generation is crucial for finding optimal perturbations.

Significance and Claims

The paper claims that ICP-MIA establishes a theoretically grounded and practical framework for auditing privacy risks in deployed LLMs. By shifting the focus from raw confidence scores to the residual learning potential (Optimization Gap) of a sample, the method overcomes the limitations of sample-intrinsic properties that plague previous approaches.

The authors emphasize that this approach provides a principled signal for membership inference that is:

Black-box compatible: Requires only inference queries.
Reference-free capable: Does not strictly require auxiliary data aligned with the target distribution.
Interpretable: The signal is directly linked to the fundamental dynamics of neural network training.

The work suggests that the "diminishing returns" observed in training dynamics create a sharp, exploitable separation between seen and unseen data, offering a new perspective for both privacy auditing and the development of more robust privacy-preserving mechanisms.