🤖 AI

JUMP: Single-Pass Membership Inference on Fine-Tuned Diffusion Language Models

This paper introduces JUMP, a single-pass membership inference attack for fine-tuned discrete diffusion language models that leverages their any-order and parallel decodability to select low-confidence positions and score them jointly, significantly outperforming the prior SAMA method in accuracy and efficiency.

Original authors: Yeachan Jun, Albert No

Published 2026-07-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yeachan Jun, Albert No

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out if a specific sentence was part of a secret recipe book used to teach a robot how to write. This is the world of Membership Inference, a branch of computer science dedicated to privacy. In this field, researchers act like digital detectives, trying to see if a model "memorized" a specific piece of data during its training. If the model behaves differently when it sees a sentence it has seen before compared to a new one, it might be leaking private information.

To understand the new twist in this story, we need to know about two types of writing robots. The old kind, called Autoregressive models, write like a person typing a letter: they can only look at what they've already written to guess the next word. They are locked in a left-to-right line. The new kind, called Discrete Diffusion Language Models (dLLMs), are more like a puzzle solver. They can look at a whole sentence with some words hidden (masked) and guess all the missing words at the same time, in any order. This gives them a superpower: they can be asked to solve the same sentence in hundreds of different ways, depending on which words you choose to hide. The big question for privacy experts is: does this new superpower make it easier or harder to tell if a sentence was in the robot's secret training book?

Enter JUMP, a new method proposed by researchers Yeachan Jun and Albert No. They discovered that the old way of testing these new robots was like trying to find a needle in a haystack by throwing random handfuls of hay at it. The previous best method, called SAMA, would randomly pick groups of words to hide, ask the robot to guess them, and repeat this process many times to get an average score. It worked, but it was slow and often picked boring, easy words that didn't tell the detective much.

JUMP changes the game by being a smart, single-pass detective. Instead of guessing randomly, JUMP first uses a "reference" robot (a version of the model that hasn't seen the secret book) to find the specific words in a sentence that are the most confusing or difficult for it to guess. These are the "low-confidence" spots. JUMP then hides only those tricky words all at once and asks the target robot to solve them. Because dLLMs can solve all hidden words in parallel, JUMP gets a full report on all the tricky spots in just one single look. It then compares how well the target robot did versus the reference robot. If the target robot is surprisingly good at fixing those specific confusing spots, it's a strong sign that the sentence was in its training data.

The results are impressive. When tested on a large model called LLaDA-8B-Base across six different topics (like Wikipedia, medical papers, and coding sites), JUMP proved to be much sharper than the old random method. It raised the average success rate (measured as ROC-AUC) from 0.82 to 0.90. Even more importantly, it became much better at spotting members when the stakes are high (low false alarm rates), catching more members at a 0.1% false positive rate where the old method barely caught anything. Perhaps the most playful part of the story is the efficiency: while the old method might need dozens of questions to get a good answer, JUMP needs just two (one for the target, one for the reference) plus a tiny bit of setup. The researchers found that even without training a special helper tool, a simple version of JUMP could still outperform the old random method, suggesting that the "confusing word" strategy is a fundamental weakness in these new models.

In short, JUMP shows that for these new, flexible writing robots, the best way to catch a privacy leak isn't to ask a million random questions, but to ask one very smart question about the words that are hardest to guess. It turns the robot's ability to work in any order from a confusing feature into a clear vulnerability, proving that these models leave a much bigger fingerprint on their training data than we might have thought.

Technical Summary: JUMP (Joint Uncertainty-Guided Mask Probing)

1. Problem Statement

The paper addresses Membership Inference Attacks (MIAs) in the context of fine-tuned Discrete Diffusion Language Models (dLLMs). Unlike autoregressive (AR) models, where membership inference relies on fixed left-to-right likelihoods, dLLMs (e.g., LLaDA, Dream) utilize a masked reconstruction objective. This allows an attacker to choose arbitrary sets of tokens to mask and reconstruct them in parallel.

The core challenge identified is that existing dLLM-specific attacks, such as SAMA (Chen et al., 2026), treat the "any-order" decoding capability primarily as a source of randomization. SAMA samples many random mask sets, averages the reconstruction gaps between a target model and a reference model, and requires a number of forward passes proportional to the number of sampled masks ( $T$ ). This approach is computationally expensive and dilutes the membership signal by including uninformative or "easy" tokens in the random subsets.

The paper poses the question: Can an attack leverage the dLLM's any-order interface to select a highly informative mask set, and then utilize parallel decoding to evaluate that set in a single pass?

2. Methodology: JUMP

The authors propose JUMP (Joint Uncertainty-Guided Mask Probing), a single-pass scoring attack designed to exploit two distinctive properties of dLLMs:

Any-order decodability: The ability to choose which tokens to mask.
Parallel decodability: The ability to reconstruct all masked tokens simultaneously in one forward pass.

Core Components

A. Diagnostic Insight (Low-Confidence Localization)
Through a diagnostic experiment, the authors observed that membership signals are strongest at positions where the reference model has low confidence in the true token. Specifically, the reconstruction gap ( $\Delta_i$ ) between the target (fine-tuned) and reference models is significantly larger for low-confidence tokens in member sequences compared to random or high-confidence tokens.

B. The JUMP Algorithm
JUMP operates in two stages:

Selector Phase (Position Selection):
- The attack identifies $K$ positions in the input sequence $x$ where the reference model $M_{ref}$ assigns low confidence to the true token.
- To avoid the cost of querying every position individually (which would require $O(L)$ passes), JUMP uses a Generic Token Selector.
- Default (White-box): Uses PRISM (Kim et al., 2025), a lightweight per-token quality head trained on the reference model's pretraining distribution to predict the "one-hole" low-confidence score without needing member/non-member labels.
- Strict (Black-box): Uses PRISM-Free, which proxies the low-confidence score using the reference model's clean-text log-probability of the true token.
- The selector outputs a set of indices $H_K(x)$ representing the $K$ most uncertain positions.
Scoring Phase (Joint Probing):
- The selected positions are masked jointly into a single mask set $S = H_K(x)$ .
- The attacker submits this single masked sequence to both the target model ( $M_{tgt}$ ) and the reference model ( $M_{ref}$ ).
- Both models reconstruct the masked tokens in parallel.
- The attack computes the reconstruction gap for each selected token: $\Delta_i(S) = \log p_{M_{tgt}}(x_i | x \setminus S) - \log p_{M_{ref}}(x_i | x \setminus S)$ .
- The final MIA statistic is the average of these gaps, with a clipping mechanism to prevent outliers from dominating the score:
  $g_{JUMP}(x) = \frac{1}{K} \sum_{i \in S} \text{clip}(\Delta_i(S), -\tau, \tau)$
  where $\tau = \log 1.5$ .

Efficiency

JUMP is designed for single-pass efficiency. Regardless of the number of selected tokens ( $K$ ) or sequence length ( $L$ ), the scoring phase requires exactly two forward passes (one for the target, one for the reference). The selector phase adds only one additional reference pass (or zero for PRISM-Free), making the total cost $O(1)$ relative to the number of sampled masks used in prior methods.

3. Key Contributions

Novel Attack Paradigm: JUMP is the first MIA for dLLMs that treats mask-set selection as a primary design component rather than a randomization step. It shifts the attack strategy from "many random masks" to "one informative joint mask."
Efficiency: It drastically reduces the computational cost (Number of Forward Evaluations, NFE) compared to SAMA. While SAMA requires $2T$ passes (where $T$ is the number of random subsets), JUMP requires only $2+1$ (or $2$) passes per sample.
Performance: The method significantly improves detection accuracy, particularly in strict low-False Positive Rate (FPR) regimes, by focusing on high-signal tokens.
Selector Flexibility: The paper demonstrates that effective position selection can be achieved with a trained auxiliary head (PRISM) or even a simple, training-free proxy (PRISM-Free) using clean-text probabilities.

4. Experimental Results

The authors evaluated JUMP on LLaDA-8B-Base fine-tuned on six domains from the MIMIR dataset (ArXiv, GitHub, HackerNews, Pile-CC, PubMed Central, Wikipedia).

Accuracy: JUMP improved the mean ROC-AUC from 0.82 (SAMA) to 0.90.
Low-FPR Detection: JUMP showed substantial improvements in True Positive Rate (TPR) at strict FPR levels (1% and 0.1%). For example, on the Wikipedia domain, JUMP achieved a TPR@0.1% of 0.68, compared to SAMA's 0.36.
Cost: JUMP reduced the query cost from $2T$ (where $T$ is typically large for SAMA) to a constant 2 (or 3 with the PRISM selector) forward passes per sample.
Ablation Studies:
- Selector Robustness: PRISM selectors trained on different generic corpora (C4, SlimPajama, FineWeb) yielded nearly identical performance, indicating the method captures generic reconstruction difficulty.
- Token Budget: The attack is robust to the number of selected tokens ( $K$ ), with $K=64$ (1/8th of a 512-token window) providing a stable operating point.
- Clipping: Clipping the reconstruction gaps was shown to be critical for improving performance in low-FPR regimes by suppressing outliers.
- PRISM-Free: The training-free variant achieved a mean ROC-AUC of 0.87, outperforming SAMA without any auxiliary training overhead.

5. Significance and Claims

The paper claims that JUMP establishes a rigorous baseline for future privacy evaluations of dLLMs. Its significance lies in demonstrating that:

Architecture Matters: The unique "any-order" and "parallel" properties of dLLMs can be exploited to create attacks that are both stronger and more efficient than those for autoregressive models.
Signal Localization: Membership leakage in dLLMs is not uniform; it is concentrated in positions where the reference model is uncertain. Targeting these positions is more effective than averaging over random subsets.
Defense Implications: The authors note that their results suggest Differentially Private (DP) fine-tuning (specifically DP-LoRA) offers a practical mitigation against this type of uncertainty-guided inference, building on existing DP-SGD frameworks.

The authors remain modest regarding the scope, noting that their evaluation focuses on fine-tuned dLLMs to isolate architectural vulnerabilities, and extending this analysis to pretraining-scale membership remains a direction for future work.