💬 NLP

Prompt Compression in Diffusion Large Language Models: Evaluating LLMLingua-2 on LLaDA

This study evaluates the transferability of the LLMLingua-2 prompt compression method to diffusion large language models (specifically LLaDA), revealing that while summarization tasks remain robust, mathematical reasoning degrades significantly due to the omission of critical reasoning information, indicating that autoregressive-focused compression strategies do not uniformly apply to diffusion architectures.

Original authors: Sterling Huang, Abigayle Brown, Jiyoo Noh, Jiakang Xu, Wantong Huo, Kaung Myat Kyaw, Jonathan Chan

Published 2026-05-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sterling Huang, Abigayle Brown, Jiyoo Noh, Jiakang Xu, Wantong Huo, Kaung Myat Kyaw, Jonathan Chan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart chef (the AI) who is trying to cook a complex meal based on a recipe you give them. Usually, this chef reads the recipe from start to finish, one word at a time, and cooks the dish step-by-step. This is how most current AI models work.

But there's a new type of chef, called a Diffusion Model (specifically one named LLaDA in this paper). Instead of reading the recipe line-by-line, this chef starts with a blank page covered in scribbles and slowly "denoises" it, refining the whole recipe all at once until the final dish appears.

The researchers wanted to see if a popular tool called LLMLingua-2, which is designed to shorten recipes for the "line-by-line" chefs, would work just as well for this new "all-at-once" chef.

Here is what they found, explained through simple analogies:

1. The "Shortened Recipe" Experiment

The team took long, detailed prompts (recipes) and used LLMLingua-2 to cut them down by about half. They kept the "main idea" but removed some words they thought were unnecessary.

The Goal: To save time and computing power (like reducing the cost of printing a long menu).
The Test: They fed these shortened recipes to the Diffusion chef (LLaDA) and asked it to do three things:
1. Solve Math Problems (like a tricky word problem about apples and oranges).
2. Summarize News (condensing a long article into a short blurb).
3. Reconstruct the Original (try to rewrite the full original recipe from the shortened version).

2. The Surprising Results: "Looks Good, Tastes Wrong"

The researchers discovered that for the Diffusion chef, a recipe that looks similar to the original doesn't always work the same way.

The "Summarization" Task (The Robust One):
When asked to summarize news or chat logs, the Diffusion chef handled the shortened recipes very well. Even though words were missing, the chef could still figure out the main story.
- Analogy: If you tell a chef, "Make a salad with lettuce and tomatoes," they can still make a great salad even if you forgot to mention the dressing. The core idea was enough.
The "Math" Task (The Fragile One):
When asked to solve math problems, the shortened recipes caused the chef to fail, even though the shortened text still sounded very similar to the original.
- Analogy: Imagine a math problem says, "I have 5 apples, and I give 2 to my friend." The compression tool might remove the word "2" because it thinks the sentence still makes sense without it. To a human reading for the "gist," it looks fine. But to the Diffusion chef, losing that specific number is like losing a crucial ingredient. The chef ends up guessing the wrong answer because a tiny, specific detail was missing.

3. The "Reconstruction" Puzzle

The researchers also asked the chef to take the shortened recipe and try to write the full original recipe back out.

The Result: The chef did a great job capturing the meaning (semantic similarity). If you compared the two texts, they looked 94% similar.
The Catch: Even though the meaning was there, the chef often missed the tiny, critical details needed to solve the math problem later. It's like a chef who can describe a cake perfectly but forgot to write down that it needs exactly 3 eggs, not 2.

4. Why Did This Happen?

The paper explains that the two types of chefs (Autoregressive vs. Diffusion) use the recipe differently:

The Line-by-Line Chef: Reads the first word, then the second. If a word is missing early on, it might not matter as much for the next step.
The All-at-Once Chef: Looks at the entire recipe at once to figure out what to write. If a specific number or relationship is missing from the "big picture," the chef gets confused immediately because they can't "fill in the blanks" later the way the other chef might.

The Bottom Line

The study concludes that you can't just use the same "recipe shaver" for both types of chefs.

Tools designed to shorten prompts for standard AI models (which read left-to-right) don't work perfectly for Diffusion models (which look at everything at once). While the shortened prompts still sounded very similar to the originals, they often stripped away the tiny, specific details that the Diffusion model needed to solve hard problems like math.

In short: For Diffusion models, "keeping the main idea" isn't enough. You have to keep the exact details, or the chef gets lost.

Technical Summary: Prompt Compression in Diffusion Large Language Models

Problem Statement

Prompt compression is a established technique for reducing inference costs and context lengths in Large Language Models (LLMs), primarily evaluated within autoregressive (AR) architectures. However, the efficacy of these compression methods when applied to Diffusion Large Language Models (DLLMs) remains largely unexplored. Unlike AR models that generate text sequentially via next-token prediction, DLLMs utilize a bidirectional denoising process over masked tokens. This architectural difference raises a critical question: do compression heuristics designed for autoregressive models (which prioritize sequential conditioning) preserve downstream behavior in diffusion models, which rely on prompt information throughout the iterative reverse denoising process? Specifically, tokens discarded by AR-centric compression might encode entities, numerical values, or discourse structures essential for stabilizing the denoising trajectory in DLLMs.

Methodology

This study investigates the transferability of prompt compression to DLLMs using LLMLingua-2 as the compression framework and LLaDA (an 8B-parameter instruction-following DLLM) as the target model. The experimental design isolates the effect of compression by maintaining identical decoding settings across all trials.

Experimental Pipeline:

Datasets: Three datasets were selected to cover distinct task structures:
- GSM8K: For mathematical reasoning and prompt reconstruction.
- DUC2004: For formal news-style summarization.
- ShareGPT: For conversational summarization.
- Sample Size: 250 prompts per dataset.
Compression: Prompts were compressed using LLMLingua-2 at an approximate 2× compression ratio (retaining ~50% of tokens).
Evaluation Conditions: The study compared four behavioral settings:
- Original Prompt: $r_{orig} = \text{LLaDA}(p_{orig})$
- Compressed Prompt: $r_{comp} = \text{LLaDA}(p_{comp})$
- Reconstructed Prompt: The compressed prompt was fed back into LLaDA to reconstruct the original prompt ( $\hat{p}_{orig}$ ), which was then re-evaluated for downstream reasoning ( $\hat{r} = \text{LLaDA}(\hat{p}_{orig})$ ).
Metrics:
- Mathematical Reasoning: Exact-match accuracy against ground truth and consistency between original/compressed responses.
- Reconstruction & Summarization: BLEU, ROUGE (1, 2, L), and BERTScore (Precision, Recall, F1).
- Hyperparameters: 128 denoising steps, max length 128, temperature 0.0, classifier-free guidance 0.0.

Key Contributions

First Evaluation of AR Compression on DLLMs: This work provides the first empirical assessment of whether prompt compression methods (specifically LLMLingua-2) transfer effectively from autoregressive to diffusion architectures.
Decoupling Semantic Preservation from Behavioral Stability: The study distinguishes between semantic similarity (measured by BERTScore) and downstream task performance. It demonstrates that high semantic similarity does not guarantee stable reasoning or generation in DLLMs.
Task-Specific Sensitivity Analysis: The research identifies that mathematical reasoning is significantly more sensitive to compression-induced information loss than summarization or reconstruction tasks, despite similar semantic similarity scores.
Diagnosis of Failure Modes: By analyzing BERTScore precision vs. recall, the study identifies that compression failures in DLLMs are driven primarily by information omission rather than semantic drift.

Results

1. Architectural Differences in Compression Transfer

Comparisons between autoregressive models and LLaDA (DLLM) reveal divergent behaviors:

Reconstruction (GSM8K): AR models outperformed DLLMs in lexical overlap (BLEU: 0.55 vs. 0.26; ROUGE-L: 0.86 vs. 0.52). However, both achieved identical BERTScore F1 (0.94), indicating DLLMs preserve semantic meaning despite weaker surface-level reconstruction.
Summarization (DUC2004): DLLMs outperformed AR models in ROUGE-L F1 (0.50 vs. 0.21) and BERTScore F1 (0.94 vs. 0.85), suggesting DLLMs better preserve content meaning even when wording differs.
Math Reasoning (GSM8K): LLaDA achieved higher accuracy with compressed prompts (0.356) compared to the AR baseline (0.220), though absolute accuracy remained low.

2. Task-Specific Sensitivity

Summarization: Remained robust under compression. DUC2004 and ShareGPT maintained high BERTScore F1 (0.944 and 0.910, respectively), indicating that broad semantic content is preserved even with ~46–47% token removal.
Mathematical Reasoning: Showed substantial degradation. While reconstructed prompts had high semantic similarity (BERTScore F1: 0.936), reasoning accuracy dropped significantly.
- Original prompt accuracy: 52.0%
- Compressed prompt accuracy: 35.6%
- Agreement between original and compressed responses: Only 35.2%.
- Partition Analysis: The largest outcome group (35.6%) consisted of cases where both original and compressed prompts produced different incorrect answers. In 27.6% of cases, the original was correct while the compressed prompt failed, suggesting the loss of specific reasoning-critical information.

3. Nature of Compression Failures

Across all tasks, BERTScore recall was consistently lower than precision. This indicates that compressed or reconstructed prompts tended to omit portions of the original semantic content rather than introducing unrelated information (drift). In mathematical tasks, these omissions removed critical anchors (quantities, relations) required for stable denoising, leading to divergent reasoning trajectories even when the remaining text appeared semantically coherent.

Significance and Claims

The paper claims that prompt compression methods designed for autoregressive models do not transfer uniformly to diffusion large language models.

Semantic vs. Behavioral Stability: High semantic similarity (e.g., high BERTScore) is insufficient to guarantee stable downstream behavior in DLLMs. The iterative denoising process appears highly sensitive to the precise distribution of conditioning information; even minor omissions can cascade into different final predictions.
Task Dependency: Summarization tasks are relatively tolerant to token removal, whereas mathematical reasoning is fragile. This suggests that DLLMs require not just semantic preservation but also the retention of precise conditioning cues (entities, numbers, constraints) to stabilize the denoising trajectory.
Need for Diffusion-Aware Strategies: Current compression heuristics, which often rely on autoregressive saliency or bidirectional context in a task-agnostic manner, may discard tokens that are redundant for sequential prediction but critical for global conditioning in diffusion models. The authors argue for the development of diffusion-aware compression strategies that prioritize denoising stability and the preservation of reasoning-critical structures over simple semantic overlap.

The study concludes that future evaluation frameworks for prompt compression must incorporate behavioral consistency and reasoning stability metrics alongside traditional semantic overlap measures, particularly for diffusion-based architectures.