💬 NLP

GLASS: Global-Local Aggregation for Inference-time Sparsification of LLMs

GLASS is a training-free framework that improves inference-time sparsification for large language models by aggregating local prompt-specific activations and global model-intrinsic priors via rank aggregation, thereby stabilizing dynamic FFN pruning and significantly enhancing generation fidelity and decoding speed, particularly in short-prompt, long-form scenarios.

Original authors: Amirmohsen Sattarifard, Sepehr Lavasani, Kunlin Zhang, Amirhossein Rajabpour, Hanlin Xu, Fengyu Sun, Negar Hassanpour, Chao Gao

Published 2026-05-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Amirmohsen Sattarifard, Sepehr Lavasani, Kunlin Zhang, Amirhossein Rajabpour, Hanlin Xu, Fengyu Sun, Negar Hassanpour, Chao Gao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, incredibly smart library (a Large Language Model, or LLM) that can write stories, answer questions, and solve problems. But this library is so massive that it doesn't fit on your phone or laptop. It's too heavy, too slow, and requires too much energy to run.

To make it work on your device, you need to "prune" it—essentially, you need to tell it to ignore 50% of its internal brain cells (neurons) while it's thinking, so it runs faster and uses less memory.

The Problem: The "Guessing Game" of Short Prompts
Existing methods try to decide which brain cells to keep by looking at the very first thing you type (the "prompt"). They say, "Okay, you asked a short question, so we'll keep these specific neurons active."

The paper argues this is like trying to plan a whole road trip based on the first step you take out the door.

Short Prompts: If you ask a short question, the model doesn't have enough information to know which neurons are actually important for the long answer it's about to generate.
Long Answers: As the model starts writing a long story, the "importance" of the neurons changes. The ones that seemed important for the short question might be useless for the rest of the story.
The Result: The model gets confused, makes mistakes, and the quality of the writing drops because it kept the wrong neurons and turned off the right ones.

The Solution: GLASS (Global-Local Aggregation)
The authors propose a new method called GLASS. Think of GLASS as a smart manager who makes decisions using two different sources of information, rather than just one.

The Local View (The "Right Now" Signal): This looks at what you just typed. It asks, "Based on this specific question, which neurons are lighting up right now?" This is good for context but unreliable for short questions.
The Global View (The "Intrinsic" Signal): This is the paper's big innovation. Before the model even sees your question, the researchers ran a special test where they let the model talk to itself using an empty prompt (just a blank space). They asked, "Which neurons does this model always use, no matter what?" This creates a "cheat sheet" of the model's most important, reliable brain cells.

How GLASS Works: The "Ranking" Analogy
GLASS doesn't just pick one or the other. It uses a clever voting system called Rank Aggregation.

Imagine you are picking a team for a sports game:

Local Coach says: "Player A is great for this specific game."
Global Coach says: "Player A is the best player we have, period, based on their entire career."

If you only listen to the Local Coach, you might pick a player who is good for one play but bad for the whole game. If you only listen to the Global Coach, you might pick a star player who doesn't fit the specific strategy.

GLASS combines both. It takes the "Local" list and the "Global" list and merges them. If a neuron is ranked high by both coaches, it definitely stays. If one coach is unsure (like when the prompt is short), the other coach's opinion saves the day.

The "Null Prompt Stimulation" (NPS) Trick
To get that "Global" cheat sheet without needing a massive dataset of books or Wikipedia articles, the authors used a trick called Null Prompt Stimulation.

Instead of feeding the model a book to read, they just let it generate text from nothing (a "null" prompt).
This forces the model to rely entirely on its own internal wiring. It's like asking a musician to play a scale without sheet music to see which fingers they naturally use most. This gives a pure, unbiased map of the model's most critical neurons.

The Results: Faster and Smarter
The paper tested GLASS on many different models (like Llama, Gemma, and Mistral) and found:

Better Quality: When asked to write long stories from short prompts, GLASS made far fewer mistakes than previous methods. It was much closer to the full, un-pruned model's quality.
Faster Speed: On a Samsung Galaxy S25 Ultra (a high-end phone), GLASS made the model run up to 11 times faster in some cases. This happened because the model became small enough to fit entirely in the phone's fast memory, avoiding the slow process of constantly swapping data in and out.

In a Nutshell
GLASS is a "plug-and-play" tool that makes big AI models run on small devices. It fixes the problem of "guessing wrong" on short questions by combining what the model is doing right now with what the model is naturally good at, ensuring the AI stays smart and fast, even when it's working with limited resources.

Technical Summary: GLASS – Global-Local Aggregation for Inference-time Sparsification of LLMs

1. Problem Statement

Deploying Large Language Models (LLMs) on resource-constrained edge devices requires reducing memory traffic and computational demand. While static pruning permanently removes parameters, dynamic pruning (skipping computation based on input) offers a flexible alternative. However, existing training-free dynamic pruning methods for Feedforward Networks (FFNs) rely exclusively on prompt-derived signals (local activations during the prefill phase) to estimate neuron importance.

The authors identify a critical failure mode in these "prompt-only" approaches:

Unreliability on Short Prompts: For short prompts, the local activation signal is insufficient to accurately predict which neurons are critical for the subsequent long-form generation.
Mask Drift: Activation patterns during the decoding phase often drift significantly from the prefill mask, leading to inaccurate pruning masks.
Degraded Fidelity: This drift results in poor generation quality (high perplexity and KL divergence) during long-form decoding, particularly when the prompt is short.

Updating masks online to correct this drift is possible but introduces computational overhead and negates the benefits of keeping a fixed, compact subset of weights resident in fast memory (a key advantage for edge deployment).

2. Methodology: GLASS

The authors propose GLASS (Global-Local Aggregation for Sparsification), a plug-and-play, training-free framework that stabilizes dynamic FFN pruning by aggregating two complementary views of neuron criticality: local prompt-specific activations and a global model-intrinsic prior.

2.1 Global Prior Estimation via Null-Prompt Stimulation (NPS)

Instead of relying on external datasets (e.g., Wikipedia) which may introduce corpus bias, GLASS computes a global prior using Null-Prompt Stimulation (NPS).

Mechanism: The model generates text from itself using a null prompt (e.g., an empty string or BOS token) under typical sampling settings.
Benefit: This probes the model's intrinsic neuron utilization without external data bias.
Two Variants of Global Importance ( $M^g$ ):
1. A-GLASS (Activation Magnitude): Computes the average $\ell_2$ -normalized activation magnitude of neurons over the NPS sequences.
2. I-GLASS (Neural Impact): Computes the impact of removing a neuron on the cross-entropy loss using a first-order Taylor expansion (similar to Taylor pruning). Gradients are computed via teacher-forcing on the self-generated NPS sequence.

2.2 Rank Aggregation

GLASS fuses the local signal ( $A^l$ , derived from the input prompt) and the global signal ( $M^g$ , derived from NPS) using a weighted rank aggregation rule.

Ranking: Both local and global importance scores are converted to ranks ( $R^{(l)}$ and $R^{(g)}$ ) to handle differences in scale and distribution.
Probabilistic Interpretation: The authors interpret GLASS as finding the Maximum-A-Posteriori (MAP) consensus ranking under a permutation-based probabilistic model (specifically, a Mallows-type model with squared Spearman rank distance).
Fusion Formula: The final score for neuron $j$ is a weighted sum of its ranks:
$GLASS_j = (1 - \lambda)R^{(l)}_j + \lambda R^{(g)}_j$
Where $\lambda$ balances the reliability of local vs. global evidence. The default setting is $\lambda = 0.5$ , assuming equal reliability.
Selection: The top- $k$ neurons with the highest GLASS scores are selected to form the pruning mask, which is then applied during the decoding phase.

3. Key Contributions

The paper outlines four primary contributions:

Identification of Failure Modes: The authors demonstrate that prompt-only training-free sparsification fails in short-prompt, long-generation scenarios due to activation drift, leading to degraded generation fidelity.
GLASS Framework: Introduction of a plug-and-play, training-free framework that fuses local prompt evidence with a global model-intrinsic prior (computed via NPS) to produce robust masks.
Theoretical Foundation: A probabilistic interpretation of GLASS as a MAP consensus ranking under a Mallows-type model, providing a principled justification for the weighted rank-aggregation rule.
Empirical Performance: Demonstration of substantial improvements over prior training-free baselines (specifically GRIFFIN) in challenging scenarios, achieving up to 45.10% lower perplexity and 25.73% lower KL divergence at 50% sparsity, alongside significant on-device speedups.

4. Experimental Results

The authors evaluated GLASS on a diverse set of open-source LLMs (including Gemma, Llama, Mistral, OPT, Qwen, and Phi families) across classification, short-generation, and long-generation benchmarks.

Long-Form Generation (Alpaca Benchmark): GLASS significantly outperforms the strongest training-free baseline (GRIFFIN).
- Gemma 3n E2B: Achieved 45.10% lower perplexity and 25.73% lower KL divergence compared to GRIFFIN.
- Gemma 2 27B: Achieved 12.48% lower perplexity and 22.80% lower KL divergence.
- Llama 3 8B: Achieved 23.28% lower perplexity and 21.98% lower KL divergence.
Short-Form and Classification: GLASS performs comparably to GRIFFIN on standard classification and short-generation tasks, indicating that the global prior is most critical for long-generation stability.
Oracle Overlap Analysis: Experiments show that the Global-Local aggregation produces masks that are significantly closer to an "oracle" set of critical neurons (Jaccard similarity increased from ~0.52 to ~0.60) compared to using local or global signals alone.
NPS vs. WikiText: Using NPS for global prior estimation consistently outperforms using WikiText-based statistics, confirming that NPS provides more reliable, bias-free importance estimates.
On-Device Performance: Evaluated on a Samsung Galaxy S25 Ultra.
- For Gemma 7B, the 50% FFN reduction allowed the model to reside entirely in RAM, yielding an ~11× decoding speedup compared to the dense baseline (which suffered from memory swapping).
- For Llama 3 8B, a 42% speedup was observed.

5. Significance and Claims

The paper claims that GLASS unlocks reliable speedups for edge devices, particularly for workloads characterized by short prompts and long generations, which are common in consumer applications.

Robustness: By fusing stable, model-intrinsic global knowledge with context-specific local evidence, GLASS mitigates the "drift" problem that plagues prompt-only methods.
Efficiency: It maintains the benefits of static masks (fixed weight residency in fast memory) while dynamically adapting to input contexts without the overhead of training auxiliary predictors or recomputing masks per token.
Generality: The method is model-agnostic and training-free, making it applicable to a wide range of pre-trained LLMs without fine-tuning.
Theoretical Soundness: The method is not an ad-hoc heuristic but is grounded in a probabilistic consensus ranking model.

The authors conclude that model-intrinsic global knowledge is a crucial, previously underutilized resource for improving prompt-driven sparsification, enabling efficient inference on resource-constrained hardware.