💻 computer science

CRISP: Pre-LLM Yet Text-Driven Visual Token Pruning for Efficient LVLM Inference

CRISP is a pre-LLM, text-driven visual token pruning framework that utilizes a two-stage pipeline to identify instruction-relevant tokens and enhance semantic diversity, achieving over 99.5% accuracy retention while reducing inference latency by more than 2x for Large Vision-Language Models.

Original authors: Xu Li, Yi Zheng, Mengyang Zhao, Yuxuan Liang, Zhe Liu, Rui Zhu, Xiaolei Chen, Wei Zhou, Baoquan Zhao, Juncen Guo

Published 2026-07-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xu Li, Yi Zheng, Mengyang Zhao, Yuxuan Liang, Zhe Liu, Rui Zhu, Xiaolei Chen, Wei Zhou, Baoquan Zhao, Juncen Guo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a mystery, but instead of a few clues, you are handed a library containing millions of books, all open at once. This is essentially how modern "Large Vision-Language Models" (LVLMs) work. These are super-smart computer programs that can look at a picture and answer questions about it, like a digital detective. To do this, they break the image down into thousands of tiny digital pieces called "tokens." Think of these tokens as individual puzzle pieces. The problem is that the computer tries to read every single piece of the puzzle, even the ones that are just blue sky or empty floor, before it even looks at your question. This makes the process incredibly slow and hungry for computer power, like trying to read a whole encyclopedia just to find out what color the cat's collar is.

For a long time, scientists tried to speed this up by either throwing away puzzle pieces randomly (which sometimes deleted the cat's collar) or by waiting until the computer had already started reading the whole book before deciding what to skip (which was too late to save time). The big question was: Could we teach the computer to look at the question first, then only pick the specific puzzle pieces needed to answer it, without slowing down the whole system?

This is where a new method called CRISP comes in. Think of CRISP as a super-efficient librarian who works before the detective even starts reading. Instead of dumping the whole library on the table, CRISP listens to your question first. If you ask, "What color is the handbag?", CRISP instantly scans the image and grabs only the puzzle pieces that show the handbag and the immediate area around it. It doesn't just stop there, though; it also grabs a few extra pieces to make sure the background context (like the street or the person holding it) isn't lost.

The researchers behind CRISP found that this "ask-first, then look" approach is much smarter than the old "look-first, then ask" methods. In their tests, they showed that CRISP could throw away up to 88.9% of the visual puzzle pieces (reducing the image from hundreds of tokens down to just 64 or 128) while still keeping the computer's accuracy almost exactly the same as if it had looked at the whole image. In fact, on some tests, it was so good at finding the right clues that it even reduced "hallucinations"—those moments when an AI makes things up that aren't there.

The paper explicitly argues against two common ways of doing this today. One way is to just pick the most "important" looking pieces of an image regardless of the question (text-agnostic), which the authors say is like staring at a painting before knowing what you are looking for. The other way is to wait until the computer is deep inside its thinking process to start deleting pieces, which the authors say wastes too much energy because the computer has already done the hard work of reading everything. CRISP rejects both of these, proving that you can be fast and smart at the same time by doing the pruning before the heavy thinking begins.

The results are measured and quite specific. On a popular model called LLaVA-1.5, CRISP kept 99.5% of the original performance while using only 128 tokens instead of the usual 576. On a newer model, LLaVA-NeXT, it kept 96.9% of the performance with just 320 tokens. Perhaps most impressively, this speed-up didn't just save accuracy; it cut the time it takes to get an answer by more than half. The authors suggest that this method is a practical, ready-to-use solution for making these smart image-reading computers much faster and cheaper to run, especially on devices that don't have massive amounts of power.

Technical Summary: CRISP – Pre-LLM Yet Text-Driven Visual Token Pruning

1. Problem Statement

Large Vision-Language Models (LVLMs) suffer from substantial inference overhead because they encode images into long sequences of visual tokens (often hundreds to thousands) that are fully processed by the Large Language Model (LLM). This creates a bottleneck for deployment in latency-sensitive or resource-constrained scenarios.

Existing visual token pruning methods face a fundamental trade-off:

Pre-LLM Pruning: Operates before the LLM using text-agnostic heuristics (e.g., CLS attention or feature similarity). While efficient, these methods are "text-agnostic," meaning they prune the same tokens regardless of the specific textual query. This mimics a "look-first-then-ask" cognitive strategy, which is less efficient than the natural "read-then-see" process and often discards query-relevant details.
Intra-LLM Pruning: Operates inside the LLM using cross-modal attention scores to prune tokens based on the current text. While text-driven, pruning at shallow layers suffers from positional bias and attention dispersion, degrading accuracy. Pruning at deeper layers preserves accuracy but requires visual tokens to traverse many LLM blocks, diminishing computational savings. Additionally, extracting attention weights disrupts FlashAttention, harming efficiency.

There is a need for a method that performs pruning before the LLM (to maximize efficiency) but remains text-driven (to preserve task-relevant information) without accessing the LLM's internal attention mechanisms.

2. Methodology: CRISP

The authors propose CRISP (CRItical-first and Semantically-covered Pre-LLM token pruning), a training-free framework that operates between the vision projector and the LLM. It utilizes a two-stage pipeline to select a compact subset of visual tokens ( $K$ ) that balances instruction relevance with scene context.

Stage 1: Critical-First Token Selection

This stage identifies visual tokens most relevant to the textual query.

Key Word Extraction: The system extracts nouns from the query (e.g., "skier," "gloves") using lightweight NLP processing, as nouns typically correspond to concrete image regions.
Cross-modal Relevance Computation: The projected visual tokens and the word embeddings (averaged from subword tokens) exist in the same semantic space. The system computes a cosine similarity matrix between visual tokens and word embeddings.
Word-level Quota Allocation: Not all words are equally grounded in the image. The system uses the CLS attention from the vision encoder to determine the importance of regions associated with each word. Words with higher CLS attention scores receive a larger token budget.
Selection: The top- $K$ visual tokens are selected based on the relevance to the specific words and their allocated quotas. If no nouns are detected, the system falls back to selecting tokens based purely on global CLS attention.

Stage 2: Semantically-Covered Token Completion

Relying solely on text-aligned tokens risks missing essential context (e.g., spatial layout, background).

The system iteratively selects additional tokens from the remaining pool to reach the total budget $K$ .
It selects tokens that are semantically complementary to the already selected set. Specifically, it chooses the token with the lowest maximum similarity to the current set, thereby maximizing semantic diversity and ensuring the preservation of essential scene context.

3. Key Contributions

Text-Driven Pre-LLM Pruning: CRISP bridges the gap between efficiency and relevance by performing pruning before the LLM while dynamically adapting to the textual query, avoiding the "text-agnostic" limitation of prior pre-LLM methods.
Two-Stage Pipeline: The combination of "Critical-First" selection (ensuring query alignment) and "Semantically-Covered" completion (ensuring contextual diversity) allows the model to retain critical evidence without sacrificing scene understanding.
Training-Free and Efficient: The method requires no additional training, does not access LLM attention weights (preserving FlashAttention compatibility), and operates entirely between the projector and the LLM.

4. Experimental Results

The authors evaluated CRISP on LLaVA-1.5-7B and LLaVA-NeXT-7B across nine benchmarks (including MME, GQA, SQA, POPE, and VQA-v2).

Performance Retention:
- On LLaVA-1.5-7B, with a pruning ratio of 77.8% (retaining only 128 tokens), CRISP achieved 99.5% of the full-token model's performance, outperforming the next-best method (97.9%).
- On LLaVA-1.5-7B, even under an aggressive 88.9% pruning ratio (64 tokens), it maintained 97.7% performance.
- On LLaVA-NeXT-7B, with an 88.9% pruning ratio (320 tokens), CRISP retained 96.9% of performance, surpassing all baselines.
- Notably, CRISP outperformed the full-token baseline on the POPE benchmark (object hallucination detection), suggesting that its focused selection effectively suppresses spurious correlations.
Efficiency Gains:
- CRISP reduced inference FLOPs and KV-cache memory usage nearly linearly with the reduction in token count.
- It achieved a >2× speedup in decoding speed and significantly reduced total end-to-end evaluation time on both models.

5. Significance and Claims

The paper claims that CRISP offers a practical and generalizable solution for efficient LVLM inference. By decoupling text-driven relevance from the LLM's internal processing, CRISP achieves a superior accuracy-efficiency trade-off compared to existing methods. The authors highlight that this approach is particularly valuable for resource-constrained scenarios and latency-sensitive deployments, enabling the use of high-performance LVLMs without the prohibitive computational cost of processing full visual token sequences. The results suggest that only a modest number of text-relevant tokens are needed to preserve critical evidence, provided that the remaining budget is used to maintain diverse contextual grounding.