🤖 AI

CIVIC: End-to-End Sequence Compactness for Efficient Vision-Language Models

CIVIC is an end-to-end framework that achieves genuine hardware efficiency in Vision-Language Models by maintaining a contiguous compact visual pathway across all inference stages, significantly reducing memory and latency without compromising accuracy.

Original authors: Fengze Yang, Bo Yu, Xuewen Luo, Cathy Liu, Chenxi Liu

Published 2026-05-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Fengze Yang, Bo Yu, Xuewen Luo, Cathy Liu, Chenxi Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to explain a complex scene from a movie to a friend.

The Problem: The "Over-Describing" Friend
Current AI models (called Vision-Language Models) are like friends who describe every single pixel of a movie frame. If the movie has a 4K resolution, they might try to describe 1,000 tiny details (tokens) for just one second of video. While this is detailed, it's exhausting. The AI has to remember all 1,000 details in its short-term memory (the "KV-cache") while it tries to write a story. This makes the AI slow, memory-hungry, and expensive to run, especially on smaller devices.

The Old Solution: The "Edit Button" Mistake
Previous attempts to fix this were like writing the full 1,000-word description first, then hiring an editor to cross out the boring parts after the fact.

The Flaw: Even though the editor crossed out words, the AI still had to write them down first, carry them around, and then delete them. It's like carrying a heavy backpack full of rocks, only to drop half of them at the finish line. You still got tired carrying the weight, and the "editing" process itself took extra time.

The New Solution: CIVIC (The "Smart Summarizer")
The paper introduces CIVIC, a new way to think about how AI sees and talks. Instead of writing the full description and then editing it, CIVIC teaches the AI to only write the important parts from the very beginning.

Here is how CIVIC works, using simple analogies:

The "Anchor" System (Smart Grouping):
Imagine looking at a crowded room. Instead of listing every single person's face, CIVIC picks a few "anchors" (like the center of a group of friends) and says, "This whole group represents one idea." It groups similar visual details together before the AI even starts processing them. This turns a messy crowd of 1,000 items into a neat list of 400 key points.
The "Continuous Path" (No Detours):
Most other methods are like a relay race where the runner passes the baton, stops to repackage it, and then runs again. CIVIC is a straight track. It keeps the "compact" (shortened) list of information all the way from the camera, through the projector, and into the AI's brain. It never switches back to the "heavy" version. This means the AI doesn't waste energy switching modes or reorganizing data.
The "Memory Saver" (KV-Cache):
Because the AI only has to remember 400 key points instead of 1,000, its short-term memory (the KV-cache) shrinks dramatically. The paper found that CIVIC uses only about one-third of the memory space compared to the standard method. It's like going from carrying a heavy suitcase to carrying a small backpack.
The "Teacher-Student" Lesson (Training):
To make sure the AI doesn't get confused by having fewer details, the researchers used a "text-aligned" teaching method. Imagine a teacher (the big, slow AI) showing a student (the new, fast AI) how to describe a picture. The teacher doesn't just say "be shorter"; they say, "Describe the meaning of the picture using fewer words, but make sure you still get the story right." This ensures the AI doesn't lose its ability to understand fine details, like where an object is located in a picture.

The Results
When the researchers tested this on a model called Qwen3-VL:

Speed: The AI finished its task much faster (dropping from ~3.5 seconds to ~2.5 seconds).
Memory: It used significantly less memory (dropping from ~122 MB to ~44 MB).
Accuracy: Despite being faster and smaller, it didn't get "dumber." It still answered complex questions and located objects in images just as well as the slow, heavy version.

In Summary
CIVIC stops the AI from doing unnecessary work. Instead of generating a massive amount of data and then trying to cut it down, it learns to generate a compact, efficient summary from the start. This makes AI faster, cheaper to run, and ready for real-world use without losing its smarts.

Technical Summary: CIVIC – End-to-End Sequence Compactness for Efficient Vision-Language Models

1. Problem Statement

Vision-Language Models (VLMs) face significant memory and latency bottlenecks when processing high-resolution images or long videos, primarily due to the massive number of visual tokens generated. While existing research has focused on token sparsification, merging, and KV-cache compression to reduce theoretical FLOPs, these approaches often fail to translate into proportional physical hardware acceleration.

The core issue identified is the "compression-realization gap." Most current methods apply post-hoc runtime compression after dense visual representations have already been constructed. This introduces structural overheads, including:

Non-contiguous memory access: Operations like token scoring, routing, gather/scatter, and dense-interface restoration disrupt GPU memory contiguity.
Geometric disorientation: Disrupting 2D grids during pruning can lead to the loss of fine-grained localization.
Ineffective downstream compression: Even if visual tokens are reduced in the encoder, downstream language model (LLM) prefill and KV-cache operations often continue to process uncompressed or partially dense interfaces, negating latency gains.

Consequently, theoretical FLOP reductions do not yield genuine wall-clock acceleration, and resource-constrained deployments remain bottlenecked by latency and KV-cache footprints.

2. Methodology: CIVIC Framework

To address these barriers, the paper proposes CIVIC (Compact Inference for Vision-Language Integrated Compression), a path-consistent framework designed to maintain compact sequence representations seamlessly across the entire inference pipeline: from the vision encoder and projection layer to the LLM prefill and KV-cache.

Core Components

Learned Anchor-Based Aggregation:
Instead of pruning tokens after encoding, CIVIC converts dense patches into consecutive tokens before visual encoding. It uses learned compact anchors ( $A$ ) to aggregate dense patch embeddings ( $X_0$ ) via a softmax-weighted mechanism. This preserves spatial mapping and converts the dense sequence into a compact latent sequence ( $Z_0$ ) without disrupting the 2D grid structure.
Compact Visual Encoding with KV Compression:
The compact visual encoder ( $E^c_v$ ) processes the aggregated sequence. Crucially, while the query projection remains standard, the keys and values are compressed into a fixed number of memory anchors ( $S$ ). This reduces the attention interaction complexity from $O(T_e^2)$ to $O(M_e \cdot S)$ , where $M_e$ is the compact token count.
Adaptive Spatial Retention Floor:
To prevent over-compression from destroying fine-grained localization, CIVIC employs an adaptive spatial retention floor. This ensures a minimum proportion of spatially relevant tokens is preserved, safeguarding visual accuracy during aggressive truncation.
Compact Multimodal Prefill:
The compact visual embeddings are projected directly into the language model's hidden space ( $\Pi_c$ ) and inserted into the LLM context. Unlike other methods that insert dense placeholders, CIVIC replaces these spans entirely with the compact sequence ( $V_c$ ), ensuring the LLM prefill and subsequent autoregressive decoding operate on a shortened sequence ( $L + M_p$ ) rather than the original dense length ( $L + T_p$ ).
Text-Aligned KL Distillation:
Since the visual token interface changes, standard distillation is insufficient. CIVIC utilizes a text-aligned KL distillation scheme. It aligns the logits of the compact student model with the dense teacher model only at non-visual text positions. This allows the compact embeddings to directly replace dense spans without requiring structural retraining of the frozen language model parameters.

3. Key Contributions

The paper outlines three primary contributions:

Identification of the Gap: The authors identify that theoretical FLOP reductions in VLMs fail to yield physical acceleration unless sequence compactness is preserved end-to-end across the vision encoder, projector, and LLM prefill.
Path-Consistent Framework: The proposal of CIVIC, which resolves geometric and training bottlenecks through learned anchor-based aggregation, an adaptive spatial retention floor, and text-aligned KL distillation to maintain end-to-end compact latent representations.
Empirical Validation: Evaluation on the Qwen3-VL architecture demonstrates that stable compact processing drastically reduces latency and KV-cache footprints while preserving fine-grained multimodal accuracy, eliminating the need for runtime routing or dense restoration.

4. Experimental Results

Experiments were conducted on the Qwen3-VL-2B-Instruct model using an NVIDIA RTX 4090 GPU, comparing CIVIC against five baseline token-reduction methods (DyMU, DiffRate, DynamicViT, VisionTrim, ZOO-Prune) across five benchmarks (MMMU, MathVision, ODinW-13, RealWorldQA, VideoMME).

Performance Metrics

Latency Reduction: CIVIC achieved a total inference latency of 2514.9 ms, a significant reduction from the baseline's 3543.0 ms. In contrast, many post-hoc methods (e.g., ZOO-Prune, DynamicViT) showed latency regressions or negligible improvements due to overhead.
KV-Cache Optimization: The KV-cache memory utilization was reduced from 122.7 MB (baseline) to 44.61 MB (CIVIC), approximately one-third of the baseline.
Sequence Compression: The prefill token count was reduced from 1122.2 to 407.9.
Accuracy Preservation: Unlike post-hoc pruning methods which suffered severe degradation on reasoning-heavy tasks (MathVision) and fine-grained localization (ODinW-13), CIVIC maintained stable performance boundaries and even enhanced localization accuracy in some cases.
Overhead: The compression overhead for CIVIC was negligible (0.49 ms), whereas baselines incurred significant overheads (e.g., 18.45 ms for ZOO-Prune) due to runtime bookkeeping.

Ablation Insights

Token Budget: Intermediate token budgets (e.g., C=256) offered the best balance between visual density and downstream workload; extreme compression (C=64) degraded decoding performance due to information starvation.
Retention Floor: An adaptive minimum keep ratio was critical; unbounded eviction (Min=0.0) destroyed visual accuracy.
End-to-End Consistency: Restoring dense placeholders before the LLM nullified all speedups, confirming that intermediate compactness is futile unless it persists into the LLM.

5. Significance and Claims

The paper claims that CIVIC successfully translates theoretical compression into physical hardware efficiency. By enforcing a contiguous compact pathway, the framework avoids the non-contiguous memory access and localized unmerging overheads that plague current methods.

The significance lies in demonstrating that holistic sequence compactness is the definitive driver of tangible VLM acceleration. The results suggest that for efficient VLM deployment, the entire inference graph must be optimized for compactness rather than treating compression as a post-hoc add-on. This approach enables larger batch sizes and extended context windows in resource-constrained scenarios without sacrificing the fine-grained multimodal reasoning capabilities required for practical applications like robotics and autonomous driving.

The authors remain modest regarding the scope, noting that current evaluations are restricted to static token budgets, single images, and the Qwen3-VL-2B architecture, with future work needed to validate scalability across larger models and dynamic, instance-adaptive contexts.