💬 NLP

Fix the Structural Bottleneck: Context Compression via Explicit Information Transmission

This paper introduces ComprExIT, a context compression framework that addresses structural bottlenecks in LLM agents by employing explicit, globally coordinated information transmission across frozen layers, achieving significant improvements in performance and efficiency over existing methods.

Original authors: Jiangnan Ye, Hanqi Yan, Zhenyi Shen, Heng Chang, Ye Mao, Yulan He

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiangnan Ye, Hanqi Yan, Zhenyi Shen, Heng Chang, Ye Mao, Yulan He

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Overstuffed Suitcase"

Imagine you are trying to send a massive library of books to a friend using a tiny, single-page postcard. This is the challenge Large Language Models (LLMs) face when dealing with very long texts (like a whole novel or a long legal contract).

Currently, these models try to fit all that information into their "working memory" (called the context window). But as the text gets longer, the model gets overwhelmed. It becomes slow, expensive to run, and starts to forget important details.

The Current Solution: The "Bad Summarizer"

To fix this, researchers have tried to teach the AI to act as a "summarizer." They give the AI a long text and ask it to squish it down into a few special "gist tokens" (tiny summary notes) that the AI can then read later.

The paper argues that current summarizers are failing.
Think of current methods like a group of students trying to summarize a 500-page book into a single paragraph, but they are all working in isolation:

The "Crowded Room" Problem (Width Bottleneck): Every student grabs a pen and tries to write down the most important parts. Because they aren't talking to each other, three students might write down the exact same sentence about the main character, while no one writes anything about the villain. Important parts of the story get ignored because everyone is fighting over the same "loud" parts.
The "Fading Memory" Problem (Depth Bottleneck): The students pass their notes down a line. As the notes travel from the first student to the last, the specific details (like a character's name or a specific date) get diluted and mixed up with general ideas. By the time the notes reach the end, the original details are gone.

The New Solution: ComprExIT

The authors propose a new system called ComprExIT. Instead of letting the AI struggle to summarize things on its own, they treat the AI like a frozen library and build a new, smarter delivery system around it.

They break the solution into two parts:

1. The "Smart Librarian" (Depthwise Transmission)

Instead of waiting for the AI to pass notes down a long line where details get lost, ComprExIT acts like a smart librarian who can jump to any shelf instantly.

How it works: The system looks at the AI's notes at every level of depth (from the very first page of the book to the last). It grabs the specific details from the early layers (where facts are clear) and the big-picture ideas from the later layers.
The Analogy: Imagine you are making a smoothie. Instead of blending everything together until the fruit turns into a brown sludge, you carefully pick the best berries from the bottom of the bowl and the best leaves from the top, then mix them perfectly. You keep the distinct flavors instead of losing them in the blend.

2. The "Traffic Controller" (Widthwise Transmission)

This is the biggest innovation. Instead of letting every "summary token" grab whatever it wants, the system uses a Traffic Controller (based on a math concept called "Optimal Transport").

How it works: Before the summary tokens are created, the system calculates a global plan. It asks: "Who needs to read which part of the book?"
- Token A is assigned to read the "Introduction."
- Token B is assigned to read the "Climax."
- Token C is assigned to read the "Character Descriptions."
The Analogy: Imagine a team of reporters covering a huge parade. In the old method, all reporters run to the same float because it's loud, and they miss the marching band. In the ComprExIT method, a commander hands out a map: "You go to the floats, you go to the band, you go to the marching units." Everyone covers a unique, non-overlapping part of the event, ensuring nothing important is missed.

The Results: Faster and Smarter

The paper tested this new system on 12 different datasets (mostly reading comprehension and question-answering tasks).

Better Accuracy: The new system got significantly higher scores (up to 18.5% better on average) than previous methods. It was much better at answering questions because it actually remembered the details.
Faster: It compressed the text more than 2 times faster than the previous fastest methods.
Lightweight: It didn't require a massive new brain; it only added about 1% more trainable parameters (like adding a small appendix to a book rather than rewriting the whole thing).

Summary

The paper claims that by stopping the AI from trying to "learn how to summarize" on its own, and instead using a coordinated plan to distribute information and grabbing details from multiple layers of the AI's memory, we can compress long texts much more effectively.

It's like replacing a chaotic group of people shouting summaries with a well-organized team using a map and a checklist to ensure every important detail is captured and delivered perfectly.

Technical Summary: ComprExIT

Problem Statement

Long-context Large Language Model (LLM) agents face significant challenges regarding token, memory, and latency costs as context lengths grow. While Context Compression (CC) methods aim to mitigate this by reducing context length, existing approaches based on the "LLM-as-a-compressor" paradigm (e.g., ICAE, 500x, Activation Beacon) remain noticeably inferior to using the full uncompressed context.

The authors identify that this performance gap stems from an inability to effectively preserve contextual information during compression. Through structural analysis of information flow in standard LLM-based compressors, they pinpoint two specific bottlenecks:

Widthwise Limitation (Lack of Allocation): In standard self-attention, gist tokens (compression tokens) aggregate information independently. This leads to a lack of coordination where multiple tokens may focus on the same context regions while others remain under-covered, resulting in redundant information capture and loss of coverage.
Depthwise Limitation (Layerwise Dilution): LLMs propagate information in a layer-wise accumulative manner. Fine-grained evidence captured in earlier layers often becomes entangled with later, more abstract, generation-oriented representations. Consequently, useful features from intermediate layers are diluted or lost by the time they reach the final layer, which is typically the only state used for compression.

Methodology: ComprExIT

To address these bottlenecks, the authors propose ComprExIT (Context Compression via Explicit Information Transmission). Unlike prior methods that tune an LLM to act as a compressor, ComprExIT treats a frozen LLM as a feature extractor and formulates compression as an explicit information transmission problem over its hidden states. The framework operates in two stages:

1. Depthwise Transmission (Mitigating Layerwise Dilution)

Instead of relying solely on the final layer's hidden states, ComprExIT adaptively aggregates features from all layers to construct Token Anchors.

Mechanism: For each token position, the method mixes layer representations using learned structural priors. It then employs a token-wise gating attention mechanism to selectively aggregate multi-layer features.
Goal: This allows the compressor to directly access fine-grained features from intermediate layers that are most relevant for the task, preventing them from being diluted by subsequent transformations. The gating is conditioned on the token, allowing different granularities of information to be preserved for different tokens and tasks.

2. Widthwise Transmission (Solving Lack of Allocation)

Once Token Anchors are established, ComprExIT explicitly allocates information from $N$ anchors to $K$ compression slots using a globally coordinated plan, rather than independent attention.

Utility Matrix: A utility matrix is constructed to measure the benefit of transmitting information from a specific anchor to a specific slot, based on cosine similarity between projected representations.
Information Capacity: Senders (anchors) are assigned learnable information capacities based on their contextual importance, while receivers (slots) have uniform capacity.
Optimal Transport Plan: The allocation is formulated as an optimal transport problem. The system solves for a transmission plan $\Pi$ that minimizes the cost (inverse of utility) subject to marginal constraints (sender/receiver capacities). This is solved efficiently using the Sinkhorn algorithm with entropy regularization.
Goal: This ensures a globally coordinated distribution where compression slots cover distinct context regions, avoiding redundancy and ensuring comprehensive coverage.

Key Contributions

Structural Analysis: The paper identifies and analyzes two key structural bottlenecks in existing LLM-as-a-compressor methods: the lack of allocation in the width dimension and information dilution in the depth dimension.
ComprExIT Framework: The authors propose a lightweight, parameter-efficient framework that decouples compression from the LLM architecture. It utilizes frozen hidden states, depthwise layer aggregation, and a coordinated widthwise transmission plan based on optimal transport.
Empirical Performance: The method is shown to consistently outperform strong soft-compression baselines across 12 datasets.

Experimental Results

Evaluated on 12 datasets (including 6 in-domain and 6 out-of-domain QA benchmarks) using Llama-3.2-1B and Llama-3.2-3B base models:

Performance: ComprExIT consistently outperforms state-of-the-art compression baselines (ICAE, 500x, Beacon). It achieves an average F1 improvement of 18.5% over the strongest compression baseline on the 1B model and 10.56% on the 3B model.
Comparison to Uncompressed: The performance remains very close to the uncompressed prompt-tuning baseline (within 0.25% on 1B and 0.64% on 3B).
Efficiency: The method adds only ~1% trainable parameters. In terms of latency, it achieves over 2× faster compression than the fastest baselines (ICAE) while maintaining competitive end-to-end decoding speeds.
Scalability: The method demonstrates robust performance under higher compression ratios (8× and 16×), on larger models (8B), and with longer contexts (8k).
Analysis: Ablation studies confirm that both the coordinated allocation (Optimal Transport) and depthwise aggregation are critical. Analysis of allocation patterns shows that ComprExIT reduces redundancy among compression tokens and improves the coverage of question-relevant entities compared to baselines.

Significance and Claims

The authors claim that ComprExIT represents a new paradigm for soft context compression. By treating compression as an explicit information transmission problem over frozen hidden states, the method overcomes the structural limitations of standard attention-based compression.

The paper posits that this approach provides a more flexible design space beyond standard attention formulations. The results suggest that effective compression does not require fine-tuning the LLM into a compressor but rather requires a mechanism to explicitly coordinate and aggregate the rich, multi-layered information already present in the frozen model's hidden states. The authors hope this work will inspire future research to explore more design choices within this new paradigm to further advance context compression.

Limitations: The authors acknowledge that they did not test on significantly larger backbones (e.g., 32B or 70B) or substantially longer contexts (e.g., 32k or 128k) due to computational constraints. Additionally, the current framework is limited to text-only models, with vision-language models left for future work.