💬 NLP

ARC-Encoder: learning compressed text representations for large language models

The paper introduces ARC-Encoder, an adaptable encoder that compresses text into continuous representations to replace token embeddings in decoder LLMs, achieving state-of-the-art performance across various scenarios while significantly improving inference efficiency and generalizing across multiple decoder models without requiring architecture modifications.

Original authors: Hippolyte Pilchen, Edouard Grave, Patrick Pérez

Published 2026-07-30

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Hippolyte Pilchen, Edouard Grave, Patrick Pérez

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to read a massive, thousand-page encyclopedia to answer a single question. Your brain (or in this case, a super-smart computer program called a Large Language Model) has to hold every single word in its memory at once to understand the story. The problem is, these computers get overwhelmed when the story gets too long. They start to slow down, get confused, or simply hit a "wall" where they can't remember anything past a certain point. This is a bit like trying to carry a library in your backpack; eventually, the straps snap, or you just can't walk anymore.

To fix this, scientists have been trying to teach these computers to "summarize" the library before they read it. Some methods try to throw away words they think are unimportant (like deleting pages from the book), while others try to turn the whole book into a single, dense "magic token" that holds all the meaning. But here's the catch: most of these magic tricks require rebuilding the computer's brain entirely to understand the new summary. It's like teaching a dog to speak French, but you have to surgically alter its brain to do it. This makes the dog great at French but terrible at its original tricks, and you can't just use that same brain on a different dog.

This is where a new study from researchers at Kyutai comes in. They asked a simple question: What if we could build a separate "translator" that turns the long book into a short, compressed version, without ever touching the computer's brain? They created a tool called ARC-Encoder. Think of it as a super-efficient librarian who reads a 1,000-page book and hands you a 125-page cheat sheet that contains all the same important ideas, but in a secret code the computer already knows how to read. The computer doesn't need to be changed at all; it just reads the cheat sheet instead of the whole book.

The researchers found that this approach works surprisingly well. By using a specific technique called "pooling"—which is like taking every four words and blending them into one super-word—they could shrink the text by a factor of 4 or 8. When they tested this on different types of computers (called "decoders"), the ARC-Encoder was able to help them answer questions, translate languages, and summarize stories just as well as if they had read the full text, but much faster. Even cooler, they discovered that one single "translator" could be trained to work with multiple different computers at the same time. It's like having one universal adapter that fits into any device, rather than needing a custom charger for every single gadget.

The study suggests that this method is a flexible way to make AI faster and smarter without breaking its original abilities. While the researchers admit it still needs practice to handle extremely long documents perfectly, they showed that by compressing the text, they could save a lot of computing power and memory. In fact, they calculated that storing these compressed summaries of the entire English Wikipedia would take up about the same space as the raw text itself, meaning we could pre-compute these summaries and use them instantly whenever needed. It's a promising step toward making AI that can read the whole library without getting tired, all while keeping its original personality intact.

Technical Summary: ARC-Encoder

Problem Statement

As Large Language Models (LLMs) are increasingly deployed for tasks requiring long contexts—such as Retrieval-Augmented Generation (RAG) and Chain-of-Thought reasoning—the computational cost of inference rises significantly due to the quadratic complexity of Transformer attention mechanisms. While context compression offers a solution to reduce input length, existing methods face critical limitations:

Hard Compression: Methods like LLMLingua prune or summarize tokens. While interpretable, they offer limited compression ratios.
Soft Compression: Methods using "memory tokens" or "gist tokens" encode context into dense vectors. However, most effective soft compression approaches require fine-tuning the target decoder or modifying its architecture. This adaptation often degrades the model's general capabilities when not used for the specific compression task and necessitates training a dedicated encoder for each individual decoder, hindering portability.

Methodology

The authors propose ARC-Encoder (Adaptable text Representations Compressor), a method that compresses text into continuous representations which replace raw token embeddings in a frozen decoder LLM. The core design ensures the decoder remains unmodified, preserving its original generative and few-shot capabilities.

Architecture

The system comprises three main components:

Text Encoder: Based on an LLM transformer (specifically Llama3.2 3B in the primary experiments) with the output head and causal mask removed.
Pooling Mechanism: A novel operation performed within the self-attention module of the encoder's final layer. Instead of using fixed memory tokens, the encoder averages consecutive query vectors to reduce the sequence length by a fixed pooling factor ( $x$ , typically 4 or 8). Keys and values remain uncompressed, while queries are merged. This allows the output size to scale linearly with input length.
MLP Projector: A lightweight, 2-layer Multi-Layer Perceptron (without activation) that maps the encoder's output to the hidden dimension of the target decoder. This acts as a dimensional bottleneck and adapter.

Training Strategy

The training process involves two distinct stages to ensure the compressed representations are both reconstructible and useful for downstream generation:

Pretraining: The model alternates between two tasks:
- Reconstruction: The decoder is teacher-forced to replicate the full text tokens from the compressed representations.
- Continuation: Subsequences of text are replaced by compressed representations, and the decoder is trained to continue the text from that point.
- Special learned tokens (<Rec> and <Cont>) are appended to specify the task.
Fine-Tuning: The encoder is fine-tuned on specific downstream tasks (QA, translation, summarization) while the decoder remains frozen. To preserve few-shot learning capabilities, the training data interleaves compressed documents with full-text queries and answers.

Multi-Decoder Adaptation

A key innovation is the ability to train a single shared encoder to work with multiple decoders (e.g., Mistral 7B and Llama3.1 8B). This is achieved by keeping the encoder weights shared while learning a separate, lightweight MLP projector and specific task tokens for each decoder. These decoder-specific parameters account for less than 1% of the total encoder parameters.

Key Contributions

ARC-Encoder Method: A novel approach to compute compressed text representations that replace raw input in LLMs without modifying the decoder. It achieves strong performance across benchmarks (QA, translation, summarization) while reducing input sequence length.
Cross-Decoder Compatibility: Demonstration that a single encoder can be trained to support multiple decoders simultaneously. Adapting to a new decoder requires training only a small MLP projector (<1% of encoder parameters), enabling a "portable" compression solution.
Context Window Extension: The method successfully extends the context window of decoders (e.g., enabling a 4k-token model to process 32k tokens) by compressing large documents into chunks processed in parallel.
Training Insights: The authors establish that both pretraining (for alignment) and fine-tuning (for task specialization) are critical. They also find that pretraining with a higher pooling factor than used in fine-tuning yields better results.

Results

Performance: ARC-Encoder achieves performance comparable to or exceeding baselines (ICAE, xRAG, PISCO, LLMLingua2) on standard benchmarks (HotpotQA, Natural Questions, SQuAD, FLORES, CNN-DailyMail) while operating on base models.
Efficiency: The method reduces prefilling computational costs (FLOPs) by approximately 1.8x to 1.9x compared to processing full text, depending on the pooling factor.
Memory: The authors show that compressed representations of Wikipedia require storage space comparable to the raw text (approx. 20–80 GiB vs. 24 GiB for raw text), making pre-computation and storage viable for RAG systems.
Generalization: The model demonstrates strong out-of-domain generalization on biomedical datasets (BioASQ, PubMedQA) where other compressors struggle, likely due to the robustness of the shared encoder and minimal decoder modification.
Long Context: In long-context benchmarks (ZeroSCROLLS), ARC-Encoder paired with a standard Llama2 Chat model outperforms models specifically fine-tuned for long contexts (e.g., Llama2-32k) on certain tasks, while leaving the decoder architecture untouched.

Significance and Claims

The paper positions ARC-Encoder as a flexible and efficient solution for portable context compression. Its primary significance lies in decoupling compression from decoder modification. By preserving the decoder's original weights, the method avoids the "catastrophic forgetting" often associated with fine-tuning decoders for specific tasks and maintains compatibility with few-shot learning and standard evaluation protocols (e.g., exact match in 5-shot settings).

The authors claim that ARC-Encoder enables the creation of universal compressed representations that can be adapted to various LLMs with minimal overhead, offering a practical path toward efficient long-context processing without the need for architectural changes to the target models. They acknowledge limitations, noting that the method's effectiveness is currently dependent on task formatting in the fine-tuning data and that higher compression ratios (beyond 8x) or extremely long contexts (e.g., 128k) remain areas for future work.