🤖 AI

One Token per Multimodal Evidence: Latent Memory for Resource-Constrained QA

This paper introduces Latent Memory, a resource-efficient paradigm that compresses multimodal evidence into single latent tokens for retrieval and generation, significantly reducing token consumption and storage costs while maintaining competitive question-answering performance across text-only and multimodal benchmarks.

Original authors: Zhi Zheng, Ziqiao Meng, Hao Luan, Wei Liu, Wee Sun Lee

Published 2026-06-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhi Zheng, Ziqiao Meng, Hao Luan, Wei Liu, Wee Sun Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Heavy Suitcase"

Imagine you are a brilliant detective (the AI) trying to solve a mystery. To do your job, you need to read through a massive library of clues (text articles and photos).

In the current way of doing things (Existing RAG systems), every time you get a new question, the librarian hands you the entire raw library.

If the clue is a long article, you read every single word.
If the clue is a photo, the librarian doesn't just give you the photo; they describe every single pixel in thousands of words so you can "see" it.

The Result: You get overwhelmed. You run out of energy (storage) and time (processing speed) because you are carrying around heavy, uncompressed suitcases full of raw data, even if you only needed one small fact from them. This makes it impossible to use on small devices like phones or edge computers.

The Solution: The "Magic Summary Card" (Latent Memory)

The authors propose a new system called Latent Memory. Instead of handing you the heavy raw suitcases, they compress every single piece of evidence (whether it's a paragraph of text or a photo) into a single, tiny, magical summary card.

Think of it like this:

Old Way: You carry a 500-page book to find one sentence.
New Way: You carry a single index card that contains the essence of that book.

How It Works: The Three-Step Process

1. The Compression Station (The "Translator")
Before the AI ever sees the clues, a small, specialized assistant (a Compressor Model) looks at every piece of evidence.

It reads the text or looks at the photo.
It distills the entire meaning down into one single "token" (a high-dimensional number vector).
It throws away the original heavy book or photo and keeps only this tiny card.
Analogy: Imagine a master chef tasting a complex stew and writing down a single secret code on a napkin that perfectly captures the flavor. You don't need the whole pot of soup anymore; just the code.

2. The Search (The "Magic Compass")
When you ask a question, the system doesn't search through the heavy books. Instead, it turns your question into a similar "code" and uses a compass to find the matching summary cards in the drawer.

Because everything is now a single card, the search is incredibly fast and takes up very little space.

3. The Answer (The "Direct Feed")
The system takes the top few matching summary cards and hands them directly to the main AI (the Generator).

The main AI doesn't need to read the original text or see the original photo. It just reads the "secret code" on the card and instantly understands the context to give you the answer.
Crucial Point: The main AI doesn't need to be retrained. It just learns to "read" these new cards instead of the old books.

Why Is This a Big Deal?

1. Massive Efficiency (The "Lightweight Backpack")
The paper claims this method reduces the "token cost" (the amount of data the AI has to process) by 3 to 10 times.

Text: Instead of sending 200 words of context, the AI processes 1 token.
Images: Instead of expanding a photo into hundreds of "visual words," the AI processes just 1 token.
Result: You can run powerful AI on smaller devices (like phones) because the "backpack" of data is now feather-light.

2. It Still Works (The "Perfect Memory")
You might worry: "If we throw away the original book, will the AI forget the details?"

The authors trained the compressor using three specific tricks to ensure the "summary card" isn't just a vague idea, but a precise key:
- Reconstruction: The system tries to "rebuild" the original text or image description from the card to make sure the details are there.
- Contrast: It learns to make sure the card for "Annie Morton" is very different from the card for "Terry Richardson" so they don't get mixed up.
- Distillation: It teaches the card to act exactly like the original evidence would have acted if the AI had read the whole thing.

3. The Results

Text Questions: On standard reading comprehension tests, this method performed just as well as the heavy, slow methods but used 3 times fewer tokens.
Image Questions: On tests involving photos (like identifying objects in a picture), it was 10 times more efficient and actually performed better than other methods because it didn't get confused by the massive amount of data usually required to process images.

The Limitations (What the Paper Says)

The paper notes that this works best when evidence can be broken down into individual "atomic" units (one paragraph, one photo).

It might struggle with things that rely on complex structure, like a giant spreadsheet where the relationship between rows and columns matters, or a long video where the order of events is crucial. Compressing those into a single card might lose the "big picture" structure.

Summary

Latent Memory is like turning a library of heavy, raw books and photos into a sleek, digital card catalog where every single item is represented by one tiny, perfect code. This allows AI to answer complex questions using a fraction of the memory and speed, making powerful AI accessible on devices that previously couldn't handle it.

Technical Summary: One Token per Multimodal Evidence: Latent Memory for Resource-Constrained QA

Problem Statement

Large Language Models (LLMs) and Vision-Language Models (VLMs) rely on external memory to ground their outputs in relevant multimodal evidence for tasks like Question Answering (QA). However, the prevailing memory paradigm is computationally expensive and storage-intensive. Existing Retrieval-Augmented Generation (RAG) systems retrieve raw text and images, passing them uncompressed to the generator. This approach incurs substantial token consumption and latency, particularly for multimodal evidence where a single image can expand into hundreds of visual tokens during generation. These costs hinder the deployment of memory-augmented systems in resource-constrained environments, such as edge devices and on-line assistants, where storage and latency are critical constraints.

Methodology: Latent Memory

The paper proposes Latent Memory, a paradigm that replaces raw multimodal evidence items with compact, high-dimensional latent tokens. The core idea is to compress each unit of evidence (text or image) into a single latent token that serves a dual purpose: retrieval and generation.

Architecture and Pipeline

Memory Compression: A small, trainable compressor LLM/VLM ( $\theta$ ) encodes each evidence item $x_i$ into a single latent token $z_i$ . This is achieved by appending a learnable [MEM] token to the input and extracting its final hidden state.
Unified Retrieval Space: The latent tokens are projected into a lower-dimensional retrieval space via an MLP. At query time, the question is embedded into the same space, and relevant latent tokens are retrieved via similarity search (e.g., inner product).
Latent-Conditioned Generation: Retrieved latent tokens are projected into the hidden dimension of a frozen large LLM/VLM generator ( $\phi$ ) using a learnable projector. The generator produces the answer based on these latent tokens, bypassing the need to process raw evidence or expanded visual tokens.
Optional Reconstruction: To maintain interpretability, a decoder can reconstruct the original text or image captions from the latent tokens, though this is not required for generation.

Training Objectives

The compressor and associated projectors are trained end-to-end using a unified loss function combining three objectives:

Reconstruction Loss ( $L_{Recon}$ ): Ensures the latent token preserves sufficient information to reconstruct the original text or image embedding (using a teacher-forcing approach for text and CLIP embedding prediction for images).
Contrastive Retrieval Loss ( $L_{Contrast}$ ): Aligns the query embedding with its supporting positive latent evidence while pushing it away from negative evidence, shaping a unified retrieval space for mixed modalities.
Distillation Loss ( $L_{Distill}$ ): Aligns the generation behavior of the frozen generator when prompted with latent tokens against its behavior when prompted with raw evidence. This ensures the generator can interpret the latent tokens as valid evidence without fine-tuning the large generator itself.

Key Contributions

Latent Memory Paradigm: The authors propose a novel memory paradigm that compiles each multimodal evidence item into a single high-dimensional latent token, significantly reducing storage and token consumption.
Unified Training Objective: They demonstrate that a single latent token can simultaneously support retrieval, information preservation, and generation through a training objective combining reconstruction, contrastive learning, and distillation.
Efficiency-Performance Trade-off: The method achieves competitive QA performance compared to advanced RAG baselines while consuming significantly fewer generator tokens (3× to 10× reduction).

Experimental Results

The method was evaluated on seven text-only QA benchmarks (e.g., HotpotQA, 2WikiMultihopQA, MuSiQue) and multimodal benchmarks (WebQA).

Text-Only QA: Using a frozen Meta-Llama-3-8B-Instruct generator, the 1-token Latent Memory achieved competitive Exact Match (EM) and F1 scores compared to RAG baselines (BM25, Dense Retrieval, Qwen3-Embedding) while using approximately 3× fewer tokens. For instance, on out-of-domain averages, it used 71 tokens at $k=5$ compared to 209 for BM25 and 208 for Dense Retrieval.
Multimodal QA (WebQA): Using a frozen LLaVA-1.5-13B generator, Latent Memory delivered the strongest image-grounded QA performance. At $k=5$ , it achieved an image F1 of 69.4 with only 82 tokens, whereas the best baseline (Nemo-Emb) required 1,885 tokens to achieve a lower F1 of 53.0.
Scalability: The approach scales linearly with the number of evidence items during compilation and maintains low complexity during inference, avoiding the quadratic growth associated with full-context prompting.
Generalization: The Latent Memory trained on HotpotQA transferred effectively to out-of-domain datasets (2WikiMultihopQA, MuSiQue) without additional tuning, demonstrating robust retrieval capabilities.

Significance and Claims

The paper claims that Latent Memory offers a viable solution for resource-constrained applications by fundamentally changing how external memory is represented. By compressing multimodal evidence into single tokens, it addresses the dual bottlenecks of storage pressure (especially for images) and token consumption during generation.

The authors emphasize that this approach allows for the deployment of memory-augmented systems on devices with tight storage or latency constraints (e.g., edge AI) without requiring the fine-tuning of large, expensive generator models. The unified latent representation space simplifies the retrieval-generation pipeline, enabling efficient handling of long-context and multi-turn interactions. While the current design assumes evidence can be decomposed into atomic text or image units, the paper positions Latent Memory as a foundational step toward more efficient, multimodal agentic systems.