🤖 machine learning

Retrieval from Within: An Intrinsic Capability of Attention-Based Models

The paper introduces INTRA, a framework that unifies retrieval and generation within a single attention-based encoder-decoder model by enabling it to directly retrieve and reuse pre-encoded evidence from its internal representations, thereby outperforming traditional separate retrieval-augmented generation pipelines.

Original authors: Elad Hoffer, Yochai Blau, Ron Banner, Daniel Soudry, Boris Ginsburg

Published 2026-05-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Elad Hoffer, Yochai Blau, Ron Banner, Daniel Soudry, Boris Ginsburg

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a complex mystery. In the traditional way of doing this (called RAG, or Retrieval-Augmented Generation), you have two distinct helpers:

The Librarian: A specialized expert whose only job is to run through a massive library, find the few books that might contain the answer, and hand them to you.
The Detective: A brilliant investigator who reads those specific books and writes the final report.

The problem with this setup is that the Librarian and the Detective speak different languages. The Librarian uses a specific filing system (embeddings) to find books, but the Detective has to re-read and re-interpret those books from scratch before they can write the report. Sometimes, the Librarian hands over the wrong books, or the Detective struggles to understand the books because they were filtered by someone else's logic.

The Paper's Big Idea: "INTRA"

This paper introduces a new system called INTRA (INTrinsic Retrieval via Attention). Instead of hiring a separate Librarian, INTRA turns the Detective into a Librarian-Detective hybrid.

Here is how it works, using simple analogies:

1. The "Internal Library" (Pre-encoding)

Imagine the Detective has already read every book in the library and created a set of mental notes (called "encoded states") for each one. These notes capture the essence of the book without needing the physical pages.

Traditional RAG: When you ask a question, the Librarian finds the books, hands them to the Detective, and the Detective has to read the physical pages again to make notes.
INTRA: The Detective already has the mental notes ready on a shelf. When you ask a question, the Detective doesn't need to re-read the books; they just pull the pre-made notes off the shelf.

2. The "Self-Scoring" Mechanism

In the old system, the Librarian decides which books are important based on a separate rulebook. In INTRA, the Detective uses their own internal intuition (the "attention" mechanism) to decide which mental notes are relevant.

The paper argues that the part of the AI that usually "pays attention" to words is actually a built-in retrieval system. It's like the Detective's brain naturally highlighting the most important sentences in their mental notes without needing an external tool to do the highlighting.
The system adds a few special "search tokens" (like mental sticky notes) that help the Detective scan their own internal library and say, "Ah, this specific note is exactly what I need for this question."

3. The "One-Stop Shop" Advantage

Because the same brain (the model) does both the searching and the answering, two major problems disappear:

No Translation Loss: The Detective doesn't have to translate the Librarian's findings into their own language. They are speaking the same language because they are the same person.
Reusing Notes: If the Detective has already made mental notes for a book, they can use those same notes for a thousand different questions. They don't have to re-read the book every time.

The Results: What the Paper Found

The researchers tested this on difficult "multi-hop" questions (questions that require connecting dots between different pieces of information, like a detective connecting clues from different crime scenes).

Better at Finding Clues: INTRA was better at finding all the necessary pieces of evidence compared to systems that use a separate Librarian. It was especially good at complex puzzles where you need to stitch together information from multiple sources.
Better Answers: Because the Detective found the right clues and didn't have to waste time re-reading them, the final answers were more accurate.
Speed: Since the "mental notes" are already made, the system is faster at generating an answer once the search is done. It skips the step of re-reading the raw text.

The Catch (Limitations)

The paper is clear about what this system doesn't do yet:

It works best with a fixed library (a specific set of documents prepared in advance). It isn't designed to go out and search the live, changing internet right now.
It relies on a specific type of AI architecture (Encoder-Decoder) that is less common than the "Decoder-only" models (like the ones you might chat with daily).
It is currently a proof-of-concept that shows the "retrieval" ability is already inside the model; it just needs to be unlocked.

In Summary:
The paper claims that we don't always need to build a separate "search engine" to help AI models find information. Instead, we can teach the model to use its own internal "attention" powers to search its own memory, making the process faster, more accurate, and more efficient. It turns the AI from a "Reader who needs a Librarian" into a "Self-Searching Detective."

Technical Summary: Retrieval from Within: An Intrinsic Capability of Attention-Based Models

1. Problem Statement

Retrieval-Augmented Generation (RAG) has become the standard approach for answering questions where the necessary information is sparse within a large corpus. However, current RAG architectures typically treat retrieval and generation as separate, modular systems. A retriever selects candidate evidence (often using dense embeddings or lexical methods), which is then passed to a generator (LLM) that re-encodes the text to produce an answer.

This separation introduces two primary inefficiencies:

Representation Mismatch: The retriever and generator operate in different representation spaces, potentially leading to a disconnect between what is retrieved and what the generator finds most useful.
Computational Redundancy: In standard RAG, selected evidence chunks must be re-encoded by the generator's encoder at query time, even if they were already encoded during the retrieval phase.

The authors ask whether an attention-based encoder-decoder model can intrinsically perform retrieval from its own internal representations, thereby unifying the retrieval and generation processes within a single model without an external retriever module.

2. Methodology: INTRA

The paper introduces INTRA (INTrinsic Retrieval via Attention), a framework where a single pretrained encoder-decoder model handles both evidence selection and answer generation using a shared representation space.

Core Mechanism

Instead of an external retriever, INTRA utilizes the decoder's cross-attention mechanism to score pre-encoded evidence chunks. The process involves:

Pre-encoding: The corpus is encoded once by the frozen encoder, producing a set of chunk representations $K = \{k_i\}$ .
Retrieval Tokens: The input query $x$ is augmented with $R$ learnable retrieval tokens $\rho$ .
Scoring via Attention: The decoder processes the augmented query against the pre-encoded chunks. The cross-attention queries ( $q_\ell$ ) from the decoder layers are used to score the chunks.
MaxSim Scoring: To convert token-level attention scores into chunk-level retrieval scores, the authors employ a MaxSim (scaled ColBERT-style late interaction) approach. For each query token, the maximum similarity to any token in a chunk is taken, and these are aggregated across layers using learnable weights $\alpha_\ell$ .
Selection and Generation: The top- $n$ chunks based on these scores are selected to form the context $K(S_{INTRA})$ , which is then used by the decoder to generate the answer.

Key Technical Innovations

Reverse-QWK (Query-Key Weight): Standard Transformer cross-attention applies layer-specific linear projections and normalization to the keys (encoder states). This would require re-computing keys for every layer, preventing the reuse of a single pre-encoded index. INTRA introduces Reverse-QWK, a technique that moves the learned key scale and projection matrix to the query side. This allows the model to store a single normalized encoder representation ( $\bar{K}$ ) that serves all decoder layers, enabling efficient retrieval scoring against a static index.
Pooled Embeddings: To reduce the computational cost of MaxSim over long chunks, the authors replace full chunk representations with fixed-length mean-pooled embeddings ( $L_p \ll L_c$ ). This approximation preserves the shared-representation design without requiring a separate compression model.
Training Objective: The retrieval tokens and aggregation weights are trained using a soft cross-entropy objective to maximize the probability mass on "oracle" (ground truth) evidence chunks, while the backbone encoder-decoder remains frozen.

3. Key Contributions

Unified Framework: The formulation of INTRA, which couples evidence selection and answer generation within a single pretrained encoder-decoder model using a shared representation space.
Minimal Architectural Recipe: Identification of the necessary components to expose intrinsic retrieval: reusing native chunk representations, using encoder-side late interaction for coarse retrieval, and refining evidence via decoder-side retrieval queries without external modules.
Empirical Performance: Demonstration that this unified design outperforms strong engineered retrieval pipelines (including dense retrievers and rerankers) on multi-hop question-answering benchmarks, achieving higher evidence recall and end-to-end answer quality.
Computational Efficiency: Characterization of the design's efficiency, specifically the "reusable-context" regime where static evidence is encoded once and reused, eliminating the re-encoding cost during the generation prefill phase.

4. Experimental Results

The authors evaluated INTRA on four Wikipedia-based QA benchmarks: HotPotQA, 2WikiMultihopQA, MuSiQue, and Natural Questions (NQ). The model used was a T5Gemma2 4B-4B checkpoint.

Retrieval Quality: INTRA achieved the highest Complete-Evidence Recall (the fraction of examples where all supporting facts are retrieved) on multi-hop benchmarks (HotPotQA, 2Wiki, MuSiQue). It outperformed sparse methods (TF-IDF, BM25), dense single-vector models (BGE, Qwen-Embedding), and hybrid RAG pipelines.
End-to-End QA: When using the same T5Gemma2 decoder for generation, INTRA surpassed all baselines in Exact Match (EM) and F1 scores on multi-hop tasks. Notably, INTRA achieved this using a frozen decoder pretrained only for generation, whereas baseline retrievers were often pretrained on large-scale retrieval corpora.
Efficiency: Benchmarks measuring Time-to-First-Token (TTFT) showed that as the number of retrieved chunks ( $k$ ) increased, INTRA remained significantly faster than standard RAG. This is because INTRA reuses pre-encoded states, avoiding the quadratic cost of re-encoding retrieved text during the generation prefill phase.
Ablation Studies: Removing the initial context or reducing the number of retrieval tokens significantly degraded performance, confirming the necessity of both the initialization and the learned retrieval scoring mechanism.

5. Significance and Claims

The paper posits a conceptual shift in how retrieval is viewed in large language models. The authors claim that attention-based models already possess an intrinsic retrieval mechanism that can be elicited rather than added as an external module.

Unified Representation: By unifying retrieval and generation, INTRA eliminates the representation mismatch between retrievers and generators.
Intrinsic Capability: The results suggest that the "query-conditioned matching" inherent in attention mechanisms is sufficient for effective retrieval when properly exposed via learnable tokens and late-interaction scoring.
Efficiency: The framework offers a practical computational advantage for static knowledge bases by amortizing the cost of encoding evidence across queries.

Limitations

The authors explicitly note that their experiments focus on a fixed context pool and do not claim INTRA is a replacement for RAG in open-web or dynamic retrieval settings. The method relies on encoder-decoder cross-attention and is not directly applicable to decoder-only models. Furthermore, while the approach is effective on short-answer QA, its generalization to broader domains or dynamic corpora remains an open question for future work.