🤖 AI

Recursive Language Models

This paper introduces Recursive Language Models (RLMs), an inference-time scaling paradigm that enables LLMs to programmatically decompose and recursively process arbitrarily long prompts, achieving performance significantly superior to existing long-context methods and frontier models while maintaining comparable costs.

Original authors: Alex L. Zhang, Tim Kraska, Omar Khattab

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Alex L. Zhang, Tim Kraska, Omar Khattab

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant but short-sighted librarian (the AI model). This librarian is incredibly smart and can answer almost any question, but they have a very strict rule: they can only hold about 272,000 pages of text in their hands at one time. If you try to hand them a library worth of books (millions of pages), they get overwhelmed, drop the books, and start forgetting the beginning of the story by the time they reach the middle. This is the current limit of most advanced AI models, known as the "context window."

The paper introduces a new way to work with this librarian called Recursive Language Models (RLMs). Instead of trying to force the librarian to hold the whole library at once, RLMs give the librarian a toolbox and a filing cabinet.

Here is how it works, using a simple analogy:

The Problem: The "One-Shot" Limit

Normally, if you ask a librarian to find a specific fact in a 10-million-page encyclopedia, they try to read it all in one go. Because they can't hold that much, they get confused, miss details, or give up. This is called "context rot"—the longer the text, the dumber the answer gets.

The Solution: The "Manager" with a Filing Cabinet

The RLM approach changes the librarian's job description. Instead of being a reader, the librarian becomes a project manager with access to a giant filing cabinet (a computer program called a REPL).

The Prompt is a File, Not a Handful of Paper: When you give the librarian a massive prompt (the 10-million-page book), the RLM doesn't shove it into the librarian's hands. Instead, it puts the book on a shelf in the filing cabinet and gives the librarian a label that says, "The book is in Box A."
Writing Instructions (Code): The librarian is allowed to write little notes (code) to interact with that box. They can say, "Open Box A, look at the first 50 pages, and tell me what's there."
The Recursive Loop (The "Sub-Manager"): If the librarian needs to check the whole book, they don't try to read it all at once. They write a loop:
- "Read the first chunk. Summarize it. Save the summary in a new folder."
- "Read the next chunk. Summarize it. Save that too."
- "Now, take all those summaries and combine them to answer the question."
Crucially, the librarian can even hire a junior librarian (a sub-call) to do the heavy lifting on specific chunks. The main librarian manages the process, while the junior librarian reads the specific pages and reports back.

Why This is a Big Deal

The paper tested this idea on some very difficult tasks:

The "Needle in a Haystack": Finding one specific sentence in a million pages.
The "Book Report": Reading a whole book and answering complex questions about the plot.
The "Code Detective": Understanding a massive software project with thousands of files.

The Results:

Superhuman Scale: The RLM system could handle inputs 10 times larger than the librarian's natural limit. It successfully processed millions of tokens (pages) where the standard librarian failed completely.
Better Quality: Even on shorter tasks, the RLM approach was smarter. It didn't just summarize; it could dig deep into specific details because it could "zoom in" on parts of the text programmatically.
Cost: Surprisingly, this didn't cost much more money. Because the system is efficient at breaking down the problem, it often used fewer resources than other methods that tried to force the whole text into the model's memory.

A Special Training Trick

The researchers also tried teaching a smaller, cheaper librarian (an 8-billion parameter model) how to use this filing cabinet system. They showed it 1,000 examples of how a super-smart librarian used the system.

The Result: The small librarian learned quickly. It became 28% better at solving long problems just by learning how to use the "filing cabinet" and the "sub-managers," even though it wasn't originally designed for it.

The Bottom Line

The paper argues that we don't need to build bigger, more expensive brains to handle massive amounts of text. Instead, we can teach our current smart brains to be better managers. By treating the text as an external object they can programmatically open, read, and summarize piece-by-piece, AI can effectively handle "infinite" amounts of information without getting overwhelmed.

What the paper does NOT claim:

It does not claim this works for medical diagnosis or legal advice (though it uses coding and research tasks as examples).
It does not say this will happen automatically in the future; it requires this specific "manager" setup to work.
It does not claim the AI is "conscious" or "thinking" in a human way; it's just a very clever way of organizing data processing.

Technical Summary: Recursive Language Models (RLMs)

Problem Statement

Frontier large language models (LLMs) face significant limitations in processing arbitrarily long prompts. Even within their context window limits, these models exhibit "context rot," where performance degrades steeply as prompt length and task complexity increase. Existing approaches to long-context processing generally fall into two categories:

Architecture/Training Modifications: Retraining models to handle longer sequences (e.g., via linear attention or state spaces).
Inference-Time Scaffolds: Methods like context compaction (summarization) or retrieval agents. However, compaction is often insufficient for tasks requiring dense access to the entire prompt, as it presumes early details can be safely forgotten. Retrieval and sub-agent delegation approaches are often handicapped by the underlying model's limited output lengths or lack of programmatic control over recursion.

The authors posit that scaling context size by orders of magnitude requires a paradigm shift where the prompt is treated not as a static input to a neural network, but as a dynamic part of an external environment that the model can interact with symbolically.

Methodology: Recursive Language Models (RLMs)

The paper introduces Recursive Language Models (RLMs), a general inference paradigm that offloads the user prompt into a persistent external environment (specifically a Python REPL) rather than feeding it directly into the model's context window.

Core Mechanism

Environment Initialization: Given a prompt $P$ , the RLM initializes a Read-Eval-Print Loop (REPL) environment where $P$ is stored as a string variable.
Symbolic Interaction: The base LLM ( $M$ ) is prompted to write code within this REPL. It is provided with metadata about $P$ (e.g., length) but does not receive the full text in its context window.
Programmatic Decomposition: The LLM writes code to inspect, decompose, and transform $P$ . Crucially, it can invoke a sub-LLM (or sub-RLM) programmatically on specific slices or chunks of the prompt variable.
Recursive Execution: The system executes the generated code. Intermediate results are stored in REPL variables. The LLM iteratively observes the output of these executions (truncated to fit the context) and generates further code until a final answer is produced.
Termination: The process stops when the LLM sets a Final variable in the REPL, which is returned as the response.

Key Design Choices

The authors distinguish RLMs from "ineffective" scaffolds (Algorithm 2 in the paper) through three critical design choices:

Symbolic Handles: The prompt is a variable in the environment, not text in the context window, allowing manipulation without copying.
Indirect Output: The model does not generate the final answer directly via a "Finish" action (which would be limited by context window size) but constructs it via variables in the REPL.
Symbolic Recursion: The code running in the environment can programmatically invoke the LLM on transformations of $P$ within loops, enabling $\Omega(|P|)$ or even $\Omega(|P|^2)$ semantic work, rather than just verbalizing a few sub-tasks.

Experimental Setup

The authors evaluated RLMs using frontier closed models (GPT-5) and open models (Qwen3-Coder-480B-A35B) across four tasks with varying complexity scales:

S-NIAH (Single Needle-in-a-Haystack): Constant complexity relative to input length.
BrowseComp-Plus: Multi-hop QA over 1,000 documents (linear complexity).
OOLONG: Semantic labeling and aggregation of a dataset (linear complexity).
OOLONG-Pairs: Aggregating pairs of chunks to construct answers (quadratic complexity).
LongBench-v2 CodeQA: Code repository understanding.

Baselines included direct LLM calls, CodeAct agents (with/without sub-calls), context compaction agents, retrieval agents, and coding agents (OpenCode, Claude Code).

Key Results

Scalability: RLMs successfully processed inputs exceeding 10 million tokens, far beyond the context windows of the base models (e.g., GPT-5's 272k limit).
Performance Gains:
- On OOLONG-Pairs (quadratic complexity), base models achieved near-zero performance ( $\le 0.1\%$ F1), while RLMs achieved 58.0% (GPT-5) and 23.1% (Qwen3-Coder).
- Across benchmarks, RLMs outperformed base models and common scaffolds by significant margins (e.g., +26% median over compaction, +130% over CodeAct with sub-calls on GPT-5).
- RLMs demonstrated robustness to "context rot," maintaining performance as input length increased, whereas base model performance degraded sharply.
Cost Efficiency: Inference costs for RLMs were comparable to or cheaper than base model calls and other scaffolds, despite the overhead of multiple sub-calls.
Fine-Tuning: A small-scale experiment fine-tuned RLM-Qwen3-8B on 1,000 trajectories distilled from a larger model. This model outperformed the base Qwen3-8B by a median of 28% and approached the quality of vanilla GPT-5 on three long-context tasks, demonstrating that RLM behavior can be learned with minimal data.
Reasoning Capabilities: On the LongCoT-mini benchmark, RLMs with decomposition hints improved solve rates by 69.5% over the base model, effectively orchestrating complex reasoning graphs.

Significance and Claims

The paper claims that RLMs represent a new axis of scaling for language models, distinct from increasing model parameters or context window sizes. By treating the prompt as an external object and enabling symbolic recursion, RLMs overcome the expressive power limitations of standard Transformers and existing agent scaffolds.

Key claims include:

Task-Agnostic Scalability: RLMs provide a general framework for handling arbitrarily long inputs without task-specific engineering.
Efficiency: The paradigm allows for dense processing of long contexts at costs comparable to standard inference.
Trainability: The authors provide initial evidence that models can be explicitly trained to act as RLMs, suggesting that "reasoning as RLM" could be a trainable capability that improves general performance and efficiency.
Limitations: The authors acknowledge current limitations, including the sequential nature of their implementation (leading to high latency), the brittleness of distinguishing final answers from thoughts in untrained models, and the potential for "exploding" sub-call costs if not managed. They note that asynchronous execution and sandboxed REPLs are future directions for optimization.

The work concludes that RLMs offer a promising path toward general-purpose language models capable of handling long-horizon tasks involving tens or hundreds of millions of tokens.