🤖 AI

Memory in the Loop: In-Process Retrieval as ExtendedWorking Memory for Language Agents

This paper argues that relocating memory retrieval from a slow, external network store to a fast, in-process store transforms it into effective extended working memory for language agents, thereby eliminating the latency-induced performance degradation that currently forces agents to limit retrieval to once per turn.

Original authors: Yusuf Khan, Carlo Lipizzi

Published 2026-07-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yusuf Khan, Carlo Lipizzi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Core Problem: The "Slow Librarian" vs. The "Notebook"

Imagine you are trying to solve a complex puzzle. You have a brilliant assistant (the AI agent) who is very smart but has a very short attention span. To help them, you give them a notebook (the "context window") where they can write down important facts.

The Old Way (Current AI):
Currently, if the assistant forgets something or needs to check a fact that isn't in their immediate notebook, they have to stop, walk over to a massive library down the street (a cloud database), ask a librarian for the book, wait for the librarian to find it, and walk back.

The Problem: This trip takes a long time (about 100 milliseconds). If the assistant has to check the library every single time they think of a new step, the whole process grinds to a halt. Because the trip is so slow, engineers tell the assistant: "Only go to the library once at the very beginning of the task, then just guess the rest." This leads to mistakes because the assistant forgets things that happened during the task.

The New Idea (Memory in the Loop):
The authors of this paper asked: "What if the library wasn't down the street, but right on the assistant's desk?"
They built a tiny, super-fast memory system that lives inside the assistant's brain (in-process). Checking this memory takes only 0.0001 seconds (100 microseconds).

The Result: The assistant can now check their memory every single time they take a step, without slowing down. It's like having a notebook that updates itself instantly, rather than having to walk to a library.

The Big Discovery: Speed Changes How We Think

The paper argues that speed isn't just about saving time; it changes the nature of the memory.

The Analogy of the "Tool" vs. The "Mind":
- If you have to walk 10 minutes to get a tool, that tool is an external object. You have to plan to use it, and you might forget to bring it.
- If the tool is in your pocket and you can grab it instantly, it becomes part of your body. You use it automatically without thinking.
- The paper claims that because this new memory is so fast, it stops being a "database the AI consults" and becomes "working memory the AI has." It allows the AI to reason continuously without breaking its flow.

The Experiments: Proving It Works

The researchers tested this with two main scenarios:

1. The "Trip Planner" Test (Can it remember facts?)

The Setup: The AI had to plan a trip based on 5 specific rules (e.g., "No dogs," "Must be cheap"). The conversation was long, so the rules would eventually "fall off" the AI's short-term notebook (the context window).
The Result:
- Without the fast memory, the AI forgot the rules completely (0 out of 5 correct).
- With the fast memory, the AI remembered most of the rules (3.6 to 4.8 out of 5 correct).
- Note: The researchers also found that if you just told the AI to "repeat the rules in every sentence," it worked perfectly. But this is expensive (like rewriting a whole book every time you add a page). The fast memory achieved similar results without that heavy cost.

2. The "Loop Guard" Test (Can it stop repeating mistakes?)

The Setup: The AI had to check a list of security alerts. Some alerts were duplicates of ones it had already seen. The goal was to stop the AI from investigating the same thing twice.
The Test: They slowed down the memory speed artificially to see what happened.
- Fast Memory (0.0001s): The AI checked every single alert against its memory. It made zero mistakes.
- Slow Memory (0.1s): The AI couldn't afford to check every alert because it would take too long. It skipped checks and repeated actions 7 times out of 12.
- The Lesson: The AI didn't get "dumber." It just couldn't afford to check its memory fast enough. When the memory was instant, the AI stopped making redundant mistakes.

The Hidden Bottleneck: The "Translator"

The researchers found one catch. While the memory itself was instant, the AI still needed to "translate" what it was thinking into a format the memory could understand (called "embedding").

If this translation happened over the internet, it was slow.
The Fix: By using a tiny, local translator (a small AI model running on the same computer), they made the entire process (thinking + translating + checking memory) take only 0.00004 seconds. This is fast enough to happen on every single step of the AI's reasoning.

Summary of Claims

Speed is the Gatekeeper: The reason AI agents don't check their memory constantly isn't because it's a bad idea; it's because current cloud databases are too slow.
In-Process is the Solution: By moving the memory to the same computer as the AI (in-process), the speed increases by 1,000 times.
It Changes Behavior: When memory is instant, the AI stops making "redundant" mistakes (doing the same thing twice) because it can check its history on every single step.
It's Not Magic: The AI didn't get smarter; the system around it got smarter. The AI still makes mistakes if the memory check is too slow, but with the fast memory, the system prevents those mistakes automatically.

In short: This paper proves that if you give an AI a "super-fast notebook" that lives inside its own brain, it can stop forgetting things and stop repeating mistakes, effectively turning external storage into a true part of its thinking process.

Technical Summary: Memory in the Loop

Problem Statement

Language agents operate on a standard loop: observe, reason, act, repeat. However, the memory they reason over is typically treated as an external resource queried at most once per turn (per-turn retrieval). This design is driven by the assumption that retrieval latency is prohibitive. Networked vector stores typically respond in 50–200 ms; if an agent consults such a store at every reasoning step, end-to-end latency can inflate by up to 83×. Consequently, current industry guidance ("memory-first") and system designs ration retrieval to once per turn or hide costs at the serving layer via scheduling.

The paper argues that this latency constraint is not a fundamental law of agent design but an artifact of where the store lives. Network latency (tens to hundreds of milliseconds) makes in-loop retrieval expensive, whereas an in-process store can answer in ~100 µs. The central problem is whether moving memory inside the reasoning loop—reading and writing it on every step—is feasible and beneficial when the latency barrier is removed.

Methodology

Core Architecture

The authors instantiate "Memory in the Loop" using an ephemeral, in-process semantic store (vxdb) wrapped as a WorkingMemory primitive. This store supports add, recall, and match operations. Crucially, the store is allocated per agent/task and is not connected to a network service, aiming for microsecond-level response times.

Experimental Setup

The study employs two primary experimental regimes:

Windowed Recall Task (End-to-End Performance):
- Scenario: A six-turn trip-planning conversation where the user states five durable constraints early in the conversation. The agent must list all five constraints at the end.
- Constraint: The context window is capped at $w=4$ messages, causing early constraints to scroll out of the visible context.
- Conditions:
  - Baseline/Aware: No memory tools (Aware is told its history is short).
  - Notes: Instructed to restate all constraints in every reply (a "no-tools" upper bound).
  - Memory: Equipped with remember/recall tools backed by the in-process store.
- Models: Tested across a "GPT-5 ladder" (gpt-5-nano, gpt-5-mini, gpt-5, gpt-5.5).
Causal Loop-Guard Task (Latency vs. Outcome):
- Scenario: An agent processes a stream of security alerts, deciding to INVESTIGATE or SKIP duplicates.
- Mechanism: A code guard checks the store before every action to prevent redundant investigations.
- Variable: Store latency is manipulated by injecting round-trip delays (0 µs to 200 ms) while keeping the per-turn memory budget fixed (100 ms).
- Arms:
  - In-process: Real-time speed (~17.5–85 µs).
  - Delayed Ladder: Identical logic with injected delays (+5, +15, +30, +50, +110 ms).
  - Per-turn RAG: Industry standard (retrieval once per turn).
- Metric: Redundant actions executed (out of 12 duplicates).

Cost Modeling

The authors define a feasibility frontier where in-loop memory is viable if the cost of retrieval ( $f \cdot c$ ) is negligible relative to the reasoning step time ( $r$ ). They measure the complete operation cost (embedding + store) to determine if per-step access is "free" relative to an LLM step.

Key Results

1. Feasibility and Latency Tax

In-Process Speed: The in-process store operates at a median latency of 80–165 µs.
Latency Tax: With 20 in-loop retrievals per turn, the total blocking time is ~1.7 ms for the in-process store. In contrast, a cloud vector store (110 ms RTT) incurs a tax of 2.2 seconds per turn.
Feasibility: At in-process speeds, an agent can afford ~953 retrievals per step (assuming a 10% latency budget), whereas networked stores afford less than 1 lookup per step, forcing rationing.

2. Task Performance (Windowed Recall)

Baseline Failure: All baseline and "aware" runs scored 0/5 constraints recalled. The context window limitation caused total failure regardless of model size.
Restatement Baseline: An instructed baseline that restated constraints in every reply achieved 5/5, demonstrating that the task is solvable if the agent actively manages context, though this incurs a token "rent" that scales with the working set.
Memory in the Loop: The in-loop memory condition achieved 3.6–4.8/5 recall across models.
- Failure Analysis: The store never lost a fact (244/244 writes preserved). Misses were exclusively read-policy failures: agents over-wrote constraints (creating duplicates) and then read only once per run through a fixed $k=8$ window, which was crowded by redundant entries. The failure was in the agent's tool usage, not the memory substrate.

3. Causal Impact of Latency (Loop Guard)

Monotonic Dose-Response: Redundant actions rose monotonically with store latency under a fixed budget.
- In-process (0 ms): 0.0 redundant actions.
- +15 ms: 1.4–1.6 redundant actions.
- +110 ms (Cloud): 7.2 of 12 redundant actions.
Statistical Significance: The difference between in-process and 110 ms latency was statistically significant ( $p=0.0079$ ) for both models tested.
Mechanism: At 110 ms, the budget (100 ms) could not afford a single lookup, so the guard was effectively disabled. The "mistakes" were not retrieval errors but the absence of checks.
Per-Turn RAG: The industry-standard per-turn retrieval performed well (0.4–0.6 redundant actions) but structurally failed to catch duplicates arising within a single turn, whereas the in-process guard caught all duplicates.

4. The Embedding Bottleneck

The dominant cost in the in-loop stack is not the store but the embedding. Network embedding APIs take 200–400 ms.
By pairing the in-process store with a small local embedder (32M parameters, 32 µs), the complete operation (embed + store) drops to **40 µs**. This removes the final barrier to in-loop memory.

Significance and Claims

Theoretical Contribution: Extended Working Memory

The paper grounds its engineering findings in the Extended-Mind Thesis (Clark & Chalmers, 1998). It argues that an external resource becomes "constitutive" of cognition (i.e., part of the agent's working memory) only if it is:

Constantly available.
Directly accessible without difficulty.
Automatically endorsed.

The authors interpret the first two criteria as a latency budget. A 100 ms network call is a tool an agent consults; a 100 µs in-process store is a resource the agent has. When latency is low enough to be transparent to the reasoning step, the store transitions from an external database to extended working memory.

Engineering Contribution

Latency as a Design Variable: The paper shifts retrieval frequency from a serving-layer constraint to an agent-level design axis. It demonstrates that store latency, not the in-loop pattern itself, is the gating factor.
Causal Link: It provides causal evidence that store latency alone determines task outcomes (redundant actions) when the budget is fixed, independent of model capability.
Architectural Shift: It proposes that memory should be treated as a loop primitive (like a scratchpad) allocated per task, rather than a connected service.

Modesty and Limitations

The authors are explicit about the scope:

They do not claim the store makes the model "smarter"; the models still failed to recognize duplicates without the guard. The system succeeded because the scaffold could afford to check the memory.
The "Notes" baseline (restatement) outperformed memory tools on the small 5-fact task, highlighting that restatement pays a per-turn cost that grows with the working set, which the store avoids at scale.
The study focuses on the temporal aspect of memory (when it participates in reasoning). Questions regarding what to retain, how to organize it, and lifecycle management (forgetting/prioritization) are identified as future work.
The results rely on specific task families (constraint recall, loop guard) and a specific window size, though sensitivity analysis confirms the "window cliff" phenomenon.

In conclusion, the paper asserts that the "in-loop" regime is not only feasible but causally superior for tasks requiring state consistency within a turn, provided the memory substrate is in-process and paired with a local embedder to eliminate network latency.