💬 NLP

AgentIR: A Workload-Adaptive Cascade Retrieval Substrate for Long-Term Conversational Memory

AgentIR introduces a workload-adaptive cascade retrieval substrate for long-term conversational memory that dynamically skips expensive dense retrieval based on confidence thresholds and utilizes a time-partitioned index to achieve sub-10ms latency and up to 132x speedup while maintaining or improving retrieval accuracy across diverse benchmarks.

Original authors: Aojie Yuan, Haiyue Zhang, Shahin Nazarian

Published 2026-05-26

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Aojie Yuan, Haiyue Zhang, Shahin Nazarian

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: A Smart Librarian for AI Agents

Imagine you have a very smart AI assistant (an "agent") that helps you for months or even years. Over time, this agent accumulates a massive diary of conversations, tool uses, and plans—millions of pages of notes.

Every time the agent takes a step to solve a problem, it has to look through this diary to find the right information. If the agent takes 20 steps, it needs to search this diary 20 times in a row. If the search takes even a fraction of a second too long, the whole conversation feels "stuck" or slow to the human user.

The problem? The tools we usually use to search libraries (like standard search engines) were built for searching the entire internet, not for searching a single, constantly growing diary where the most important notes are usually the ones written just now.

AgentIR is a new, specialized search engine built specifically for these AI agents. It's faster, smarter about what to search, and scales up without slowing down.

1. The "Time-Travel" Shelves (Temporal Partitioning)

The Problem: Imagine a library where books are added every second. If you ask for a book, a standard librarian might have to check the shelves from the oldest book to the newest, one by one. As the library grows to millions of books, this search gets slower and slower.

The AgentIR Solution: AgentIR organizes the library differently. Instead of one giant row of books, it puts them on shelves based on when they were written.

The Analogy: Think of it like a "Time-Travel" library. The most recent books are on a special, easy-to-reach shelf right at the front. Older books are pushed further back.
How it works: Because AI agents usually need information from recent conversations (like "what did we just fix?"), the system only looks at the front shelves first. If it finds the answer there, it stops immediately. It doesn't waste time checking the dusty back shelves.
The Result: Even if the library grows 1,000 times bigger, the search time barely increases. It's like finding a needle in a haystack, but the needle is always in the top 1% of the hay.

2. The "Two-Tool" Strategy (Hybrid Search)

The Problem: Sometimes you need to search by exact words (like finding a specific error code), and sometimes you need to search by meaning (like finding a conversation about "fixing a bug" even if the word "bug" isn't used).

Tool A (BM25): Great at exact word matching, super fast.
Tool B (Dense/Vector): Great at understanding meaning, but slow and heavy.

The AgentIR Solution: AgentIR doesn't force you to use both tools for every single question. It acts like a smart traffic cop.

The Analogy: Imagine you are looking for a specific item in a store.
- If you ask, "Where is the red hammer?", the system knows you just need to look at the "Hammer" aisle (Tool A). It skips the expensive, slow "Meaning" scanner.
- If you ask, "Where is the thing that fixes the squeaky door?", the system knows it needs the "Meaning" scanner (Tool B) to understand the concept.
The Cascade: The system tries the fast tool first. If the fast tool is very confident it found the answer, it stops there. If it's unsure, then it calls in the slow, heavy tool. This saves a massive amount of time.

3. The "Shape-Shifting" Strategy (Workload-Adaptive)

The Problem: Different types of questions need different search strategies.

On one dataset (LongMemEval), mixing word-matching and meaning-matching worked best.
On another dataset (LoCoMo), just word-matching was actually better and faster.
Old systems force you to pick one strategy and stick with it forever.

The AgentIR Solution: AgentIR is a "chameleon." It has a tiny, fast classifier (a decision-maker) that looks at your question and says, "Ah, this is a 'Time' question, let's use Strategy A," or "This is a 'Facts' question, let's use Strategy B."

The Result: It automatically tunes itself. On one test, it skipped the slow tool 63% of the time. On another test, it skipped it 100% of the time because the fast tool was perfect. It adapts to the job at hand without needing to be retrained.

4. The "Super-Speed" Engine (GPU & CPU Optimization)

The Problem: Doing these searches on standard computer chips (CPUs) is fast, but doing them on powerful graphics cards (GPUs) is usually tricky because the math doesn't line up perfectly. Also, standard search engines often have hidden bugs that make them slightly less accurate when you try to speed them up.

The AgentIR Solution:

The Engine: They built a custom engine that runs on both CPUs and GPUs perfectly.
The "Bug Fix": The authors found three subtle "bugs" that often happen when people try to speed up search engines (like normalizing numbers wrong or forgetting to clear memory). These bugs make search results 6 to 8 times worse without anyone noticing. AgentIR fixed these, ensuring that the super-fast GPU version gives the exact same correct answers as the slower CPU version.
The Result: They achieved speeds up to 132 times faster than standard systems on certain tasks, while keeping the accuracy exactly the same.

Summary of Results

Speed: It can handle millions of records and still answer in under 100 microseconds (that's 1/10,000th of a second) for recent data.
Efficiency: It can serve 8 different AI agents at the same time on a single computer without slowing down.
Accuracy: It matches or beats the best existing search engines (like Lucene) in finding the right answers, but does it much faster.
Cost: Because it's so fast and efficient, it costs a tiny fraction of what commercial cloud search services charge.

In a nutshell: AgentIR is a specialized, high-speed search engine that knows when to look at recent notes, when to use simple word matching, and when to skip the heavy lifting entirely, ensuring AI agents stay fast and responsive even as their memories grow huge.

Technical Summary: AgentIR

Problem Statement

Long-term conversational memory for LLM agents presents a retrieval workload fundamentally different from classical Information Retrieval (IR) systems like Lucene, FAISS, or PISA. While traditional engines assume static corpora, stationary query distributions, and latency budgets measured in tens of milliseconds, agent memory exhibits five distinct properties that break these assumptions:

Extreme Recency Bias: Retrieval queries concentrate heavily on the most recent fraction of records (e.g., the top 20–27% of sessions), unlike the slow drift of web search.
Non-Static Index Growth: The corpus grows monotonically during the query stream (at conversation rate), making traditional index rebuilds or incremental updates (which incur high tail latency) impractical.
Heterogeneous Query Types: A single agent session contains factual lookups, multi-step reasoning, and temporal queries, where no single fusion strategy (BM25, Dense, or RRF) dominates all types.
Strict Latency Budgets: With 5–20 retrievals per agent step, the per-query budget is sub-10 ms. Dense encoders (e.g., SPLADE++) often exceed this budget due to encoding overhead.
Role-Structured Records: Data includes metadata (role, session ID, tool type) requiring specific filtering and fusion logic beyond standard text matching.

Existing systems fail to exploit these properties, often treating agent memory as "small IR" and suffering from linear scaling costs and suboptimal fusion strategies.

Methodology: AgentIR Substrate

AgentIR is a workload-adaptive cascade retrieval substrate designed to address these structural properties through a two-axis adaptive control surface: (1) selecting the optimal fusion strategy per query, and (2) deciding whether to execute the expensive dense channel at all.

1. Architecture and Pipeline

The system runs three concurrent channels over a single CSR (Compressed Sparse Row) substrate:

SIMD-Accelerated BM25: A CPU-based sparse search using AVX2/NEON vectorization and MaxScore pruning.
Dense Channel: An HNSW-based ANN search for dense embeddings (e.g., BGE-small).
Temporal Partitioned Index: A time-windowed index structure that partitions documents into 7-day buckets.

Two-Axis Adaptation:

Fusion Selection: The system chooses between BM25, Dense, RRF, or an "agent-aware" RRF (which adds a recency bonus and importance weight) based on the query type.
Cascade Routing: A confidence-triggered router decides whether to skip the dense channel entirely. If the BM25 top- $k$ margin is high (indicating high confidence), the system exits early; otherwise, it escalates to the full hybrid pipeline.

2. Key Optimizations

Sub-linear Temporal Scaling: Leveraging the exponential recency bias of agent workloads, the temporal index searches only the most recent partitions. Theorem 4.1 proves that under this bias, expected work is $O(\log(1/\varepsilon))$ , independent of total corpus size $N$ .
Heterogeneous CPU/GPU Pipeline:
- CPU: Optimized for memory-bound HNSW and compute-bound BM25 via SIMD and multi-threaded parallelism.
- GPU: A two-kernel CUDA pipeline for batched BM25 scoring (CSR layout) and top- $k$ selection. The system identifies and fixes three specific correctness pitfalls in re-implementing BM25 on modern hardware: pre-normalized term frequency, linear-gain nDCG, and stale shared-memory reads in GPU top- $k$ kernels.
Workload-Conditional Fusion: A runtime router (using TF-IDF or lightweight classifiers) predicts the query type and selects the best fusion strategy. A "soft router" blends rank lists based on classifier posteriors to handle ambiguous queries.

Key Contributions

Sub-linear Regime Implementation: Demonstrates that a time-partitioned index can achieve $O(\log(1/\varepsilon))$ work for recency-skewed workloads. Empirically, growing the corpus 1,234× (4K to 5M records) increases latency by only 3.6×, achieving a 1,769× speedup over sequential scan at 5M records with sub-100 $\mu$ s $p50$ latency.
High-Performance Hybrid Engine: Delivers Lucene-quality retrieval (within $\pm 0.020$ nDCG@10) across 9 BEIR datasets (up to 8.8M docs) while running 10× faster than Pyserini (8-thread) and 11× faster than PISA-1T (BlockMax-WAND) on average. It also validates bit-perfect correctness between CPU and GPU implementations.
Cross-Workload Evidence for Conditional Fusion: Shows empirically that the optimal fusion strategy varies by workload and query type.
- On LongMemEval (500 questions), a hybrid RRF + recency strategy outperforms static baselines. A learned router achieves the "discrete oracle" bound (0.300 LLM-judged accuracy) under gpt-4o.
- On LoCoMo (1,982 questions), BM25 alone is the strongest system.
- A Cascade Router auto-tunes across these benchmarks: it skips the dense channel for 63% of LongMemEval queries (2.67× speedup) and 100% of LoCoMo queries (132× speedup) while maintaining parity in accuracy.

Results

Quality: Matches or exceeds Pyserini Lucene on 5 of 9 BEIR datasets. Hybrid RRF with BGE-small improves mean nDCG@10 by +0.033 over BM25-only. On LongMemEval, agent-aware fusion achieves R@10=0.978 and LLM-judged accuracy of 0.254 (vs. 0.246 for BM25).
Latency & Throughput:
- Temporal Scaling: At 5M records, temporal partitioning serves queries in <100 $\mu$ s ( $p50$ ), whereas sequential scan takes ~159 ms.
- Multi-tenant: An 8-core VM serves 8 concurrent agents with invariant $p50$ latency (~0.38 ms) and 5.5× aggregate throughput.
- Cascade Efficiency: The cascade router reduces amortized latency from 53.2 ms to 19.9 ms (2.67×) on LongMemEval and to 0.4 ms (132×) on LoCoMo.
Correctness: The paper documents and fixes three subtle bugs (pre-normalized TF, linear-gain nDCG, GPU stale memory) that previously caused 6–8× regressions in nDCG@10. Post-fix, CPU and GPU agree within 0.0002 nDCG@10.

Significance and Claims

The paper argues that agent memory is not merely "small IR" but a distinct workload regime requiring architectural specialization. The significance of AgentIR lies in:

Exploiting Structural Properties: By treating recency skew, monotonic growth, and query heterogeneity as design inputs, the system achieves orders-of-magnitude lower per-query work than generic engines.
Workload-Conditional Adaptation: It demonstrates that a single deployment can auto-tune to vastly different operating points (e.g., hybrid fusion vs. BM25-only) based on the specific workload, using a lightweight router and cascade mechanism.
Correctness and Reproducibility: The paper provides a verified, bit-perfect implementation of BM25 on modern CPU/GPU hardware, correcting silent regressions common in re-implementations.
Cost Efficiency: The retrieval substrate cost is estimated to be 2–3 orders of magnitude lower than the LLM tier, making it a critical component for scalable, cost-effective agent systems.

The authors conclude that the combination of temporal partitioning, workload-conditional fusion, and cascade routing allows a single system to serve long-term agent memory at sub-millisecond latencies on commodity hardware, a regime previously unattainable with static IR engines.