💻 computer science

When Your Agent Opens the Chat App: Agent-Controlled Search over Raw Chat Logs Rivals Structured Memory

This paper introduces ReFind, an agent-controlled search system that achieves state-of-the-art accuracy on conversational memory tasks by performing iterative lexical retrieval over unmodified raw chat logs, demonstrating that sophisticated semantic memory structures are often unnecessary for precise evidence-grounded questioning.

Original authors: Ruizhe Li, Licheng Zhang, Benfeng Xu, Mingxuan Du, Zheren Fu, Weidong Chen

Published 2026-08-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Ruizhe Li, Licheng Zhang, Benfeng Xu, Mingxuan Du, Zheren Fu, Weidong Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find a specific conversation in a massive, endless chat history. You remember the other person said something about "pizza" and "Tuesday," but you don't know exactly when or who said it. In the world of Artificial Intelligence, this is the "memory problem." When AI agents (smart computer programs that can chat, write code, or solve problems) talk to us for a long time, they get overwhelmed. They can't remember everything in their "brain" at once.

To fix this, most AI systems today try to be like a super-organized librarian. Before you even ask a question, they take all those messy chat logs, read them, and rewrite them into neat summaries, draw complex maps of connections, or build fancy 3D structures of facts. They do this hoping it will make finding answers easier later. But this approach has a catch: to build these fancy structures, the AI has to guess what is important before it knows what you are going to ask. If it throws away a tiny detail while summarizing, that detail might be gone forever.

This brings us to a big question: Do we really need these fancy, pre-built libraries? Or is it possible that the AI just needs a really good way to search through the original, messy chat logs directly? This is the puzzle a new study tackles. It asks if an AI can be just as smart at finding answers by acting like a curious human scrolling through a chat app, rather than relying on a pre-processed memory bank.

The Paper: "ReFind" – The AI That Just Opens the Chat App

The researchers behind this study, led by Ruizhe Li and colleagues, decided to test a radical idea: What if the AI doesn't build any special memory structure at all?

Instead of rewriting the chat history into summaries or graphs, their system, called ReFind, leaves the chat logs exactly as they are. It treats the conversation history like a raw, unedited diary. When the AI needs to answer a question, it doesn't consult a pre-made index. Instead, it "opens the chat app" and starts searching, just like a person would.

How It Works: The Detective's Toolkit

The paper suggests that humans are actually pretty good at finding old messages. We don't usually type one perfect sentence into a search bar and get the answer. Instead, we use a mix of tricks:

We search with keywords: "Pizza."
We look at the neighbors: If we find a message about pizza, we read the few messages before and after it to get the full story.
We remember the time: "That was last month."
We skip what we've seen: "I already checked that conversation, let's look somewhere else."

ReFind copies these human tricks. It uses a simple, fast search tool (called BM25, which just looks for matching words) but wraps it in a smart "loop" that lets the AI agent control the search.

The Loop: The AI asks a question, looks at the results, and if it's not sure, it changes the search terms and tries again.
The Controls: It uses four special "chat-native" tools:
- Context Expansion: If it finds a hit, it grabs the surrounding messages so the AI understands the joke or the context.
- Session Awareness: It knows that if one message in a conversation is relevant, the whole conversation is probably important, so it boosts the ranking of that whole chat session.
- Time Travel: It can filter results to only look at messages from a specific date range.
- Skip the Seen: It remembers which conversations it has already checked so it doesn't waste time reading the same chat twice.

The Results: Raw Power vs. Fancy Structure

The researchers put ReFind to the test against the "fancy" systems that build complex memory structures (like graphs and trees). They used a massive set of about 2,800 questions covering things like "Who said what?" (single-hop), "How did event A lead to event B?" (multi-hop), and "What is the latest version of this fact?" (fact tracking).

Here is what they found:

ReFind won. It achieved the highest average accuracy of 58.2% across all the tests.
The strongest "fancy" system, called HippoRAG 2 (which uses complex graphs), came in second with 53.2%.
Even a simple, one-time search without the smart loop (called BM25-RAG) only got 48.8%.

This suggests that the "fancy" structures aren't actually doing the heavy lifting. The real secret sauce was giving the AI control over the search process. The AI didn't need a pre-built map; it just needed the ability to drive the car, look at the road, and decide when to turn left or right.

The "Backbone" Test

To make sure this wasn't just a fluke with a specific AI model, the researchers tried it again with a more powerful AI (GPT-5-mini) on even longer conversations (some with over 500,000 words).

ReFind scored 93.2% on shorter long-conversations and 89.3% on the massive ones.
It beat every other system, including the ones with complex graphs and trees, by a significant margin.

What the Study Rules Out

The paper is very careful to say what didn't work. They ran several "ablation" tests (taking parts of the system away to see what happens):

It's not just the loop: If you let the AI search multiple times but remove the "chat-native" controls (like skipping seen sessions or expanding context), the score drops significantly. The AI needs the specific tools designed for chat, not just a generic search loop.
It's not the "fancy" math: They tried using complex "dense" search methods (which try to understand the meaning of words rather than just matching them) and found that simple word-matching (BM25) was actually better or equal. The AI's ability to reformulate its own questions was more important than the type of search engine.
It's not the model size: The system worked well even with a smaller, cheaper AI model (GPT-4o-mini), proving that the method of searching is what matters, not just having a super-powerful brain.

The Big Takeaway

The authors conclude that for finding precise facts in chat logs, structure is overrated. We don't need to spend time and computing power turning raw conversations into complex graphs before we ask a question. Instead, we should keep the raw data safe and unmodified, and let the AI agent act as a smart, curious detective who can search, expand, and filter the history in real-time.

In short, the paper suggests that the best memory for an AI isn't a pre-written encyclopedia; it's a raw, unedited diary and a smart agent who knows how to read it.

Technical Summary: ReFind – Agent-Controlled Search over Raw Chat Logs

Problem Statement

Current agent-memory systems predominantly rely on pre-built semantic structures (e.g., knowledge graphs, hierarchical trees, recursive summaries, or vector embeddings) to manage long-term interaction histories. These systems transform raw conversation data into compact representations before a question is asked. The authors argue this approach carries significant risks:

Information Loss: Preprocessing decisions (what to abstract or discard) are made without knowledge of the future query, potentially omitting recoverable details.
Maintenance Costs: Indices must be built offline and incrementally updated as new messages arrive.
Unverified Benefit: It remains unclear how much performance gain comes from the structure itself versus the mere ability to retrieve competently from the history.

The paper posits that for precise, evidence-grounded questions, the benefit of elaborate structures might be recoverable by allowing an agent to perform controlled search over unmodified raw records, deferring structural decisions until query time.

Methodology: ReFind

ReFind is an agent-controlled search interface that builds no semantic structure (no LLM-generated summaries, graphs, or embeddings). Instead, it operates on the raw chat archive using a two-stage process:

1. Data Representation & Indexing

Raw Preservation: The system uses the existing organization of chat archives: turn boundaries, session identifiers, timestamps, and raw text.
Lexical Indexing: It employs a BM25 inverted index at the "turn" granularity (a user utterance paired with an assistant response).
Incremental Updates: New turns are appended to the index without summarization or entity extraction, making the archive searchable immediately upon creation.

2. System Architecture

The system decouples Retrieval (evidence collection) from Reasoning (answer generation).

Stage 1: Iterative Retrieval (ReAct Loop)
A ReAct-style agent autonomously decides search keywords and parameters over multiple rounds. The core innovation lies in four chat-native controls grounded in empirical refinding studies:
- Local Context Expansion: When a turn matches, the system returns the matched turn plus $\pm 2$ neighboring turns to restore context (pronouns, ellipses) often lost in isolated lexical matches.
- Session-Aware Rank Fusion (RRF): Uses Reciprocal Rank Fusion to combine turn-level BM25 scores with session-level aggregate scores. If multiple turns in a session match, the entire session is boosted, acknowledging that a session often represents a coherent topic.
- Temporal Narrowing: The agent can specify time ranges (e.g., "last month") to filter candidates by timestamp before scoring.
- Seen-Session Filtering (Deduplication): The system tracks sessions already inspected in previous rounds and excludes them from future searches to maximize information gain per iteration.
Stage 2: Reasoning
Collected evidence (notes) is grouped by session, sorted chronologically, and presented to the LLM for answer generation. This ensures the answer model receives a compact, evidence-only view without competing for context space with the search loop.

Key Contributions

ReFind Interface: A novel system combining iterative lexical search with session-aware reranking, context expansion, temporal filtering, and deduplication, driven by an agent that collects evidence before generating an answer.
Empirical Evidence: Demonstrates that for precise retrieval and factual-update tracking, the performance benefits attributed to complex memory structures can be recovered without any LLM-based index construction, provided the agent controls the search.
Scope Delineation: Establishes that this approach is highly effective for precise retrieval and tracking factual updates, serving as a complementary alternative to semantic, low-latency memory mechanisms.

Experimental Results

The system was evaluated on ~2,800 questions across six benchmarks (MemoryAgentBench and LongMemEval) covering single/multi-hop QA, event ordering, and fact consolidation.

Performance with GPT-4o-mini Backbone

ReFind achieved the highest mean accuracy of 58.2% across all benchmarks.
It outperformed the strongest structured baselines, including HippoRAG 2 (53.2%), GraphRAG, and RAPTOR.
It also surpassed single-shot BM25-RAG (48.8%), demonstrating the value of the iterative, controlled loop.
Notable gains were observed in multi-hop QA (+13 points over BM25-RAG) and factual consolidation, where iterative reformulation and session-aware controls are critical.

Performance with GPT-5-mini Backbone (LongMemEval)

On LongMemEval-S (50 questions, ~115k tokens) and LongMemEval-M (15 questions, ~500k tokens), ReFind achieved 93.2 ± 3.3% and 89.3 ± 6.0% respectively.
This significantly outperformed STITCH (86.0/80.0) and GAM (70.0/60.0), the latter being a similar agent-based system lacking the specific chat-native controls.

Ablation Studies

Controlled comparisons confirmed the necessity of specific components:

Chat-Native Controls: Removing local expansion, session RRF, temporal filtering, or deduplication (leaving only a generic agentic BM25 loop) caused a drop of 7.1 to 14.5 points, proving that iteration alone is insufficient.
Multi-Round Loop: A "one-search" variant (no reformulation) dropped 20.4 points on the M subset, highlighting the need for evidence-conditioned reformulation.
Backend: Sparse BM25 outperformed dense/hybrid embeddings, suggesting that lexical matching is not the bottleneck and that the agent's semantic adaptation via reformulation is the key driver.

Significance and Claims

The paper claims that preserving raw records and exposing their conversational structure to an adaptive agent is a superior strategy for precise, evidence-grounded tasks compared to pre-transforming data into memory representations.

Question-Directed Computation: ReFind shifts computation from offline index construction (which happens regardless of the query) to online, question-directed search. This makes the system immediately updatable and auditable.
Design Principle: The results suggest a modular design principle for conversational memory: begin with faithful storage and a controllable search interface, adding derived structures only for workloads requiring specific latency or abstraction layers.
Recoverability: Much of the benefit credited to elaborate memory structures is recoverable through competent, controlled search over the unmodified record, challenging the necessity of pre-built semantic indices for many agent memory tasks.