💬 NLP

CLAG: Adaptive Memory Organization via Agent-Driven Clustering for Small Language Model Agents

CLAG is an adaptive memory framework for small language model agents that enhances reasoning and knowledge reuse by actively organizing experiences into semantically coherent, self-contained clusters with generated profiles, thereby reducing cross-topic interference and improving retrieval efficiency.

Original authors: Taeyun Roh, Wonjune Jang, Junha Jung, Jaewoo Kang

Published 2026-03-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Taeyun Roh, Wonjune Jang, Junha Jung, Jaewoo Kang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant but slightly overwhelmed assistant (a "Small Language Model" or SLM) who helps you with complex tasks. This assistant has a massive notebook where they write down everything that happens during your day: your meetings, your hobbies, your work projects, and your random thoughts.

The Problem: The "One Big Pile" Mess
Currently, most AI assistants use a "Global Memory" system. Imagine your assistant's notebook is just one giant, unorganized pile of sticky notes.

When you ask, "What time is my meeting?", the assistant has to dig through every single note in the pile to find the answer.
Because the pile is so messy, they might accidentally grab a note about your "basketball game" or "dinner plans" that looks vaguely similar to "meeting."
This confuses the assistant, especially if they aren't a super-genius (which is the case for smaller, cheaper AI models). They get distracted by irrelevant information, leading to wrong answers or forgetting important details.

The Solution: CLAG (The Smart Librarian)
The paper introduces CLAG (Clustering-based AGentic memory). Think of CLAG not as a messy pile, but as a highly organized library managed by a smart librarian (the "Agent").

Here is how CLAG works, using simple analogies:

1. The Sorting Room (Agent-Driven Clustering)

Instead of throwing every new note into the same big pile, the librarian immediately sorts them into specific shelves based on the topic.

The Analogy: When you write a note about "Project X," the librarian doesn't just drop it on the floor. They look at it, realize it's about work, and place it on the "Work Shelf." If you write about "Weekend Hiking," it goes to the "Hobbies Shelf."
The Magic: The librarian (the AI agent) doesn't just use a simple keyword search; it understands the meaning. It creates a "Profile" for each shelf (e.g., "This shelf is for Work: Meetings, Deadlines, and Emails").

2. The Neighborhood Effect (Localized Evolution)

In the old "Global Pile" system, if you updated a note about "Project X," the assistant might accidentally change a note about "Hiking" because they were sitting next to each other in the chaos.

The CLAG Way: Updates only happen within the neighborhood. If you update a note on the "Work Shelf," the librarian only looks at other notes on the Work Shelf to see if they need updating.
The Benefit: Your "Hiking" notes stay perfectly safe and untouched. This prevents "cross-talk" where work stress accidentally messes up your vacation plans.

3. The Two-Stage Search (Smart Retrieval)

When you ask a question, the old system screams, "EVERYONE LOOK AT THIS!" and checks every single note in the building.

Stage 1 (The Gatekeeper): The librarian first looks at the Shelf Profiles. You ask, "When is the meeting?" The librarian checks the "Work Shelf" profile, sees it matches, and says, "Okay, I'll only check the Work Shelf." They completely ignore the Hiking, Sports, and Cooking shelves.
Stage 2 (The Deep Dive): Now, the librarian only looks at the notes inside the Work Shelf.
The Result: The search is faster, and the assistant is much less likely to get distracted by irrelevant "noise."

Why Does This Matter?

This is especially important for Small Language Models (SLMs).

Big Models are like geniuses who can read a messy pile of 10,000 notes and still find the right one.
Small Models are like smart students who get easily confused by a messy room. If you give them a messy pile, they panic and guess wrong.
CLAG gives the small model a clean, organized room. It doesn't need to be a genius to find the answer; it just needs to look in the right box.

The Real-World Result

The paper tested this on three different types of questions (general conversation, complex reasoning, and medical/bio questions).

Without CLAG: The small AI got confused by irrelevant info and gave wrong answers.
With CLAG: The small AI became as accurate as much larger, more expensive models. It answered questions faster and didn't get distracted by "distractors" (irrelevant information).

In a Nutshell:
CLAG turns a chaotic, overwhelming memory dump into a well-organized library with a smart librarian. It ensures that when the AI needs to remember something, it looks in the right place, ignores the noise, and doesn't accidentally mix up your work life with your personal life. This makes even small, affordable AI agents much smarter and more reliable.

1. Problem Statement

Large Language Model (LLM) agents rely on external memory to support long-horizon tasks, knowledge reuse, and complex reasoning. However, current memory systems typically store experiences in a single global retrieval pool. This architecture suffers from two critical issues, particularly for Small Language Models (SLMs) which are highly vulnerable to irrelevant context:

Search Space Expansion & Distractors: As the memory buffer grows, global retrieval becomes prone to retrieving semantically plausible but task-irrelevant memories (distractors), confusing the agent.
Cross-Topic Interference & Corruption: Memory evolution mechanisms (e.g., reflection, rewriting) operate across the entire unstructured pool. This exposes updates to mixed-topic neighborhoods, leading to "noisy" updates that degrade the quality of stored knowledge over time.

Existing agentic memory frameworks attempt to solve this with active management but often retain the global pool structure, failing to isolate topics effectively.

2. Methodology: The CLAG Framework

The authors propose CLAG (CLustering-based AGentic memory), a framework that imposes lightweight structure on long-horizon memory by treating clustering as an agent-controlled, online operation rather than a static offline step. The system uses a single backbone SLM to perform three core functions via role-specific prompts:

A. Agentic Routing (Memory Structuring)

Instead of a flat database, memories are organized into semantically coherent clusters.

Process: When a new memory arrives, an SLM-router analyzes its semantic profile. It first identifies top- $k$ candidate clusters via vector distance, then uses the SLM to select the most appropriate cluster based on cluster profiles (summaries and tags).
Adaptivity: If a memory is too dissimilar to existing clusters (below a similarity threshold $\tau$ ), a new cluster is created. If a cluster grows too large (exceeds $\tau_{split}$ ), it is dynamically bisected using K-Means to prevent over-saturation.
Output: Each cluster maintains a profile (topic summary, descriptive tags) acting as a self-contained functional unit.

B. Localized Evolution

Memory updates are confined strictly within the assigned cluster.

Mechanism: When a new memory is routed to a cluster, the system identifies top- $k$ semantically aligned neighbors within that cluster.
Action: The SLM generates explicit links (causality, temporal sequence) between the new memory and neighbors. It then refines existing memories in the cluster to incorporate new information.
Benefit: This mimics human cognitive schema refinement, ensuring new experiences update relevant concepts without perturbing unrelated knowledge domains, thereby reducing cross-topic interference.

C. Cluster-Aware Two-Stage Retrieval

At inference time, CLAG replaces flat global retrieval with a hierarchical pipeline to reduce noise.

Stage 1 (Cluster Selection): Given a query, the agent filters the global pool by selecting a small subset of relevant clusters using their profiles. This acts as a high-level semantic filter, drastically reducing the search space and excluding distractors.
Stage 2 (Intra-Cluster Retrieval): Fine-grained retrieval is performed only within the selected clusters.
Benefit: This prevents the agent from retrieving semantically similar but task-irrelevant memories that often appear in global similarity searches.

3. Key Contributions

Agent-Driven Clustering: A novel mechanism where an SLM agent actively routes memories into semantic groups online, ensuring robust long-horizon organization without pre-processing.
Localized Memory Evolution: A strategy that consolidates and updates memories strictly within topic-consistent neighborhoods, stabilizing long-term memory quality and mitigating interference.
Two-Stage Retrieval Scheme: A hierarchical retrieval process that filters irrelevant clusters before fine-grained search, significantly improving robustness for capacity-limited models by suppressing distractors.

4. Experimental Results

The authors evaluated CLAG on three benchmarks (LoCoMo, HotpotQA, BioASQ) using three different SLM backbones (Qwen3-0.6B, Llama-3.2-1B, DeepSeek-R1-Distill-1.5B).

Performance Gains: CLAG consistently outperformed baselines (Standard RAG, A-mem, MemoryOS, GAM) across all datasets and models.
- On BioASQ (biomedical domain), CLAG achieved an Evidence F1 of 25.11, significantly outperforming RAG (2.29) and A-mem (2.20), demonstrating superior handling of domain-specific jargon and noise.
- On LoCoMo (long-context dialogue), CLAG showed consistent improvements in F1 and BLEU-1 scores, particularly in Multi-Hop and Temporal reasoning tasks.
Efficiency: Despite the added complexity of routing, CLAG is highly efficient. On the Qwen3-0.6B backbone, it achieved orders-of-magnitude lower end-to-end latency compared to complex iterative frameworks like GAM, while maintaining higher accuracy.
Ablation Studies:
- Replacing Agentic Clustering with geometric clustering (K-Means or Cosine-based) resulted in significantly lower performance, proving that semantic understanding by the agent is crucial for organizing specialized domains.
- Removing Localized Evolution or Two-Stage Retrieval individually caused measurable drops in F1 scores, confirming both components are essential.
Case Study: In a qualitative analysis, CLAG was the only method to correctly retrieve specific details from a 680-note mixed-topic memory pool, successfully pruning irrelevant clusters (e.g., sports, social interactions) to focus on the literature cluster.

5. Significance and Conclusion

CLAG addresses a critical bottleneck in deploying Small Language Models (SLMs) as autonomous agents. SLMs are particularly susceptible to "context pollution" where irrelevant information degrades reasoning. By shifting from a static global pool to dynamic semantic neighborhoods, CLAG:

Enhances Robustness: It allows SLMs to operate effectively in noisy, long-context environments by filtering out distractors before the generation step.
Improves Scalability: The localized evolution and two-stage retrieval reduce computational overhead compared to global pairwise comparisons or iterative global searches.
Practicality: It offers a lightweight, high-performance memory layer that does not require large model backbones, making it suitable for on-device or cost-sensitive agentic applications.

The paper concludes that structuring memory via agent-driven clustering is a fundamental step toward enabling reliable, long-term reasoning in resource-constrained AI agents.