💬 NLP

TeleMem: Building Long-Term and Multimodal Memory for Agentic AI

TeleMem is a unified long-term and multimodal memory system for agentic AI that overcomes the limitations of existing retrieval-augmented generation methods through narrative dynamic extraction, a structured writing pipeline, and ReAct-style reasoning, achieving superior accuracy, efficiency, and speed on long-term role-play benchmarks.

Original authors: Chunliang Chen, Ming Guan, Xiao Lin, Jiaxu Li, Luxi Lin, Qiyi Wang, Xiangyu Chen, Jixiang Luo, Changzhi Sun, Dell Zhang, Xuelong Li

Published 2026-01-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chunliang Chen, Ming Guan, Xiao Lin, Jiaxu Li, Luxi Lin, Qiyi Wang, Xiangyu Chen, Jixiang Luo, Changzhi Sun, Dell Zhang, Xuelong Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to have a deep, years-long conversation with a very smart friend (an AI). The problem is, this friend has a very short attention span. If you talk for too long, they start forgetting what you said at the beginning, or they get confused about the order of events. They might even make up facts that never happened because they are trying to guess what you meant.

The paper introduces TELEMEM, a new system designed to give this AI friend a "super memory" that works like a human's long-term memory, but organized perfectly so they never get lost.

Here is how it works, using simple analogies:

1. The Problem: The "Scattered Note" vs. The "Story Thread"

Most current AI memory systems work like a pile of sticky notes.

You write a note about your dog.
Later, you write a note about your dog's name.
Later, you write a note about your dog's favorite toy.
When the AI needs to answer a question, it grabs a few random sticky notes that look similar.
The flaw: The notes are disconnected. The AI might know the toy but forget the name, or it might mix up the order of events. It's like trying to read a book where the pages are shuffled.

TELEMEM changes this. Instead of a pile of notes, it builds a giant, organized family tree of stories.

It connects every new piece of information to the specific story it belongs to.
It knows that "Dog's Name" depends on "Meeting the Dog," which depends on "The Day You Moved."
This creates a thread. When the AI needs to remember something, it doesn't just grab a note; it pulls the entire thread, ensuring the story makes sense from start to finish.

2. Writing Memories: The "Editor's Desk"

When you talk to the AI, it doesn't just save every single word you say (that would be too messy and expensive).

Old way: Save every sentence.
TELEMEM way: Imagine a busy editor. When you finish a conversation, the editor:
1. Summarizes the key points.
2. Checks if this is new or if it updates something they already know.
3. Groups similar topics together (like putting all "vacation" stories in one folder).
4. Consolidates them into one clean, clear entry.
This happens in batches, so the AI doesn't get overwhelmed. It saves the "essence" of the conversation, not the noise.

3. Reading Memories: The "Detective's Map"

When you ask the AI a question later, it doesn't just search for keywords.

Old way: "Find me anything about 'vacation'." (Result: A random mix of sentences).
TELEMEM way: It acts like a detective following a map.
- It finds the starting point (the "seed" of the question).
- It then traces the dependencies backward: "To understand this vacation, I need to know who went, when they left, and what happened before."
- It gathers the whole chain of events (the "closure") so the AI has the full context to give a smart answer, rather than a fragmented one.

4. Seeing and Doing: The "Video Detective"

Most AI memory systems only understand text. TELEMEM is special because it can also remember videos.

If you show the AI a video of a complex event, it doesn't just store the video file.
It uses a "Think-Act-Observe" loop (like a detective solving a case):
1. Think: "I need to know what happened at the 5-minute mark."
2. Act: It goes back to the video to check that specific moment.
3. Observe: It sees the details and adds them to its memory.
This allows the AI to reason about complex visual events, not just read a transcript of them.

The Results: Why It Matters

The authors tested this system against other top memory systems (like Mem0) and found:

Smarter Answers: It got 19% more questions right on a difficult test about long conversations.
Cheaper & Faster: It used 43% less computing power (tokens) and was 2.1 times faster.
No Hallucinations: Because it keeps the "story threads" organized, it is much less likely to make up facts or get confused about the past.

In short: TELEMEM turns the AI's memory from a messy box of loose papers into a well-organized, interconnected library where every story is linked to the next, allowing the AI to remember long conversations clearly, efficiently, and accurately.

Technical Summary: TELEMEM

1. Problem Statement

Large Language Models (LLMs) face significant limitations in long-term interactive settings due to finite context windows and the degradation of recall over extended histories. While Retrieval-Augmented Generation (RAG) alleviates storage capacity issues, conventional pipelines treat memories as independent, unordered fragments. This approach lacks principled mechanisms for consolidation, updating, and causal organization, leading to:

Fragmented Context: Retrieved information often omits critical prerequisites or temporal order.
Unstable Reasoning: The absence of causal structure induces inconsistent long-horizon reasoning.
Inefficiency: Frequent, unbatched LLM invocations for writing and retrieval result in high token usage and latency.
Multimodal Gaps: Existing systems are predominantly text-centric, struggling to integrate visual, auditory, and temporal information for embodied or multimedia agents.
Hallucination Risks: Reliance on large pre-defined schemas often leads to hallucinated attributes when real conversations provide sparse signals.

2. Methodology: The TELEMEM Framework

TELEMEM is a unified long-term and multimodal memory system that organizes memory as structured, evolvable semantic trajectories rather than isolated fragments. Its architecture consists of three tightly coupled components:

A. Memory Graph Formulation (The Structure)

Memory is formalized as a Directed Acyclic Graph (DAG), $G = (V, E)$ , where:

Nodes ( $V$ ): Represent stabilized semantic memory states (profile states, event states, or entity/object states). Each node contains consolidated semantic content, an embedding vector, and an effective timestamp reflecting semantic validity order.
Edges ( $E$ ): Define threaded causal dependencies. A directed edge $p \to v_i$ indicates that node $v_i$ is semantically conditioned on prior node $p$ .
Minimal Causal Skeleton: The graph maintains only irreducible dependencies. Redundant or transitive edges are pruned to ensure compact, interpretable root-to-node threads that capture the evolution of topics and states.

B. Memory Writing Pipeline (The Update Mechanism)

TELEMEM decouples representation updates from index updates, employing a structured writing pipeline to amortize costs and ensure consistency:

Summarization: Dialogue turns or multimodal observations are summarized into textual units.
Retrieval Alignment: New summaries are matched against existing memory to identify related historical nodes.
Global Clustering: New and retrieved candidates are merged and clustered into semantic groups.
LLM-based Consolidation: An LLM processes each cluster to resolve redundancy and inconsistencies, assigning actions (add, delete, update, no-op) to produce refined, stabilized nodes.
Graph Maintenance:
- Offline: Nodes are processed in timestamp order to construct the graph index using an Insert operator that respects temporal constraints and pruning criteria.
- Online: Real-time updates use lightweight Insert and ReInsert operators. ReInsert is triggered when a node's content changes, recomputing its incoming edges and refreshing its children to maintain threaded consistency without modifying the global DAG property.

C. Memory Reading & Reasoning (The Retrieval Mechanism)

Instead of "Top-K fragment stitching," TELEMEM employs Closure-Based Retrieval:

Seed Identification: Relevant seed nodes are identified via embedding similarity.
Closure Expansion: The system recursively traverses dependency edges backward from seeds to collect all reachable ancestor nodes (prerequisites) up to the virtual root.
Context Linearization: The resulting closed subgraph is sorted chronologically and serialized into a coherent context for downstream reasoning.

Multimodal Extension:
For complex video content, TELEMEM integrates a ReAct-style reasoning framework. An agent iterates through think–act–observe cycles, utilizing tools for video retrieval, video-RAG (grounded summarization), and video-QA (visual detail extraction) to perform closed-loop reasoning over raw video streams.

3. Key Contributions

Unified Framework: Introduction of TELEMEM, a system that maintains coherent, hallucination-resistant user profiles through narrative-driven extraction and structured organization.
Structured Memory Graph: A proposal for a memory graph with threaded dependencies and closure-based retrieval, enabling dependency-aware context reconstruction and stable long-horizon reasoning.
Efficient Writing Pipeline: A design performing batched retrieval, semantic clustering, and LLM-based consolidation, which reduces storage redundancy, token overhead, and end-to-end latency.
Multimodal Reasoning: Development of a multimodal module with ReAct-style reasoning, enabling end-to-end observe–think–act capabilities for complex video understanding.

4. Experimental Results

Experiments were conducted on the ZH-4O (Chinese role-playing) and LoCoMo (English long-context) benchmarks.

ZH-4O Performance: TELEMEM achieved 86.33% QA accuracy, outperforming the state-of-the-art Mem0 (70.20%) by 19% in relative accuracy. It also reduced token usage by 43% and achieved a 2.1× speedup compared to baselines.
LoCoMo Performance: TELEMEM demonstrated competitive performance in single-hop (64.53%) and temporal reasoning (78.47%), approaching the performance of Long Context LLMs in these categories. However, it showed weaker performance on multi-hop reasoning (20.56%) compared to specialized baselines like Memobase (66.04%), indicating remaining challenges in complex relational reasoning.
Read-Write Scaling Law: Experiments varying model sizes for memory writing and reading LLMs revealed that performance improves monotonically as either component scales. The best results are achieved when both writing and reading capacities are scaled jointly, highlighting their interdependent nature.

5. Significance and Claims

The paper claims that TELEMEM addresses fundamental limitations of current RAG systems by shifting from unordered fragment retrieval to structured, causal memory trajectories. Its significance lies in:

Stability: By reconstructing coherent causal contexts via closure-based retrieval, it stabilizes long-horizon reasoning.
Efficiency: The structured writing pipeline significantly reduces token consumption and latency, making long-term memory practical for real-time agents.
Robustness: Narrative-driven extraction avoids schema-driven hallucinations, maintaining reliable user profiles.
Multimodality: It extends memory capabilities beyond text, supporting closed-loop reasoning over complex visual content.

The authors conclude that while no single approach dominates all reasoning types, TELEMEM's coordinated read-write design and structured organization offer a superior balance of accuracy, efficiency, and scalability for agentic AI in long-term interactive settings.