LatentMem: Customizing Latent Memory for Multi-Agent Systems

This paper introduces LatentMem, a learnable multi-agent memory framework that addresses memory homogenization and information overload by using an experience bank and a memory composer to generate customized, token-efficient latent memories, further optimized via Latent Memory Policy Optimization (LMPO) to significantly enhance multi-agent system performance.

The Gist

Imagine a team of AI agents working together to solve a complex puzzle, like building a software app or planning a robot's path through a maze. In the past, these teams had a major problem: they were all trying to remember the exact same things in the exact same way.

Think of it like a group of friends trying to plan a road trip. If everyone is forced to carry the entire map, every receipt, and every photo from the last 100 trips in their pockets, they get overwhelmed. The driver gets lost in the paperwork, the navigator forgets the turn-by-turn directions, and the mechanic can't find the specific tool they need. This is what happens in current AI systems: too much information, and everyone remembers it identically, regardless of their specific job.

LatentMem is a new framework that fixes this by giving the team a "smart, personalized memory system." Here is how it works, broken down into simple concepts:

1. The "Experience Bank" (The Raw Library)

Imagine a massive, lightweight library where the team stores the raw transcripts of every road trip they've ever taken. It doesn't try to summarize or edit these trips yet; it just stores the raw data efficiently.

• The Problem it solves: Instead of rewriting the whole story every time, the system just keeps the original logs. It's like having a digital recorder for every conversation.

2. The "Memory Composer" (The Personalized Summarizer)

This is the magic part. When a specific agent (say, the "Driver") needs to make a decision, it doesn't read the whole library. Instead, it asks the Memory Composer for help.

• How it works: The Composer looks at the raw library, sees what the Driver needs, and says, "Ah, you're the Driver. Here is a 3-sentence summary of the last 5 times you took a left turn on a rainy road."
• The Magic: If the "Mechanic" asks the same question, the Composer gives a totally different summary: "Here is the last 3 times the engine overheated."
• The Result: The memory is customized. The Driver isn't distracted by engine specs, and the Mechanic isn't confused by turn signals. This stops the team from "homogenizing" (all thinking the same way).

3. "Latent Tokens" (The Secret Handshake)

Usually, AI agents talk to each other using long, chatty sentences. This takes up a lot of space and time (like sending a 10-page email when a text message would do).
LatentMem compresses these memories into Latent Tokens.

• The Analogy: Imagine instead of writing a paragraph about "how to fix a flat tire," the agent just sends a secret, compressed code (a "latent token") that instantly triggers the knowledge of "fix flat tire" in the other agent's brain.
• The Benefit: It's incredibly fast and uses very little "bandwidth" (tokens), allowing the team to work much faster without getting bogged down by long texts.

4. "LMPO" (The Coach that Learns)

How does the Memory Composer know what to summarize? It uses a training method called **Latent Memory Policy Optimization (LMPO).

• The Analogy: Think of LMPO as a coach watching the team play. If the team wins, the coach tells the Composer, "Great job! The summary you gave the Driver was perfect." If they lose, the coach says, "You gave the Mechanic too much info about the weather; next time, focus on the engine."
• The Result: The system learns to create better, more useful summaries over time without needing a human to rewrite the rules.

Why is this a big deal?

The paper shows that with LatentMem:

1. They are smarter: The team solves puzzles (like coding or logic games) up to 19% better than before.
2. They are faster: They use 50% fewer words (tokens) to get the job done, saving time and money.
3. They are adaptable: They can walk into a brand new type of game or a new team structure and still perform well, because they know how to customize their own memories on the fly.

In short: LatentMem stops AI teams from drowning in a sea of identical data. Instead, it gives every team member a personalized, compressed "cheat sheet" that is perfectly tailored to their specific role, making the whole team smarter, faster, and more efficient.

Technical Summary

Here is a detailed technical summary of the paper "LatentMem: Customizing Latent Memory for Multi-Agent Systems".

1. Problem Statement

Large Language Model (LLM)-powered Multi-Agent Systems (MAS) rely on memory to accumulate experiences and adapt over time. However, existing memory architectures suffer from two fundamental bottlenecks:

• Memory Homogenization: Most current systems use a "one-size-fits-all" memory strategy, ignoring the functional heterogeneity of different agents (e.g., a coder vs. a tester). This lack of role-awareness undermines role adherence, amplifies correlated errors, and weakens system robustness.
• Information Overload: MAS interactions often involve long contexts. Existing multi-granularity memory designs (storing raw trajectories, distilled insights, or skill schemas) introduce massive volumes of discrete textual entries. This overwhelms agents, obscuring critical decision signals and increasing token costs.

The core challenge is to design a learnable, role-aware, and token-efficient memory system that avoids extensive manual engineering while enabling continual adaptation in complex MAS environments.

2. Methodology: LatentMem

The authors propose LatentMem, a framework that generates agent-specific memories in a compact latent space rather than relying on handcrafted textual units. The system consists of three core components:

A. Lightweight Experience Bank

• Function: Stores raw interaction trajectories (agent index, input prompt, output) from previous MAS executions.
• Mechanism: It operates without human priors (like manual summarization). Upon a new query, it retrieves a subset of relevant trajectories ($K$) based on similarity (using embeddings like MiniLM) to the current query.
• Update: New trajectories are appended online, enabling incremental learning without retraining foundation models.

B. Memory Composer

• Function: A learnable neural network ($\sigma_\phi$) that acts as a "distiller."
• Input: It takes the retrieved raw trajectories ($T_q$) and the specific Role Profile ($\gamma$) of the active agent.
• Output: It synthesizes a fixed-length latent memory matrix ($m_j \in \mathbb{R}^{L' \times D}$).
• Integration: These latent tokens are concatenated with the agent's input token embeddings, effectively injecting memory into the reasoning process without modifying the underlying agent architecture. This ensures the process remains end-to-end differentiable.

C. Latent Memory Policy Optimization (LMPO)

To train the Memory Composer to generate high-utility representations, the authors introduce LMPO, a variant of Group Relative Policy Optimization (GRPO).

• Differentiability: The latent memory serves as a differentiable interface. Gradients from task-level rewards flow back through the agent's generation process to update the Memory Composer's parameters ($\phi$), while the agent backbone remains frozen.
• Token-Level Objective: Unlike standard RL which optimizes at the trajectory level, LMPO adopts a token-level surrogate objective. It calculates advantages based on group-relative rewards and optimizes the probability of generating specific tokens conditioned on the latent memory. This prevents the "dilution" of gradients in long-horizon tasks and encourages the composer to distill critical coordination patterns.

3. Key Contributions

1. Role-Aware Customization: LatentMem conditions memory generation on agent role profiles, ensuring that each agent receives a memory representation tailored to its specific function, thereby mitigating homogenization.
2. Token Efficiency via Latent Encoding: By encoding memory as fixed-length latent vectors rather than unbounded text, the system drastically reduces context window usage and mitigates information overload.
3. End-to-End Learnability: The introduction of LMPO allows the memory system to be optimized directly via reinforcement learning signals, enabling autonomous internalization of high-utility knowledge without rigid architectural constraints.
4. Framework Agnosticism: The method does not require modifications to the underlying MAS frameworks (e.g., AutoGen, MacNet), acting as a plug-and-play memory layer.

4. Experimental Results

The authors evaluated LatentMem across six benchmarks (TriviaQA, KodCode, StrategyQA, PopQA, BigCodeBench, PDDL) and four mainstream MAS frameworks (AutoGen, MacNet, CAMEL, DyLAN).

• Performance Gains:
  • LatentMem achieved up to 19.36% improvement over vanilla (no-memory) settings.
  • On specific tasks, it improved AutoGen by 16.20% (TriviaQA) and MacNet by 18.45% (KodCode).
  • It consistently outperformed state-of-the-art baselines (including MetaGPT, ChatDev, OAgents, and G-Memory) across both in-domain and out-of-domain tasks.
• Generalization:
  • Out-of-Domain: On unseen datasets like PDDL, LatentMem improved performance by 7.10%, whereas other methods often degraded.
  • Unseen Frameworks: When applied to unseen MAS frameworks like CAMEL, it showed a 7.90% gain, demonstrating robustness to new agent roles and collaboration patterns.
• Efficiency:
  • Token Usage: LatentMem used 50% fewer tokens compared to mainstream memory designs. In some cases (e.g., AutoGen on KodCode), it used even fewer tokens than the "No-Memory" baseline while achieving higher accuracy.
  • Inference Time: It reduced inference time to approximately 2/3 of competing memory-heavy architectures.
• Comparison with Fine-Tuning: LatentMem outperformed MARTI, a multi-agent fine-tuning baseline trained under identical computational budgets, proving that memory customization is more effective than direct backbone fine-tuning for MAS.

5. Significance and Impact

• Paradigm Shift: LatentMem shifts the memory paradigm from "handcrafted textual retrieval" to "learnable latent synthesis," addressing the scalability limits of context windows in multi-agent settings.
• Scalability: By decoupling memory storage (raw trajectories) from memory usage (latent tokens), the system scales efficiently with task complexity without overwhelming the LLM's context window.
• Robustness: The ability to maintain performance across unseen domains and frameworks suggests that role-aware latent memory is a critical component for building truly generalizable and adaptive multi-agent systems.
• Practicality: The plug-and-play nature of the framework makes it immediately applicable to existing MAS deployments without requiring complex re-architecting of agent workflows.

In conclusion, LatentMem demonstrates that customizing memory at the latent level is a superior strategy for multi-agent systems, offering a balance of high performance, extreme efficiency, and strong generalization that current text-based memory systems cannot match.