INMS: Memory Sharing for Large Language Model based Agents

This paper proposes INMS, an asynchronous multi-agent framework that establishes a shared conversational memory pool through real-time filtering and retrieval, enabling dynamic knowledge exchange and collective self-enhancement to significantly improve LLM-based agent performance in open-ended scenarios.

The Gist

Imagine you are in a room full of brilliant experts, but they are all wearing noise-canceling headphones. They are trying to solve a difficult puzzle or write a poem, but they can only talk to themselves. They can't hear each other's ideas, so they keep making the same mistakes or getting stuck in the same loops. This is how most current AI agents work today: they are isolated, relying only on their own internal memory or a static, outdated library of books.

The paper "INMS: Memory Sharing for Large Language Model based Agents" proposes a solution to this isolation. It introduces a system called INMS (Interactive Memory Sharing), which acts like a giant, living "group chat" or a shared whiteboard for these AI agents.

Here is a breakdown of how it works, using simple analogies:

1. The Problem: The "Echo Chamber"

Currently, if an AI agent tries to solve a problem, it looks at its own past notes. If those notes are bad, the agent keeps making bad decisions. It's like a student who only studies from a textbook written in 1990; they miss out on new discoveries. Even worse, if they are in a group, they usually can't share their "aha!" moments with each other.

2. The Solution: The "Shared Whiteboard" (INMS)

INMS changes the rules. Instead of working in isolation, all the agents are connected to a Shared Memory Pool. Think of this as a giant, digital whiteboard in the center of the room that everyone can see and write on.

Here is the step-by-step process of how this "whiteboard" works:

• Step 1: The Attempt (The "Draft")
  An agent gets a question (e.g., "Write a poem about a sad robot"). It tries to answer it using what it knows plus a few helpful notes it pulls from the shared whiteboard.

• Step 2: The "Teacher" (The Scorer)
  Once the agent writes its answer, a special "Teacher AI" (a scorer) reads it. The Teacher doesn't just say "Good" or "Bad." It uses a specific rubric (like a grading sheet for a school assignment) to check if the answer is creative, accurate, and helpful.

  • Analogy: Imagine a strict but fair editor at a newspaper. If the story is boring or full of errors, they throw it in the trash. If it's great, they pin it to the "Best of the Week" board.

• Step 3: The Upgrade (The "Shared Memory")
  If the Teacher gives the answer a high score, that specific Question-and-Answer pair is saved to the Shared Whiteboard. Now, every other agent in the system can see this new, high-quality example.

  • The Magic: If Agent A solves a tricky logic puzzle, Agent B (who is trying to write a story) might see that solution and get inspired, even though they are doing different tasks. They are learning from each other's "muscle memory."

• Step 4: The Smart Search (The "Retriever")
  The system doesn't just dump everything on the whiteboard. It has a Smart Librarian (a dynamic retriever). As the whiteboard gets fuller, this librarian gets smarter. It learns exactly which notes are useful for which questions. It stops showing you old, irrelevant notes and starts showing you the perfect examples for the job at hand.

3. Why This is a Big Deal

The paper tested this system on three very different types of tasks:

1. Creative Writing: (Writing poems like Sonnets or Limericks).
2. Logic Puzzles: (Solving riddles or tricky brain teasers).
3. Planning: (Making study plans or travel itineraries).

The Results:

• Collective Intelligence: The agents got significantly better at their jobs. They didn't just get better at their own specific task; they got better because they were "standing on the shoulders" of their peers.
• Breaking the "Echo Chamber": The system proved it could fix its own mistakes. Even if the agents started with bad ideas (a biased starting point), the "Teacher" and the "Smart Librarian" eventually filtered out the bad stuff and filled the whiteboard with high-quality knowledge, allowing the agents to recover and improve.
• No More Static Databases: Traditional AI relies on a fixed database (like a library that never updates). INMS creates its own library in real-time. The more the agents talk and solve problems, the smarter the library becomes.

The Bottom Line

Think of INMS as turning a room of isolated geniuses into a collaborative society. Instead of everyone trying to reinvent the wheel, they share their best ideas, grade each other's work, and build a collective brain that is smarter than any single agent could ever be on its own. It transforms AI from a lone wolf into a pack that learns, adapts, and evolves together.

Technical Summary

Here is a detailed technical summary of the paper "INMS: Memory Sharing for Large Language Model based Agents".

1. Problem Statement

Current Large Language Model (LLM) based agents often operate in isolated environments, relying on static databases or their own internal historical context. This isolation presents several critical limitations:

• Lack of Dynamic Knowledge Exchange: Unlike human intelligence, which evolves through social interaction and continuous dialogue, current agents miss the dynamic exchange of knowledge found in multi-agent discourse.
• Static Retrieval Limitations: Existing Retrieval-Augmented Generation (RAG) systems function as one-way, static lookups. They depend heavily on the quality and up-to-date status of external databases, which often suffer from scarcity or obsolescence, particularly in specialized or rapidly evolving fields.
• Echo Chamber Effects: Agents operating in isolation or with biased initial data tend to reinforce their own errors or limited perspectives, lacking a mechanism to correct biases through diverse, collective interaction.
• Inefficient Collective Learning: Current memory enhancement methods focus on individual agents utilizing their own history, neglecting the potential for collective self-enhancement where agents learn from each other's unique conversational histories and specialized training.

2. Methodology: The INMS Framework

The authors propose INteractive Memory Sharing (INMS), an asynchronous interaction paradigm designed to transform isolated agents into a collectively intelligent society. The framework operates through a continuous cycle of generation, evaluation, sharing, and retrieval.

Core Architecture

1. Prompt-Answer (PA) Pair Generation: Agents interact by generating PA pairs. These pairs are conceptualized as "interactive memories" or asynchronous communicative acts.
2. Quality Control (LLM Scorer): A dedicated LLM acts as a dialogue moderator. It evaluates every new PA pair against domain-specific rubrics (e.g., literary quality, logical consistency, feasibility).
   • Scoring Mechanism: Instead of a single score, the LLM assigns a score range for each rubric. The final score ($S_{final}$) is calculated as the average of the sum of lower bounds and upper bounds:
     $$S_{final} = \frac{1}{2} \left( \sum L_i + \sum H_i \right)$$
   • Only PA pairs exceeding a preset threshold are accepted into the shared pool.
3. Shared Memory Pool: Accepted high-quality PA pairs form a shared conversational memory pool accessible to all agents. This pool is not static; it grows dynamically as agents interact.
4. Adaptive Retrieval & Training:
   • Retrieval: When an agent receives a query, a dense retriever fetches relevant memories from the shared pool to augment the prompt (In-Context Learning).
   • Continuous Retriever Training: Unlike traditional RAG where retrievers are static, INMS employs an autonomous retriever that is continuously retrained. As new memories are added, the retriever is updated to adapt to the evolving distribution of data, ensuring high retrieval relevance over time.
5. Interactive Learning: To overcome initial data scarcity, agents can iteratively generate new prompts based on existing answers, creating a self-sustaining loop of memory expansion.

3. Key Contributions

• The INMS Framework: Introduces a novel asynchronous interaction paradigm that shifts agents from isolated reasoning to collective intelligence via dynamic memory sharing.
• Dynamic Retriever Optimization: Proposes a mechanism where the retrieval model is continuously fine-tuned on newly generated, high-quality memories, solving the adaptability issue of static RAG systems.
• Novel Dataset Construction: Created a specialized dataset covering three diverse open-ended domains:
  • Literary Creation: Wuyanlvshi (Chinese poetry), Limericks, Sonnets.
  • Unconventional Logic: Puzzles, Riddles, Puns.
  • Plan Generation: Study, Travel, and Fitness plans.
• Bias Mitigation: Demonstrates a mechanism to dismantle "echo chambers" by filtering out low-quality/biased interactions and diluting them with diverse, high-quality collective memories.

4. Experimental Results

The authors evaluated INMS across three domains using nine specialized agents (3 per domain) and multiple LLM backbones (open-mistral-7b, gpt-3.5-turbo, gpt-4o).

• Performance Improvement: INMS significantly outperformed baselines (Random, BM25, Contriever, SBERT, TAS-B, SimCSE) in both F1 scores and LLM-Judge (semantic consistency) metrics.
  • Example: In the "Riddle" task, INMS achieved an F1 score of 0.71 compared to 0.65 for the best dense retriever (SBERT) and 0.28 for Random.
• Memory Quantity: Performance generally improved as the number of retrieved shared memories increased (up to 3 memories), confirming the value of collective knowledge.
• Domain Specificity vs. Integration: Experiments showed that domain-specific memory pools yielded better results than a single integrated pool, suggesting that specialized context is crucial for open-ended tasks.
• Cross-Model Generalization: Memories generated by agents using one LLM backbone (e.g., GPT-4) effectively improved the performance of agents using different backbones (e.g., Mistral-7b), proving the universality of the shared knowledge.
• Echo Chamber Resilience: In experiments with a biased initial memory pool (75% biased data), the system successfully recovered. As high-quality memories were filtered and added, agent performance rebounded to match unbiased scenarios, validating the robustness of the scoring and retrieval mechanism.

5. Significance and Impact

• Paradigm Shift: INMS moves the field from "isolated agent reasoning" to "collective intelligence," mimicking human social learning dynamics.
• Reduced Dependency: It reduces reliance on static, external databases by enabling agents to build their own high-quality, evolving knowledge bases through interaction.
• Scalability: The framework is scalable; as more agents interact, the shared memory pool becomes richer, leading to a positive feedback loop of performance improvement.
• Future Direction: The paper lays the groundwork for multi-agent societies where agents continuously learn from one another, offering a solution to the stagnation and bias issues prevalent in current LLM deployments.

In conclusion, INMS demonstrates that by integrating real-time memory filtering, storage, and adaptive retrieval, LLM agents can overcome the limitations of isolation, achieving higher accuracy, creativity, and robustness in open-ended scenarios.