🤖 machine learning

Rosetta Memory: Adaptive Memory for Cross-LLM Agents

The paper introduces Rosetta Memory, a memory-centric framework that employs jointly trained, profile-conditioned operators and a minimum-gain sampling curriculum to enable adaptive memory sharing and robust performance across diverse and frequently switched large language models.

Original authors: Hao Yang, Shiqi Shen, Haoxuan Li, Zhipeng Wang, Zhi Gong, Xu Chen

Published 2026-06-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hao Yang, Shiqi Shen, Haoxuan Li, Zhipeng Wang, Zhi Gong, Xu Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Lost in Translation" Agent

Imagine you have a team of workers (AI agents) trying to solve a complex mystery, like a detective story. To solve it, they need to pass notes to each other.

Worker A is a fast, casual coder who writes notes in quick, shorthand bullet points.
Worker B is a slow, careful writer who needs full, polite sentences to understand things.
Worker C is a visual thinker who needs information organized in a table.

In the past, if Worker A wrote a note and passed it to Worker B, Worker B often got confused. The note was written in "Worker A's language," not "Worker B's language." The team would lose track of clues, forget important details, or misunderstand the instructions. This is what happens when different Large Language Models (LLMs) try to share memory: the "language" of the memory doesn't match the "language" of the reader.

The Solution: Rosetta Memory (RoMem)

The authors created a system called Rosetta Memory. Think of it as a universal translator and a specialized notary for your AI team.

Instead of forcing every worker to write in the same way, Rosetta Memory sits in the middle and adapts the notes based on who is writing and who is reading.

It uses two special tools (operators):

The "Translator" (Write Operator): When a worker writes a note, this tool looks at who wrote it. It takes their raw, messy notes and rewrites them into a "standardized" format that is easy for anyone to understand later. It strips away the writer's personal quirks.
The "Adapter" (Read Operator): When a worker needs to read a note, this tool looks at who is reading it. It takes that standardized note and reshapes it into the specific format that reader prefers (e.g., turning a paragraph into a table for the visual thinker, or a bullet list for the coder).

How They Taught the System

To teach this system how to work, the researchers didn't just show it one type of worker. They created a "training camp" with many different types of AI models (some fast, some slow, some from different companies).

They used a clever training trick called "Minimum-Gain Sampling."

Imagine a teacher who notices that the "slow, careful writer" (Worker B) is struggling the most to understand the notes.
Instead of practicing with the easy students, the teacher forces the system to practice extra hard with the struggling students.
This ensures the system gets really good at helping the models that usually get left behind, rather than just getting good at the average case.

They also used a "scorecard" that only gave points if the system actually improved the team's performance compared to just passing raw notes around. This ensured the system was learning to translate, not just memorizing answers.

What They Found

The researchers tested this on three different "mystery" games (datasets called HotpotQA, 2WikiMultihopQA, and MuSiQue) where the AI had to connect multiple clues to find an answer.

The Result: Teams using Rosetta Memory solved the mysteries much better than teams using standard methods.
The "Unseen" Test: Even when they tested the system with a new type of worker they had never seen before during training, it still worked well. It didn't break when the team composition changed.
The "Mismatch" Proof: They showed that without this system, if you switch workers mid-task, the team often fails because the notes don't make sense to the new worker. With Rosetta Memory, the notes always make sense, no matter who is reading them.

In a Nutshell

Rosetta Memory is a smart middleman that ensures AI agents can swap notes seamlessly, even if they speak different "languages" or have different styles. It translates the memory when it's written and adapts it when it's read, making sure the whole team stays on the same page, regardless of which specific AI model is doing the work at any given moment.

Technical Summary: Rosetta Memory (RoMem)

1. Problem Definition: Cross-LLM Memory Mismatch

The paper addresses a critical gap in the deployment of Large Language Model (LLM) agents: the assumption that memory systems are designed for a single, static backbone. In practice, agents frequently switch between different LLMs (e.g., using Claude for coding and GPT for writing, or routing steps to different models for cost-efficiency). The authors identify that memory written by one model often fails to be effectively consumed by another due to two primary sources of mismatch:

Write-Side Divergence: Different source LLMs, given the same state, produce different actions and thus store different content in memory.
Read-Side Sensitivity: Even semantically equivalent memory entries can lead to different agent outcomes when read by different target LLMs due to variations in prompt sensitivity and context interpretation.

The authors formalize the Cross-LLM Memory Adaptation Problem: How can an agent's memory system adapt so that memory written by an upstream LLM effectively activates a downstream LLM, regardless of the specific models involved in the trajectory? The goal is to maximize the expected return across all possible model assignments in a heterogeneous pool, rather than optimizing for a single fixed backbone.

2. Methodology: Rosetta Memory (RoMem)

RoMem shifts the perspective from "LLM-centric memory design" to "memory-centric LLM adaptation." It treats memory as a transferable interface between upstream writers and downstream readers.

Core Architecture

The system introduces two learnable, LLM-aware operators:

Write Operator ( $f_{write}$ ): Takes the source LLM's identifier (profile), the observation, and the action, and outputs a distilled, normalized memory entry.
Read Operator ( $f_{read}$ ): Takes the target LLM's identifier (profile), the current observation, and the memory bank, and outputs a prompt-ready context adapted for the specific reader.

Profile Conditioning

To handle diverse models without fine-tuning the operators for every specific model, RoMem uses profile embeddings. Each LLM is associated with a lightweight profile record (model ID, textual description, metadata like backend, family, scale). A learnable mapping function $g$ converts these profiles into continuous vectors (soft prefix tokens) that condition both $f_{write}$ and $f_{read}$ . This allows the operators to generalize to unseen LLMs by understanding their "style" and capabilities.

Training Strategy

The operators are trained using an Expert Iteration loop with two specific innovations to ensure robustness across the heterogeneous model pool:

Counterfactual Gain Reward: To isolate the contribution of the learned operators from the intrinsic capability of the LLMs, the system calculates a performance gap ( $\Delta R$ ). This is the difference between the reward achieved with the learned operators and a naive baseline (raw memory concatenation). This ensures the model learns to improve performance specifically through adaptation, not just by leveraging a strong backbone.
Minimum-Gain Sampling Curriculum: During training, the system prioritizes "underserved" LLMs—those with the lowest counterfactual gain. The sampling probability for a model at step $t$ is biased toward models with low $\Delta R$ and is also diversified by embedding distance to ensure behavioral diversity within a single trajectory. This prevents the model from overfitting to high-performing pairs and ensures coverage for difficult model combinations.

3. Key Contributions

Problem Formalization: The paper empirically demonstrates (via pilot studies on HotpotQA, 2WikiMultihopQA, and MuSiQue) that memory mismatch is a significant issue in cross-LLM settings and formally defines the cross-LLM memory adaptation problem.
Rosetta Memory (RoMem): A novel framework that jointly learns profile-conditioned write and read operators to adapt memory for heterogeneous agents.
Training Innovations: The introduction of a minimum-gain sampling curriculum to improve coverage of underperforming model pairs and a performance-gap reward to disentangle operator utility from backbone strength.
Empirical Validation: Extensive experiments showing that RoMem outperforms memory-free baselines (CoT, Self-Ask), raw-buffer baselines (Step-Accumulate), and structured memory baselines (Mem0, Memento) across multiple datasets.

4. Experimental Results

The authors evaluated RoMem on three multi-hop QA datasets: HotpotQA, 2WikiMultihopQA, and MuSiQue.

Overall Performance: RoMem consistently achieved the highest scores in Exact Match (EM), F1, and semantic accuracy (AccLLM) across all datasets. Notably, on the difficult MuSiQue dataset, it showed clear margins in F1 and AccLLM, suggesting effective preservation of partial evidence across complex chains.
Ablation Studies: Removing the profile conditioning or the write operator significantly degraded performance, confirming that both the adaptation of the memory content and the adaptation of the presentation to the target model are necessary.
Generalization to Unseen Models: When tested on six held-out LLMs (e.g., gpt-5-mini, deepseek-reasoner) not seen during training, RoMem maintained robust performance, often outperforming baselines even as the number of unseen models in the trajectory increased.
Minimum-Gain Sampling: Sensitivity analysis showed that moderate reweighting (e.g., $\eta \approx 1.2$ to $1.6$) improves performance, while overly strong reweighting can be detrimental.
Qualitative Analysis: Inspections revealed that the write operator successfully removes source-model uncertainty and search instructions, while the read operator adapts the ordering and explicitness of facts to match the target model's preferences (e.g., table format for Gemini vs. chain-of-thought for Claude).

5. Significance and Claims

The paper claims that storing evidence is insufficient; the memory must be written and read in a form adapted to the specific LLM consuming it. By treating memory as a transferable interface, RoMem enables heterogeneous LLM agents to share episode memory effectively, reducing the need for redundant computation and improving robustness when models are swapped due to cost, latency, or availability constraints.

The authors position this work as a "first step" toward memory-centric adaptation. They acknowledge limitations, noting that their evaluation is restricted to controlled multi-hop QA episodes with fixed evidence chains and does not yet cover open-ended web navigation or very long-horizon planning. They also caution that learned memory interfaces could potentially propagate biases or errors if not carefully audited, suggesting the need for data minimization and human inspection in high-stakes deployments.