💬 NLP

Long-Horizon Embodied Decision-Making via Multimodal Memory Compression

This paper introduces DunphyBench, a new benchmark for evaluating long-horizon embodied decision-making aligned with human preferences, and proposes MeMento, a preference-conditioned memory compressor that significantly improves agent accuracy while drastically reducing memory usage by addressing the bottlenecks of raw multimodal history.

Original authors: Bingxuan Li, Rui Yang, Cheng Qian, Jiateng Liu, Jeonghwan Kim, Zhenhailong Wang, Manling Li, Tong Zhang, Heng Ji

Published 2026-08-04

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Bingxuan Li, Rui Yang, Cheng Qian, Jiateng Liu, Jeonghwan Kim, Zhenhailong Wang, Manling Li, Tong Zhang, Heng Ji

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a personal assistant to help you find the perfect apartment in a new city. You don't just want someone to open doors; you want a partner who can remember every detail from a hundred different tours, compare them against your vague wishes like "I need a quiet place to cook" or "My dog needs a sunny spot," and finally pick the one home that fits your life best. This is the world of embodied AI, where computer agents don't just chat on a screen but "walk" through virtual worlds, looking at rooms and making decisions. The big challenge here is long-horizon decision-making: keeping track of a massive amount of information over a long time without getting confused or tired. Just like humans get "decision fatigue" after looking at too many houses, these digital agents often get overwhelmed by the sheer volume of photos, maps, and notes they collect, leading them to make poor choices.

This paper, titled "Long-Horizon Embodied Decision-Making via Multimodal Memory Compression," tackles exactly that problem. The researchers created a new test called DunphyBench, which is like a giant, high-stakes apartment hunting simulation. They found that current AI agents are struggling badly; even the smartest ones only get about 58% of the decisions right, while humans get 83%. The main culprit? The agents are drowning in their own history. When they try to remember everything they've seen, the extra noise actually makes them dumber. To fix this, the team built a new tool called MeMento. Think of MeMento as a super-efficient memory filter that reads through the agent's entire tour history and only keeps the tiny, crucial notes that match what the user actually wants, throwing away the rest. This simple trick helped the AI jump its accuracy by 7.18% while using 85.38% less memory than before, proving that sometimes, forgetting the unimportant stuff is the key to making the right choice.

The Apartment Hunt Simulation

To understand why this matters, picture the researchers setting up a massive game of "House Hunters" for robots. They built a benchmark called DunphyBench where an AI agent has to visit a series of virtual houses. The agent gets a list of user preferences, which can be clear instructions like "I need three bedrooms" or tricky hints like "I love cooking for friends" (which implies a need for a big kitchen). The agent has to walk through each house, look at the rooms, check the windows, and count the objects. Then, it has to pick the one house that fits all the clues best.

The researchers designed this to be really hard. They didn't just give the AI easy tasks; they added "distractors"—houses that look good at first glance but fail on a specific detail. They also tested two types of thinking: Explicit (where the rules are clear) and Implicit (where the AI has to guess what the user really means).

The Problem: Too Much Memory, Too Little Sense

When they ran the test, the results were a wake-up call. The best AI agents managed to get only 58.3% of the answers correct, while human testers got 83.3%. The gap was huge. The researchers dug into why the robots were failing and found three main issues:

Complexity Overload: As the tasks got harder (more rooms, more confusing clues), the AI's performance crashed. When the number of "distractor" houses increased, accuracy dropped by over 20 points.
The "Implicit" Wall: The agents were terrible at reading between the lines. When the preference was vague (like "I work from home"), the AI struggled much more than when the rule was strict (like "needs a desk").
The Memory Trap: This was the biggest surprise. The researchers thought that giving the AI more memory (showing it every photo and note from its tour) would help. Instead, it often hurt. When the AI tried to remember everything, the extra information acted like noise, confusing the agent and making it forget what actually mattered.

It turns out that for these agents, more history isn't always better. In fact, after a certain point, adding more unfiltered memories actually made the agents worse at making decisions.

The Solution: MeMento, the Memory Filter

To solve the "memory trap," the team invented MeMento. Imagine you are packing for a trip. Instead of throwing your entire closet into a suitcase, you have a smart assistant who knows exactly what you need for the weather and your activities. MeMento does this for the AI.

Here is how it works:

The Filter: MeMento looks at the user's preferences (e.g., "I need a quiet place") and creates a special "query."
The Compression: As the AI explores the houses, MeMento scans every photo and note. It asks, "Does this help answer the user's question?" If the answer is no, it throws that information away. If the answer is yes, it compresses it into a tiny, efficient "memory token."
The Result: Instead of feeding the AI a massive library of 60,000 tokens (which is what the old methods did), MeMento feeds it a tiny, focused summary of only about 3,800 tokens.

The Results: Smarter and Leaner

The experiments showed that this approach worked wonders. When the researchers used MeMento with their AI agents:

Accuracy Boost: The agents got 7.18% more correct answers (a 15.74% relative improvement).
Memory Savings: They used 85.38% less memory than the strongest previous methods.
Generalization: The team also tested MeMento on web-browsing tasks (finding specific items online), and it worked there too, suggesting this "smart filtering" idea isn't just for houses.

The researchers also ran a "memory probe" test, asking the AI to recall specific details from its tour (like "Was there a window near the sink?"). MeMento was able to answer these questions correctly 71.9% of the time, which was much closer to the perfect "oracle" score than any other method. This proved that MeMento wasn't just deleting information randomly; it was keeping the right information.

What This Means

The paper suggests that the future of helpful AI assistants isn't about giving them infinite memory or making them read every single page of a book. Instead, it's about teaching them to forget the unimportant stuff. By learning to compress their experiences into what truly matters for the user's specific goals, these agents can make better, more human-like decisions. While the gap between AI and humans still exists, MeMento shows a clear path forward: be selective, be efficient, and remember only what counts.

Technical Summary: Long-Horizon Embodied Decision-Making via Multimodal Memory Compression

1. Problem Definition

The paper addresses the challenge of long-horizon, human-centered embodied decision-making. Unlike standard embodied tasks that focus on procedural planning or immediate goal completion (e.g., "pick up the cup"), this setting requires agents to act as decision-makers on behalf of users over extended interactions. The core problem involves navigating multiple embodied environments (specifically housing), accumulating multimodal evidence over time, and selecting the optimal candidate based on multi-dimensional human preferences.

Key challenges identified include:

Preference Under-specification: User preferences are often implicit, indirect, or persona-driven (e.g., "I cook frequently" implies a need for a large kitchen) rather than explicit checklists.
Long-Horizon Grounding: Agents must link textual preferences to visual observations across multiple environments and time steps.
Memory Management: Current agents struggle to manage large volumes of accumulated evidence. The paper posits that raw, unfiltered multimodal history introduces noise that hinders decision quality, creating a non-monotonic relationship between memory size and performance.

2. Methodology

2.1 DunphyBench: A New Benchmark

To systematically evaluate this problem, the authors introduce DunphyBench, a benchmark for long-horizon human-centered embodied decision-making.

Task: The agent must explore a set of candidate housing environments (generated from ProcTHOR) and select the "golden" environment that best aligns with a set of user preferences.
Structure: Each instance includes a preference set ( $N=7$ ) and a candidate pool ( $1 + 3M$ ), where $M$ represents distractors of varying difficulty (Easy, Medium, Hard).
Preference Taxonomy: Preferences are categorized into seven semantic types: room counts/types, dimensions, windows/lighting, object presence, object counts, spatial connectivity, and materials/finishes.
Tracks: The benchmark features two tracks:
1. Explicit: Preferences are structured constraints (e.g., "3 bedrooms").
2. Implicit: Preferences are natural language statements requiring inference (e.g., "I love cooking").
Evaluation Metrics: Accuracy (correct final decision), Normalized Satisfaction Ratio (partial correctness), and Exploration Efficiency (balance between steps taken and accuracy).

2.2 MeMento: Preference-Conditioned Multimodal Memory Compression

Motivated by the finding that excessive context degrades performance, the authors propose MeMento, a framework to selectively compress decision-relevant information from long-horizon history.

Architecture: MeMento is a preference-conditioned Perceiver-style module that uses a fixed set of learnable memory tokens (queries) to compress raw multimodal history.
Encoding:
- Preference Encoding: User requirements (explicit or implicit) are transformed into keywords and pooled into a single preference embedding ( $\bar{z}_r$ ).
- Step Encoding: Each exploration step (ego-view, map-view, observations, actions) is encoded into a step representation, augmented with temporal and environment IDs.
Compression Mechanism:
- Preference-Conditioned Queries: Learnable queries are injected with the preference embedding via a projection network ( $Q_r = Q_0 + \phi(\bar{z}_r)$ ).
- Hierarchical Compression: The queries attend to step representations within each environment independently (Cross-Attention), followed by self-attention to fuse information across steps, producing compact environment-level tokens. These are concatenated to form global memory tokens.
Injection: The compressed tokens are projected into the frozen VLM embedding space via a trainable adapter and prepended to the current prompt as a structured prefix.
Training Strategy:
- Stage 1 (Preference-Guide): Trains the compressor to predict step-level preference relevance scores and aligns attention weights with relevance labels.
- Stage 2 (Alignment): Trains the compressor and adapter to make the compressed memory readable by the frozen VLM using preference-related QA tasks.

3. Key Results

3.1 Benchmark Evaluation (DunphyBench)

Human vs. Agent Gap: There is a substantial performance gap. The best agent configuration (GPT-5.4 with multimodal memory and no tools) achieved 58.3% accuracy, while human annotators achieved 83.3%.
Complexity Sensitivity: Performance degrades sharply with task complexity. Accuracy drops from ~49% to ~25% (Explicit) and ~26% to ~16% (Implicit) as difficulty increases from Easy to Hard.
Implicit Preference Bottleneck: Agents struggle significantly with implicit preferences compared to explicit ones (e.g., 26.7% vs. 49.1% in Easy settings).
Memory Impact: Incorporating multimodal memory nearly doubles accuracy compared to memory-less agents for some models. However, unfiltered memory scaling is non-monotonic; performance peaks at a moderate history length (15 turns) and declines with excessive context due to noise.

3.2 MeMento Performance

Accuracy Improvement: On DunphyBench, MeMento improved the accuracy of the base model (Qwen3VL-8B) by 7.18% (absolute) and 15.74% (relative) compared to the strongest baseline.
Memory Efficiency: MeMento reduced memory usage by 85.38% compared to the strongest baseline, Multimodal (Recent) Memory (~60K tokens), utilizing only ~3.8K tokens.
Ablation Studies:
- Removing preference conditioning caused the largest performance drop (from 52.78% to 15.67%), confirming the critical role of preference grounding.
- Removing visual memory or hierarchical compression led to moderate degradation, highlighting the synergy of multimodal evidence and structured aggregation.
Generalization: MeMento was tested on 100 web-agent tasks (Embodied Web Agent), achieving 83% success with only 512 tokens, outperforming recent memory (69% with 4.5k tokens) and text summarization (72% with 1k tokens).

4. Significance and Claims

The paper claims that effective long-horizon multimodal memory management, rather than simply scaling context windows, is the key to building robust decision-making agents.

Diagnosis: The work identifies that raw multimodal history introduces noise that hinders decision quality, and that current agents fail to track visited states or aggregate evidence effectively over long horizons.
Solution: MeMento demonstrates that selectively compressing history based on user preferences allows agents to retain decision-relevant information while discarding noise, significantly improving both accuracy and efficiency.
Benchmark Contribution: DunphyBench provides a necessary evaluation framework that moves beyond procedural planning to test an agent's ability to infer implicit preferences and reason over long-term, multi-environment evidence.

The authors conclude that their approach serves as an improved baseline for long-horizon embodied decision-making, generalizing beyond housing tasks to web-based decision scenarios.