🤖 AI

Multi-scale Mixture of World Models for Embodied Agents in Evolving Environments

MuSix is a novel framework for embodied agents that enhances multi-scale reasoning and adaptation in evolving environments by introducing a scale-aware Mixture of Experts architecture with experiential distance-based routing and scale-dependent forgetting mechanisms.

Original authors: Jinwoo Jang, Daniel J. Rho, Sihyung Yoon, Hyunsuk Cho, Honguk Woo

Published 2026-07-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jinwoo Jang, Daniel J. Rho, Sihyung Yoon, Hyunsuk Cho, Honguk Woo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a robot trying to navigate a house that is constantly changing. Sometimes you just need to pick up a cup (a simple, immediate task). Other times, you need to figure out how to escape a fire spreading through the hallway (a complex, abstract, and urgent situation).

The paper introduces a new system called MuSix designed to help robots handle these different situations better. It does this by acting like a team of specialists, where the robot knows exactly which specialist to call based on how "new" or "strange" the current situation feels.

Here is a breakdown of how it works, using simple analogies:

The Problem: The "One-Size-Fits-All" Robot

Current robots often use a single "brain" or a fixed set of rules. The authors argue this is inefficient because:

It doesn't know the scale: If a robot is confused, it might try to use a high-level "philosopher" brain to solve a simple "pick up the spoon" problem, or use a simple "reflex" brain to solve a complex "fire escape" plan.
It learns too slowly or too fast: If the robot updates its knowledge every time it sees something new, it might forget important long-term rules (like "fire is bad"). If it never updates, it can't learn that a specific room is now on fire.

The Solution: MuSix (The "Specialist Team")

MuSix treats the robot's brain like a Mixture of Experts. Imagine a hospital with different departments:

Low-Scale Doctors: Experts in immediate, physical details (e.g., "Is the floor slippery?").
High-Scale Doctors: Experts in abstract planning and big-picture rules (e.g., "What is the safest exit route?").

MuSix has two main innovations to make this team work perfectly:

1. The "Novelty Meter" (Experiential Distance)

Instead of guessing which doctor to call, MuSix has a "Novelty Meter." It measures Experiential Distance.

The Analogy: Think of your own memory. If you walk into your kitchen, it feels familiar (low distance). If you walk into a room you've never seen before, it feels strange and new (high distance).
How it works: The robot compares the current situation to everything it has experienced before.
- Familiar situation? The meter stays low. The robot calls the Low-Scale experts to handle quick, routine tasks.
- Strange/New situation? The meter spikes. The robot calls the High-Scale experts to figure out the big picture and plan a new strategy.

2. The "Two-Stage Switch" (Routing)

Old systems just picked a doctor randomly or based on a vague feeling. MuSix uses a Two-Stage Routing system:

Stage 1: The "Novelty Meter" decides which level of expertise is needed (Low, Medium, or High).
Stage 2: Once the level is chosen, a specific expert within that level is selected to do the work.
Why it matters: This ensures the robot doesn't waste time using a complex planner for a simple task, and doesn't use a simple reflex for a dangerous crisis.

3. The "Memory Update" System (Evolution)

The robot needs to learn from new experiences without forgetting old ones. MuSix handles this with Scale-Dependent Forgetting:

Low-Scale Memory (Fast Update): Details about the immediate environment (like "the floor is wet right now") change quickly. MuSix updates these memories fast and forgets them just as fast when they are no longer true.
High-Scale Memory (Slow Update): Big rules (like "don't touch fire") should stay stable. MuSix updates these very slowly so the robot doesn't accidentally "unlearn" safety rules.
The Gated Transfer: Sometimes, a high-level realization (e.g., "The fire is spreading") needs to tell the low-level reflexes (e.g., "Run left"). MuSix has a "gate" that allows information to flow between the levels only when necessary, keeping the whole team coordinated.

The Results: Does it Work?

The authors tested MuSix in two main scenarios:

Complex Reasoning (EmbodiedBench): In tasks requiring the robot to understand objects, navigate rooms, and follow complex instructions, MuSix performed better than previous top methods. It was particularly good at tasks requiring both physical dexterity and abstract planning.
Dynamic Disasters (HAZARD): They simulated environments where conditions changed rapidly, like a spreading fire or rising flood. MuSix was better at adapting its strategy on the fly. It saved more objects and caused less damage to the environment compared to other robots because it could switch between quick reactions and strategic planning effectively.

Summary

In short, MuSix is a smarter way for robots to organize their knowledge. Instead of having one brain that tries to do everything at once, it uses a "Novelty Meter" to decide whether to use a reflex (for familiar things) or a strategist (for new things). It also updates its memory at different speeds depending on the importance of the information, ensuring it stays agile in a changing world without losing its core principles.

Technical Summary: MuSix – Multi-scale Mixture of World Models for Embodied Agents

Problem Statement

Embodied agents powered by Vision-Language Models (VLMs) face significant challenges when operating in real-world, non-stationary environments. While recent advances have improved instruction following, these agents struggle with multi-scale reasoning (integrating low-level physical dynamics with high-level abstract inference) and continuous adaptation to changing conditions.

The authors identify two specific limitations in applying standard Mixture of Experts (MoE) architectures to this domain:

The Mixture Challenge: Conventional MoE routing lacks an explicit notion of "scale." Routing decisions are based on latent representations without tying them to identifiable scales, preventing targeted updates to specific knowledge granularities at test time.
The Evolution Challenge: A uniform update policy cannot accommodate the different rates at which knowledge becomes outdated. Low-level knowledge (e.g., local dynamics) changes frequently, while high-level abstractions (e.g., rules) remain stable. Standard MoE forces all experts to evolve at the same rate, hindering effective adaptation.

Methodology: MuSix Framework

The paper proposes MuSix, a framework that integrates world models through scale-aware mixture and evolution. The core innovation is grounding routing decisions in experiential distance, a scalar measure of situational novelty inspired by Construal Level Theory (CLT). CLT posits that psychological distance governs the level of abstraction in human reasoning; MuSix operationalizes this by mapping the gap between current observations and accumulated experience to a continuous scale space.

1. Two-Stage Scale-Based Routing

To address the mixture challenge, MuSix decouples routing into two stages:

Stage 1: Meta-Router (Scale Determination): A continuous meta-router ($MR$) maps experiential distance ( $\delta$ $δ$ ) to a weight function over a continuous scale space ( $S$ $S$ ).
- Experiential Distance ( $\delta$ ): Calculated as the Mahalanobis distance between the current observation embedding and a weighted distribution of the agent's past experience within an episode. High $\delta$ indicates novelty; low $\delta$ indicates familiarity.
- Scale-Aware Loss: A loss function ( $\mathcal{L}_{space}$ ) aligns the expected scale magnitude with $\delta$ , ensuring familiar situations route to low-scale models (concrete patterns) and novel situations route to high-scale models (abstract generalization).
Stage 2: Base Router (World Model Selection): The scale weights from the meta-router are integrated with per-scale base router scores to select a top- $k$ subset of world models from $G$ groups. This ensures the scale of the routing decision is explicit.

2. Intra- and Inter-Scale Knowledge Adaptation

To address the evolution challenge, MuSix introduces mechanisms for test-time training:

Intra-Scale Adaptation: Knowledge increments (derived from prediction errors) are distributed across groups based on the meta-router's output. Crucially, scale-dependent forgetting rates are applied: low-scale groups have high forgetting rates (rapid refresh), while high-scale groups have low forgetting rates (persistence of stable abstractions).
Inter-Scale Adaptation: A gated cross-scale transfer mechanism allows knowledge to flow bidirectionally between neighboring scales. Gating parameters modulate information flow, ensuring that updates at one level can inform adjacent levels while maintaining hierarchical coherence.

Key Contributions

MuSix Framework: A multi-scale mixture of world models that enables dynamic mixing and evolution of world models at test time, grounded in experiential distance.
Two-Stage Routing: A novel routing architecture that explicitly separates scale determination from world model selection, enabling targeted updates at the appropriate scale.
Adaptation Mechanisms: Intra- and inter-scale adaptation strategies that respect distinct temporal characteristics, allowing rapid adjustment of transient details while preserving stable abstractions.
Empirical Validation: Demonstration of the framework's effectiveness on multi-scale reasoning and dynamic adaptation benchmarks.

Experimental Results

The authors evaluated MuSix on EmbodiedBench (multi-scale reasoning) and HAZARD (dynamic adaptation in disaster scenarios).

EmbodiedBench (Habitat): MuSix achieved a 6.05 percentage point (p.p.) improvement over the state-of-the-art SayCanPay baseline (40.44% vs. 34.39% average success rate). It showed substantial gains in complex reasoning (Cpx) and visual understanding (Vis) tasks.
HAZARD (Fire/Flood): MuSix achieved a 1.49 p.p. improvement over FLARE in the Fire scenario (43.71 vs. 42.22 rescue value). In the Flood scenario, while rescue values were comparable, MuSix significantly reduced the damage ratio (4.65 vs. 6.60–7.49 for baselines), indicating more efficient behavior.
Real-World Robotics: Experiments on a Franka Research 3 robot arm showed MuSix performing comparably to the strongest baseline (SayCanPay) in average success rate (56.2% vs. 58.4%) while achieving the highest worst-case performance across diverse tasks.
Ablation Studies: Removing the meta-router caused the largest performance degradation, confirming the necessity of two-stage routing. Disabling scale-dependent forgetting or cross-scale transfer also consistently reduced performance, validating the adaptation mechanisms.

Significance and Claims

The paper claims that MuSix successfully addresses the limitations of conventional MoE in embodied AI by introducing scale awareness and hierarchical evolution. By grounding routing in experiential distance, the framework allows agents to:

Dynamically select the appropriate level of abstraction for a given situation.
Update specific knowledge granularities without disrupting the entire system (mitigating catastrophic forgetting).
Balance rapid adaptation to local changes with the stability of high-level strategies.

The authors note that the framework's effectiveness stems from the routing mechanism itself rather than specific world model architectures, as it performed robustly across various architectural variants. However, they modestly acknowledge that the framework inherits the limitations of its underlying VLM backbone and that broader validation across diverse robotic platforms is a necessary future step.