Lifelong Embodied Navigation Learning

This paper introduces Uni-Walker, a lifelong embodied navigation framework that addresses catastrophic forgetting in large language model-based agents by decoupling navigation knowledge into shared and task-specific components using DE-LoRA, knowledge inheritance, and expert subspace orthogonality to enable continuous adaptation across diverse scenes and instruction styles.

The Gist

Imagine you are teaching a robot butler named Uni-Walker how to navigate your house.

In the past, if you wanted the robot to learn a new room (like a kitchen) or a new way of giving instructions (like speaking in riddles instead of clear commands), you had to retrain the robot from scratch. The problem? Every time you taught it something new, it would forget how to do the old things. This is called "catastrophic forgetting." It's like a student who studies for a history exam and immediately forgets everything they learned in math class.

This paper introduces a new way to train robots so they can learn forever without forgetting. Here is how they did it, explained simply:

1. The Big Problem: The "Amnesia" Robot

Currently, if a robot learns to find the "sofa" in the living room, and then you ask it to find a "bed" in a bedroom, it often gets confused. It might try to find the sofa in the bedroom, or it might forget how to follow step-by-step instructions because it's trying to learn a new style of talking.

The researchers created a challenge called LENL (Lifelong Embodied Navigation Learning). Think of this as a "Marathon of Learning." The robot has to run a series of different navigation tasks one after another, and at the end, it must be able to do all of them perfectly, not just the last one.

2. The Solution: The "Swiss Army Knife" Backpack

To solve this, the team built Uni-Walker, a robot brain with a special backpack called DE-LoRA.

Imagine the robot's brain is a giant library.

• Old Way: Every time the robot learned a new task, they built a whole new library wing. Eventually, the building was too big, and the robot couldn't find anything.
• Uni-Walker's Way: They use a modular backpack.
  • The Shared Pocket (Task-Shared Knowledge): This is a common pocket for things all tasks need, like "how to walk forward" or "how to read a map." This pocket stays the same and gets better over time.
  • The Specialized Pockets (Task-Specific Knowledge): These are small, detachable pouches for specific skills. One pouch is for "finding a bed," another is for "following a dialogue," and another is for "finding a specific object."

When the robot learns a new task, it doesn't rewrite its whole brain. It just adds a new specialized pouch to the backpack and updates the shared pocket slightly.

3. How It Learns Without Forgetting

The paper describes three clever tricks Uni-Walker uses:

• The "Family Tree" Trick (Knowledge Inheritance):
  When the robot learns a new task (like finding a bed), it looks at its "family tree" of past knowledge. If it already knows how to find a "chair," it uses that knowledge as a starting point to learn about the "bed." It's like a chef who knows how to bake a cake using a specific oven; when they learn to bake a pie, they don't start from zero—they just tweak the recipe using the same oven knowledge.

• The "Group Chat" Trick (Co-Activation):
  When the robot faces a new task, it doesn't just use the new pouch. It "wakes up" a few other relevant pouches from the past to help out. It's like asking a group of friends for advice. Even if the task is new, the robot asks, "Hey, who here has dealt with something similar?" and combines their wisdom.

• The "Silos" Trick (Orthogonality):
  To make sure the new knowledge doesn't mess up the old knowledge, the robot keeps the specialized pouches strictly separate. Imagine putting different types of spices in sealed jars so the cinnamon doesn't mix with the salt. This ensures that learning to find a "bed" doesn't accidentally make the robot forget how to find a "sofa."

4. The "Smart Selector" (Task-Aware Aggregation)

Here is the tricky part: When the robot is out in the real world, it doesn't have a label saying "This is Task #5." It just sees a room and hears an instruction.

Uni-Walker has a Smart Selector. It looks at the scene (the room) and the instruction (what you said) and instantly figures out: "Oh, this looks like the 'find the bed' task, but with a twist. Let me grab the 'bed' pouch and the 'dialogue' pouch and combine them." It's like a librarian who instantly knows which book you need just by hearing your question, without you having to tell them the book's title.

5. The Result: The Ultimate Robot Butler

The researchers tested Uni-Walker against other robots.

• Other Robots: After learning 10 tasks, they were great at the 10th task but terrible at the first 9. They had amnesia.
• Uni-Walker: After learning 10 tasks, it was great at all of them. It could follow step-by-step directions, find specific objects, and understand complex conversations, all while remembering how to do everything it learned previously.

In a nutshell:
This paper teaches robots how to be lifelong learners. Instead of overwriting their memory every time they learn something new, they build a flexible, organized system where new skills are added like new chapters in a book, without erasing the old pages. This brings us one step closer to robots that can truly live with us, learn our habits, and adapt to our changing homes forever.

Technical Summary

Here is a detailed technical summary of the paper "Lifelong Embodied Navigation Learning" (Uni-Walker), published as a conference paper at ICLR 2026.

1. Problem Definition: Lifelong Embodied Navigation Learning (LENL)

The paper addresses a critical gap in embodied AI: while Large Language Model (LLM) based agents excel at single navigation tasks, they suffer from catastrophic forgetting when required to learn a sequence of new tasks over time.

• The Challenge: Existing multi-task agents struggle to adapt to new scenes and diverse user instruction styles (e.g., switching from step-by-step navigation to object localization or dialogue-based navigation) without erasing previously learned knowledge.
• The Setting (LENL): The authors formalize Lifelong Embodied Navigation Learning (LENL), where an agent must sequentially learn a series of $T$ navigation tasks ($T_1, T_2, ..., T_t$).
  • Tasks: Each task involves a unique 3D scene and a specific instruction style:
    • VLN (Vision-Language Navigation): Step-by-step route following.
    • OLN (Object Localization Navigation): Finding specific objects based on high-level descriptions.
    • DUN (Dialogue Understanding Navigation): Navigating based on multi-turn user dialogues.
  • Constraints: The agent learns tasks sequentially. During testing, the task ID is agnostic (the agent must know which strategy to use without being told the task type). The goal is to maximize performance on the current task while retaining performance on all previous tasks.

2. Methodology: Uni-Walker Framework

The proposed solution, Uni-Walker, is a lifelong learning framework designed to decouple navigation knowledge into task-shared and task-specific components. It leverages a Decoder Extension LoRA (DE-LoRA) architecture.

A. Core Architecture: Decoder Extension LoRA (DE-LoRA)

Instead of training a separate LoRA adapter for every task (which ignores shared knowledge) or a single shared adapter (which causes interference), Uni-Walker uses a hybrid approach:

• Shared Subspace ($A$): A global low-rank adapter shared across all tasks to capture common navigation knowledge (e.g., basic movement, scene understanding).
• Expert Subspaces ($B_t$): Task-specific low-rank adapters. For each new task $t$, a new expert $B_t$ is dynamically added.
• Forward Pass: The output is computed as $y = W_0 x + \sum (B_{active} \cdot A \cdot x)$, where a subset of experts is activated based on the current input.

B. Key Strategies for Lifelong Learning

To manage the trade-off between learning new tasks and retaining old ones, Uni-Walker employs four specific mechanisms:

1. Knowledge Inheritance Strategy (KIS):

   • Goal: Accelerate learning of new tasks by initializing them with relevant prior knowledge.
   • Mechanism: When a new task $T_t$ arrives, the new expert $B_t$ is initialized not randomly, but by performing Principal Component Analysis (PCA) on the experts of previous tasks with the same instruction style. This aligns the new expert with the principal subspace of similar tasks.

2. Experts Co-Activation Strategy (ECAS):

   • Goal: Leverage shared knowledge during inference and training.
   • Mechanism: Instead of activating only the specific expert for the current task, the model activates the Top-K most relevant experts (including frozen experts from previous tasks) alongside the trainable current expert. This allows the model to "borrow" knowledge from related past tasks dynamically.

3. Expert Subspace Orthogonality Constraint (ESOC):

   • Goal: Prevent task-specific knowledge from overlapping and interfering (catastrophic forgetting).
   • Mechanism: An orthogonality loss is applied to ensure the new expert subspace $B_t$ is orthogonal to all previous expert subspaces. This forces the model to learn distinct representations for different instruction styles.

4. Navigation-Specific Chain-of-Thought (NSCoT):

   • Goal: Enhance reasoning capabilities tailored to specific instruction styles.
   • Mechanism: The model uses distinct Chain-of-Thought (CoT) prompts for VLN, OLN, and DUN tasks.
     • VLN: Focuses on step-by-step path tracking.
     • OLN: Focuses on object identification and semantic reasoning.
     • DUN: Focuses on dialogue history and intent inference.

5. Task-Aware Knowledge Aggregation (TAKA):

   • Goal: Solve the task-agnostic inference problem (deciding which expert to use without a task ID).
   • Mechanism: The system maintains a retrieval index of scene and instruction embeddings from past tasks. At inference, it computes cosine similarity between the current input and stored embeddings to select the Top-K most relevant experts to activate.

6. Shared Smoothing Consolidation (SSC):

   • A regularization loss using the Fisher Information Matrix to ensure the shared subspace $A$ updates smoothly, preventing drastic shifts that would erase old knowledge.

3. Experimental Setup & Results

• Benchmark: The authors constructed a new LENL Benchmark using the Matterport3D simulator, consisting of 18 sequential tasks (15 for training, 3 for unseen generalization testing) covering VLN, OLN, and DUN styles.
• Baselines: Compared against standard Continual Learning methods (Seq-FT, LwF, EWC) and advanced LoRA/MoE methods (HydraLoRA, BranchLoRA, MoLE, SD-LoRA).
• Metrics: Success Rate (SR), Success weighted by Path Length (SPL), Oracle Success Rate (OSR), and their respective Forgetting Rates (lower is better).

Key Results:

• Performance: Uni-Walker achieved an average Success Rate (SR) of 66%, surpassing the previous best (SD-LoRA + TAKA at 59%) by 7%.
• Forgetting: It achieved a forgetting rate of only 5%, significantly better than the prior best of 16%.
• Generalization: On unseen scenes (Tasks 16-18), Uni-Walker achieved 62% SR, outperforming the next best method (57%).
• Ablation: Removing any component (KIS, ECAS, ESOC, NSCoT, TAKA) resulted in significant performance drops, confirming the necessity of the full architecture. For instance, removing NSCoT caused a 16.2% drop in SR, highlighting the importance of style-specific reasoning.

4. Key Contributions

1. Problem Formulation: Introduced LENL, a novel benchmark and problem setting for lifelong embodied navigation that requires adapting to sequential scenes and instruction styles without forgetting.
2. Architecture: Proposed Uni-Walker with DE-LoRA, which effectively decouples shared and specific knowledge using a dynamic expert expansion mechanism.
3. Strategies: Developed four novel strategies:
   • KIS & ECAS for efficient transfer and refinement of shared knowledge.
   • ESOC & NSCoT for isolating and enhancing task-specific reasoning.
   • TAKA for task-agnostic expert selection.
4. Empirical Validation: Demonstrated State-of-the-Art (SOTA) performance in both retention (low forgetting) and generalization (unseen scenes), proving the feasibility of universal embodied agents via lifelong learning.

5. Significance

This work represents a significant step toward universal embodied agents. By solving the catastrophic forgetting problem in complex, multi-modal navigation scenarios, Uni-Walker demonstrates that robots can continuously evolve their skill sets in real-world environments (e.g., learning to navigate a new house or understand new user dialects) without needing to be retrained from scratch or storing massive amounts of historical data. The framework bridges the gap between static multi-task training and the dynamic, lifelong learning capabilities observed in humans.