All-day Multi-scenes Lifelong Vision-and-Language Navigation with Tucker Adaptation

Imagine you are teaching a robot dog how to navigate your house. You give it a simple command: "Go to the kitchen, turn left, and stop at the fridge."

In a perfect world, the robot learns this once and remembers it forever. But in the real world, things get messy. What if you ask it to do the same thing at 3:00 PM when the sun is blazing through the window (overexposure)? Or at 3:00 AM when it's pitch black (low-light)? Or on a foggy day when the air is thick with dust (scattering)?

If you try to teach the robot to handle the "dark" scenario, it often forgets how to handle the "bright" scenario. This is called catastrophic forgetting. It's like a student who studies for a math test, passes it, but then immediately forgets how to read because they studied for a history test right after.

This paper introduces a solution called AlldayWalker, a robot brain designed to learn everything without forgetting anything, no matter the time of day or the weather.

Here is how it works, explained with simple analogies:

1. The Problem: The "Two-Dimensional" Trap

Most current robot learning methods (like a technique called LoRA) are like flat spreadsheets.

Imagine a spreadsheet where you have one column for "Shared Knowledge" (like how to walk) and one column for "Specific Knowledge" (like how to walk in the dark).
The problem is that real life is more complex. You need to know how to walk in the dark AND in the kitchen AND in the living room.
A flat spreadsheet gets messy when you try to add too many columns. It can't easily separate "Darkness" from "Kitchen" from "Fog." It just sees a jumbled mess, so the robot gets confused and forgets old skills.

2. The Solution: The "Rubik's Cube" of Knowledge

The authors propose a new method called TuKA (Tucker Adaptation). Instead of a flat spreadsheet, imagine a Rubik's Cube (a 3D or even 4D puzzle).

The Core (The Center of the Cube): This holds the Shared Knowledge. It's the robot's basic brain: "How to walk," "What a door looks like," "How to follow a voice." This part stays the same for everyone.
The Layers (The Slices):
- One slice handles Scenes (Kitchen, Bedroom, Living Room).
- Another slice handles Environments (Sunny, Dark, Foggy, Bright).
The Magic: Because this is a 3D/4D cube, the robot can twist and turn the layers independently. It can say, "I need the 'Kitchen' slice AND the 'Dark' slice," without messing up the 'Living Room' or 'Sunny' slices.

This allows the robot to decouple (separate) its knowledge. It learns that "Darkness" is a specific setting, and "Kitchen" is a specific setting, and it can combine them instantly without overwriting its memory of "Sunny Living Room."

3. The Learning Strategy: The "Library" Approach

The paper also introduces a strategy called DKIL (Decoupled Knowledge Incremental Learning). Think of this as a very organized librarian.

The Shared Librarian (Core Tensor): There is one main librarian who knows the rules of the library (how to walk, how to listen). This librarian never changes, ensuring the robot doesn't lose its basic common sense.
The Specialized Assistants (Experts):
- There is an assistant for "Low Light."
- There is an assistant for "Overexposure."
- There is an assistant for "The Kitchen."
The Process: When the robot enters a new room in the dark, it doesn't fire up the whole brain. It just calls the "Low Light Assistant" and the "Kitchen Assistant."
The Safety Net: If the robot learns something new about the "Low Light" assistant, the system makes sure it doesn't accidentally erase what the "Sunny" assistant knows. It uses a "consistency check" to ensure the new learning fits perfectly alongside the old learning.

4. The Result: The "All-Day Walker"

The team built a robot named AlldayWalker. They tested it in a simulated world where they could change the lighting from bright noon to pitch black, or add fog and glare.

Old Robots: When they tried to learn the "Dark" task, they forgot the "Bright" task. Their success rate dropped to near zero.
AlldayWalker: It learned the dark task, the bright task, the foggy task, and the kitchen task. When tested on all of them later, it remembered everything. It didn't forget the bright days just because it learned the dark nights.

Why This Matters

This isn't just about robots walking in houses. It's about creating AI that can evolve.

Current AI: Like a student who forgets last week's lesson when they study for today's test.
AlldayWalker: Like a wise elder who accumulates knowledge over a lifetime, remembering every season, every room, and every weather condition, getting smarter and more adaptable every day without losing a single memory.

In short, the paper teaches us how to build robots that don't just learn one thing at a time, but learn everything at once, keeping their knowledge organized in a high-dimensional "Rubik's Cube" so they can navigate our messy, changing world all day long.

1. Problem Definition: All-Day Multi-Scenes Lifelong VLN (AML-VLN)

The paper addresses a critical gap in deploying Vision-and-Language Navigation (VLN) agents in real-world scenarios. While existing VLN agents perform well in controlled environments, they struggle with catastrophic forgetting when adapting to diverse, dynamic conditions (e.g., changing scenes, low-light, overexposure, scattering).

The Challenge: Traditional fine-tuning on a specific scenario degrades performance on previously learned scenarios. Existing parameter-efficient adapters (like LoRA and its MoE variants) typically use two-dimensional matrix representations. These fail to capture the multi-hierarchical nature of navigation knowledge, which spans:
1. Shared Core Skills: General navigation logic common to all tasks.
2. Scene-Specific Knowledge: Unique layouts of specific environments.
3. Environment-Specific Knowledge: Adaptation to lighting and weather conditions (e.g., night vs. day, fog vs. clear).
The Goal: To build an agent that can continually learn across a sequence of tasks (different scenes $\times$ different environments) without forgetting, effectively achieving "all-day" navigation in diverse conditions.

2. Methodology: Tucker Adaptation (TuKA)

The authors propose Tucker Adaptation (TuKA), a novel parameter-efficient fine-tuning method that moves beyond 2D matrices to high-order tensors to decouple multi-hierarchical knowledge.

A. Architecture: High-Order Tensor Representation

Instead of representing task weights as a simple matrix ( $\Delta W = BA$ ), TuKA represents the adaptation weights as a 4th-order tensor $\mathcal{X} \in \mathbb{R}^{a \times b \times M \times N}$ , where:

$a, b$ : Dimensions of the LLM backbone weights.
$M$ : Number of distinct scenes.
$N$ : Number of distinct environments.

This tensor is decomposed using Tucker Decomposition:
$\mathcal{X} = \mathcal{G} \times_1 U_1 \times_2 U_2 \times_3 U_3 \times_4 U_4$

$\mathcal{G}$ (Core Tensor): Represents shared navigation knowledge (common skills across all tasks).
$U_1, U_2$ (Encoder/Decoder): Shared transformation matrices that align the high-order tensor with the 2D LLM backbone.
$U_3$ (Scene Experts): A set of matrices where each row $U_3[s, :]$ represents knowledge specific to scene $s$ .
$U_4$ (Environment Experts): A set of matrices where each row $U_4[e, :]$ represents knowledge specific to environment $e$ .

For a specific task $t$ (scene $s$ , environment $e$ ), the adaptation weight is reconstructed by selecting the specific experts and contracting them with the core tensor:
$\Delta W_t = U_1 \cdot (\mathcal{G} \times_3 U_3[s, :] \times_4 U_4[e, :]) \cdot U_2^T$

B. Decoupled Knowledge Incremental Learning (DKIL)

To train this architecture continuously without forgetting, the authors introduce DKIL, which manages the learning process through three mechanisms:

Shared Knowledge Consolidation: The core tensor $\mathcal{G}$ and shared encoders/decoders ( $U_1, U_2$ ) are updated using Elastic Weight Consolidation (EWC). This constrains changes to parameters critical for previous tasks, preserving shared knowledge.
Expert Consistency: When revisiting a known scene or environment, the corresponding expert rows ( $U_3[s, :], U_4[e, :]$ ) are initialized from previous values and constrained via a consistency loss to prevent drift.
Orthogonal Exploration: For new tasks, the specific expert rows are optimized to be orthogonal to previously learned experts. This ensures that new task-specific knowledge is stored in a distinct subspace, minimizing interference with existing knowledge.

C. Inference Strategy

During inference, the agent does not require a task ID. Instead, it uses a retrieval mechanism:

The agent extracts visual features from the current observation using a CLIP encoder.
It computes cosine similarity against stored feature sets for all known scenes and environments.
The most similar scene and environment indices are selected to retrieve the corresponding expert rows ( $U_3[s, :], U_4[e, :]$ ) for weight reconstruction.

3. Key Contributions

Problem Formalization: Defined the AML-VLN problem, explicitly modeling the challenge of lifelong learning across the Cartesian product of scenes and environments.
TuKA Architecture: Proposed a novel Tucker Decomposition-based adapter that decouples shared, scene-specific, and environment-specific knowledge into a high-order tensor, overcoming the limitations of 2D matrix-based LoRA.
DKIL Strategy: Developed a training strategy that combines EWC for shared knowledge and orthogonal constraints for specific experts to mitigate catastrophic forgetting.
Benchmark & Agent:
- Created AlldayWalker, a lifelong VLN agent implementing TuKA.
- Extended the Habitat simulator with All-Day-Habitat, a benchmark featuring 24 tasks (5 scenes $\times$ 4 environments including low-light, scattering, overexposure) and real-world deployments.

4. Experimental Results

The authors evaluated AlldayWalker against state-of-the-art baselines (Seq-FT, LwF-LoRA, EWC-LoRA, Dense/Sparse MoLE, HydraLoRA, BranchLoRA, SD-LoRA) on the AML-VLN benchmark.

Performance: AlldayWalker achieved a Success Rate (SR) of 65% on average across 24 tasks, significantly outperforming the best baseline (SD-LoRA at 56%) and MoE variants.
Forgetting Rate: It demonstrated the lowest Forgetting Rate (F-SR) at 11%, compared to 18% for SD-LoRA and much higher rates for others (e.g., 87% for Seq-FT). This indicates superior retention of past knowledge.
Ablation Studies:
- Tensor Order: 4th-order tensors (decoupled scene/environment) significantly outperformed 3rd-order tensors (coupled), proving the necessity of explicit decoupling.
- Shared Components: Removing the shared core tensor or encoder led to performance drops, confirming the value of shared knowledge representation.
- Generalization: The model generalized well to unseen scenarios (new scenes/environments), achieving 55% SR compared to 39-40% for baselines.
Real-World Validation: The agent was successfully deployed on a quadruped robot (DeepRobotDog) in real-world environments, validating its robustness beyond simulation.

5. Significance

This paper represents a significant step forward in embodied AI and continual learning:

Theoretical Advancement: It demonstrates that high-order tensor representations are superior to matrix-based approaches for modeling complex, multi-faceted knowledge structures in robotics.
Practical Deployment: By solving the catastrophic forgetting problem in diverse environmental conditions, it enables VLN agents to operate "all-day" in real-world settings without requiring retraining for every new lighting condition or location.
Efficiency: TuKA achieves these gains with a parameter count comparable to standard LoRA (~0.3M trainable parameters), making it scalable for large language model backbones.

In summary, TuKA provides a principled framework for disentangling and learning multi-hierarchical knowledge, enabling robust, lifelong navigation agents capable of adapting to the complexity of the real world.