💻 computer science

Beyond Transformers: Linear Attention Policy for Open-Vocabulary Object Goal Navigation

This paper introduces LANav, a navigation policy that replaces standard Transformer-based self-attention with a structured linear attention mechanism—specifically enhanced by Weighted State-Expansion Linear Attention (WSLA)—to achieve superior open-vocabulary object goal navigation performance, computational efficiency, and robust sim-to-real transfer compared to existing baselines.

Original authors: Jiahong Zhang, Yifan Lin, Yandong Zhang, Sijun Shen, Kexin Wang, Yuqi Pan, Hongjuan Pei, Wei Wang, Guoqi Li

Published 2026-07-22

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiahong Zhang, Yifan Lin, Yandong Zhang, Sijun Shen, Kexin Wang, Yuqi Pan, Hongjuan Pei, Wei Wang, Guoqi Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find a specific object, like a "vintage toaster," in a house you've never seen before. You can only see what's directly in front of you through a narrow window, and you have to remember where you've been, what you've seen, and what you're looking for to make your next move. This is the daily life of a robot navigating the world, a field of science called Embodied AI. To do this, robots use a "brain" (a policy network) that acts like a short-term memory, constantly updating its internal state based on new sights and sounds. For a long time, scientists tried two main ways to build this memory: one that compresses history into a single, shrinking summary (like an old-school notebook), and another that keeps a long list of everything seen recently and re-reads the whole list every time to find connections (like a super-organized but slow librarian). The big question was: which method helps a robot find its way best when the object it's looking for might have a weird name or be in a completely new house?

This paper introduces a new approach called LANav (Linear Attention-based Navigation) and a special upgrade called WSLA. The researchers discovered that the "super-librarian" method (using Transformers with self-attention), which was thought to be the state-of-the-art, actually hits a wall when the robot needs to remember a long journey. Surprisingly, making the robot look at a longer history didn't help the Transformer get better; in fact, it sometimes got worse. Instead, the team found that a method called Linear Attention, which updates its memory like a steady, structured stream rather than re-reading a whole list, works much better. They took this idea and supercharged it with WSLA, which splits the memory into multiple "sub-streams" and learns how much weight to give each one. In tests, this new system found objects 36.4% of the time in a challenging simulation, beating the best Transformer models by a clear margin. Even cooler, they tested it on a real robot dog in a real room, and it succeeded 82% of the time, proving that this "streaming memory" idea works not just in computers, but in the real world.

The Problem: The Robot's Memory Crisis

Imagine you are walking through a giant, unfamiliar maze looking for a "blue chair." You can only see a few steps ahead. Every time you turn a corner, you see something new, but you also forget what you saw ten seconds ago. To find the chair, your brain needs to hold onto clues: "I passed a red door," "I heard a fan," "I turned left twice."

For years, robot scientists tried to solve this with two main types of memory:

The Compressor (RNNs): These are like a robot that squishes all its past memories into one tiny, dense ball. It's fast, but as the journey gets longer, the ball gets so small and messy that the robot forgets the important details.
The Re-Reader (Transformers): These are like a robot that keeps a perfect, long scroll of everything it has seen. Every time it needs to decide where to go next, it re-reads the entire scroll from start to finish to find connections. This is powerful, but it's slow and gets messy if the scroll gets too long.

The researchers asked: "If we give the robot a longer scroll (a longer history), will it get better at finding things?" They tested this with the "Re-Reader" (Transformer) approach on a task called Open-Vocabulary Object Goal Navigation. This is a fancy way of saying: "Find an object I describe to you, even if I use a weird name you've never heard before, like 'a thing that holds soup' instead of 'a pot'."

The Surprise: More History Didn't Help

The team ran a series of controlled experiments. They kept everything the same—the robot's eyes, the maps, the training rules—and only changed the memory system. They expected that if they gave the Transformer robot a longer scroll to read, it would find the objects more often.

But here is the twist: It didn't work.

When they increased the length of the history the Transformer could see, the robot's performance didn't improve. In fact, in some cases, it got slightly worse. It was as if the robot was drowning in its own memories, unable to figure out which parts of the long scroll actually mattered. The "Re-Reader" method seemed to hit a ceiling. It turns out, for a robot wandering a maze, constantly re-reading the whole history isn't the best way to update its state.

The Solution: The "Streaming" Memory

The researchers then tried a different approach: Linear Attention. Instead of re-reading the whole scroll, imagine a robot that has a "streaming memory." As it walks, it updates its internal state step-by-step, like a river flowing. It doesn't look back at the whole river; it just updates the water level based on the new rain (new observation) and the current flow.

They built a system called LANav using this streaming method. The results were immediate and impressive.

Better than the old ways: LANav beat both the "Compressor" (RNN) and the "Re-Reader" (Transformer) models.
Longer is better: Unlike the Transformer, when they gave the LANav robot a longer history to learn from, it actually got better. The longer the training history, the smarter the robot became.

The Upgrade: WSLA (The Multi-Stream Brain)

The researchers didn't stop there. They realized that even a streaming memory could be improved. They asked, "What if our robot didn't just have one stream of memory, but several, and could learn which stream to listen to?"

They created WSLA (Weighted State-Expansion Linear Attention).

The Analogy: Imagine the robot's brain has four different "notebooks" (sub-states) instead of one. One notebook tracks colors, another tracks shapes, another tracks turns, and another tracks sounds.
The Magic: A special "conductor" (learnable weighting) decides how much to listen to each notebook at any given moment. If the robot is looking for a "red box," the conductor might crank up the volume on the "color" notebook and turn down the "sound" notebook.

This WSLA system turned out to be the champion.

In the HM3D-OVON simulation (a massive, realistic 3D house dataset), the WSLA robot achieved a 36.4% success rate.
The best Transformer model only got 30.1%.
That's a 6.3 percentage point improvement, which is huge in this field.

Why It Matters: Distance and Real Life

The researchers also looked at how the robot navigated. They found that the new system was especially good at long journeys.

Short trips: Both robots did okay.
Long trips: The Transformer robot often got lost or gave up early. The WSLA robot, however, kept its cool. It successfully navigated distances of 15 meters or more with a success rate of 14.36%, while the Transformer only managed 6.68%.

But the real test was taking it out of the computer and into the real world. The team put their LANav system on a Unitree Go2 robot (a real, physical robot dog). They asked it to find things like a trash bin, a plant, or a chair in a real room.

They ran 50 trials.
The robot succeeded 41 times.
That's an 82% success rate in the real world!

This proves that the "streaming memory" idea isn't just a cool computer trick; it actually works for robots moving through real spaces, handling the messiness of real life.

The Takeaway

The paper suggests that for robots trying to find their way in complex, unknown environments, the old "re-read everything" method (Transformers) might not be the best tool. Instead, a method that updates its memory in a structured, streaming way (Linear Attention), especially when enhanced with multiple specialized sub-memories (WSLA), is far more effective. It's faster, scales better with longer journeys, and, most importantly, it actually works when you put it on a robot dog in a real room. The future of robot navigation might not be about remembering more history, but about remembering it better.

Technical Summary: Beyond Transformers: Linear Attention Policy for Open-Vocabulary Object Goal Navigation

Problem Formulation

The paper addresses Open-Vocabulary Object Goal Navigation (OVON), a task where a mobile agent must explore an unseen indoor environment to locate an object specified by a language description. Unlike closed-set navigation, the target category may be unseen during training or described by novel names and synonyms. The task operates under partial observability, where the agent receives only egocentric RGB observations and a language goal, without explicit pose or ego-motion data.

The core challenge lies in state integration: the policy must maintain an internal state that summarizes past observations to make decisions over long horizons. While recent approaches utilize Transformer-based backbones with self-attention over a context window to integrate temporal information, the authors' controlled experiments reveal a critical limitation: performance does not scale with context length under Transformer-based policies. This suggests that standard self-attention may be unsuitable for the specific state integration requirements of navigation, particularly when dealing with redundant observations and sparse goal-relevant cues.

Methodology

The authors propose Linear Attention-based Navigation (LANav), a framework that replaces the self-attention mechanism in Transformer policies with Linear Attention (LA). Instead of computing pairwise interactions over a fixed context window (which incurs $O(l^2)$ complexity), LA maintains a structured, recurrent internal state that is updated incrementally as new observations arrive, achieving $O(l)$ complexity.

Key Architectural Components

Policy Backbone (LANav):
- The architecture follows the DAgRL design, separating visual encoding (frozen SigLIP), goal encoding (frozen SigLIP), and action embedding.
- The temporal backbone processes a sequence of per-step embeddings.
- Linear Attention Mechanism: Each attention head maintains a state matrix $S_t$ independent of the context length. At each step, the state is updated using the current key-value pair, and the output is computed by reading from this updated state.
- The paper evaluates several LA variants: Linear Transformer, DeltaNet, and Gated DeltaNet, finding that more structured and regulated state update rules yield better performance.
Weighted State-Expansion Linear Attention (WSLA):
To further enhance state expressivity, the authors introduce WSLA, which improves upon standard LA through two mechanisms:
- State Expansion: Each attention head is expanded into $E$ independent sub-heads. The query and key vectors are projected into these sub-heads, while the value vector is shared. Each sub-head maintains its own recurrent state.
- Learnable Weighted Readout: A learnable logit vector $\lambda$ is used to compute softmax weights for aggregating the outputs of the expanded sub-heads. This allows the model to regulate the contribution of different sub-states based on their relevance to the current navigation decision.
- The final output is a gated, RMS-normalized combination of these weighted sub-head outputs.

Key Contributions

Introduction of LANav: The paper introduces LA as a policy backbone for OVON. Through controlled comparisons with matched architectures and training settings, it demonstrates that LA backbones consistently outperform both RNN (e.g., GRU) and Transformer baselines.
Empirical Analysis of State Updates: The study reveals three key findings:
- LA-based policies benefit from larger training context lengths, whereas Transformer-based policies do not.
- Performance improves as state update mechanisms become more structured and regulated (Linear Transformer < DeltaNet < Gated DeltaNet < WSLA).
- The design of the state update mechanism is more critical than simply increasing context length.
Proposal of WSLA: The authors propose WSLA to improve state update effectiveness. It achieves a 36.4% average Success Rate (SR) on the HM3D-OVON benchmark, outperforming the strongest Transformer-based baseline (30.1%) by 6.3 percentage points while maintaining computational efficiency.

Experimental Results

The paper evaluates methods on the HM3D-OVON benchmark (Habitat-Matterport3D) and the HSSD dataset for generalization.

HM3D-OVON Performance:
- WSLA achieves an average SR of 36.4% and an SR of 39.8% on the "VAL SEEN SYN" (unseen but semantically similar) split, significantly outperforming Transformers.
- Context Length Scaling: While Transformer performance degrades or stagnates as training context length increases, LA-based models show consistent improvement with longer training contexts ( $l_{train} = 500$ ).
- Distance-Stratified Analysis: WSLA shows larger gains over Transformers in long-distance episodes (15m+), with an SR improvement of +7.68% compared to +6.71% for short distances. This indicates WSLA is better at sustaining exploration over extended horizons.
Generalization (HSSD):
- In zero-shot transfer to HSSD, WSLA improves SR by 9.62% over the Transformer baseline.
- After fine-tuning on HSSD, the gap widens, with WSLA achieving 64.50% SR vs. 56.17% for the Transformer, demonstrating robustness to dataset shifts.
Efficiency:
- Computational Cost: Transformer FLOPs per action scale quadratically with context length (rising from 3.52G at $l=100$ to 42.53G at $l=1000$ ). In contrast, WSLA maintains a nearly constant cost of ~0.048G FLOPs/action regardless of context length.
- Parameter Count: WSLA incurs only a moderate increase in parameters compared to other LA variants due to state expansion.
Real-World Deployment:
- LANav with WSLA was deployed on a Unitree Go2 robot. Across 50 trials involving 5 object categories, the system achieved an 82% success rate, validating the sim-to-real transfer capability.

Significance and Claims

The paper claims that the primary bottleneck in current OVON policies is not the lack of context, but the inefficiency of the state update mechanism used to integrate that context.

Beyond Self-Attention: The work challenges the prevailing assumption that self-attention is the optimal mechanism for temporal integration in embodied navigation. It posits that the quadratic complexity and lack of structured state updates in Transformers hinder performance on long-horizon tasks.
Structured State Updates: The authors argue that the success of LA, and specifically WSLA, stems from its ability to perform structured, regulated state updates. The introduction of sub-states and weighted readout allows the policy to better manage the trade-off between retaining useful historical evidence and filtering out irrelevant or redundant observations.
Scalability and Efficiency: By decoupling computational cost from context length, LANav offers a scalable solution for long-horizon navigation that is both more effective and computationally cheaper than Transformer-based approaches.

The paper concludes that while current implementations use shallow backbones and bounded windows, the proposed direction of using linear attention with structured state updates represents a promising design paradigm for future embodied intelligence policies.