🤖 AI

CoFL-S: Spatially Queryable Sector Flow Fields for Local Language-Conditioned Navigation

The paper proposes CoFL-S, a low-level vision-language-action framework that predicts language-conditioned flow fields for continuous trajectory generation, demonstrating superior performance over existing baselines in both a new continuous-time Habitat benchmark and zero-shot real-world deployment.

Original authors: Haokun Liu, Zhaoqi Ma, Yicheng Chen, Wentao Zhang, Masaki Kitagawa, Zicen Xiong, Jinjie Li, Moju Zhao

Published 2026-07-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Haokun Liu, Zhaoqi Ma, Yicheng Chen, Wentao Zhang, Masaki Kitagawa, Zicen Xiong, Jinjie Li, Moju Zhao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to navigate a house using only spoken instructions like, "Walk through the first door on the right." Most current robots are like students who memorize a strict list of steps: "Step forward, turn left, stop." If the floor is slightly different or the robot is a tiny bit late in turning, they get confused and crash.

This paper introduces CoFL-S, a new way to teach robots how to move. Instead of giving them a list of steps, CoFL-S gives them a living, breathing map of "pushes" and "pulls" based on what they see and hear.

Here is a simple breakdown of how it works, using everyday analogies:

1. The Old Way vs. The New Way

The Old Way (Action Tokens/Chunks): Imagine a robot that only knows how to say, "I will turn left now," or "I will walk forward for 3 seconds." It's like a driver who only knows how to press the gas or the brake at specific moments. If the road curves unexpectedly, the driver is stuck because they only planned for the next second.
The New Way (CoFL-S): Imagine the robot doesn't just plan a single step; it creates a wind map over the floor in front of it.
- If the instruction is "Go to the chair," the map has a gentle wind blowing toward the chair.
- If there is a wall, the map has a strong wind pushing the robot away from the wall.
- The robot doesn't just pick one direction; it feels the wind everywhere around it. If it drifts slightly off course, the "wind" naturally pushes it back on track without needing a new command.

2. How It Sees the World

The robot uses a camera (RGB) and a depth sensor (like a 3D eye) to see the room. It also listens to a short, local instruction (e.g., "Go past the sofa").

The "Sector Flow Field": Think of the robot's view as a slice of a pie (a sector). CoFL-S paints a colorful flow field over this slice.
- Blue arrows might point toward the goal.
- Red arrows might point away from obstacles.
- The robot queries this map: "If I am standing here, which way does the wind blow?" It then moves in that direction.

3. Training the Robot

To teach this robot, the researchers didn't just show it a finished video of a person walking. They broke the video down frame-by-frame.

The Analogy: Imagine watching a movie of someone walking to a door. Instead of just saying "They walked to the door," the researchers paused the movie at every single second. They asked, "At this exact second, if the person wanted to go to the door, which way should they be pushed?"
They created millions of these "micro-moments," pairing the robot's view with a tiny instruction and the correct "wind map" for that split second. This taught the robot to understand not just where to go, but how to move smoothly around furniture.

4. The Results: Simulation and Real Life

The researchers tested this in two ways:

In the Computer (Habitat): They put the robot in a virtual world. They compared CoFL-S against robots that use the old "step-by-step" method. CoFL-S won consistently. It was better at reaching the goal, smoother in its movements, and got stuck less often. It worked well whether the robot checked its map 2 times a second or 10 times a second.
In the Real World: They took the robot trained in the computer and put it in a real building (offices and outdoors) without teaching it anything new.
- The Result: The robot successfully navigated real hallways, avoided real chairs, and followed human instructions.
- Why it worked: The "wind map" approach was robust. Even if the robot was slightly delayed in processing the image (a common problem with real computers), the map still provided a safe path. The old "step-by-step" robots tended to oscillate (wiggle back and forth) or crash into walls because they couldn't adapt quickly enough.

The Bottom Line

CoFL-S changes the robot's brain from a step-counter to a feeler. Instead of memorizing a rigid sequence of moves, it learns to sense the "flow" of the environment. This allows it to navigate smoothly and safely, even when things aren't exactly as expected, making it much better at following language instructions in the real world.

Technical Summary: CoFL-S

Problem Statement

Vision-Language Navigation (VLN) has seen significant progress in high-level semantic reasoning, memory construction, and instruction decomposition. However, the low-level action representation—the mechanism that translates local semantic intent into executable robot motion—remains under-explored. Standard VLN evaluations often entangle high-level reasoning with low-level execution, making it difficult to isolate the effectiveness of different action interfaces. Existing interfaces, such as discrete action tokens (e.g., "turn left") or finite-horizon action chunks, specify actions for the current state or a short future sequence but fail to explicitly model how desired motion should vary across nearby workspace locations, obstacle configurations, or replanning frequencies. The paper posits that a low-level interface capable of representing spatially indexed guidance, similar to classical potential fields, is necessary for robust, continuous control in partially observable environments.

Methodology: CoFL-S Framework

The authors propose CoFL-S (CoFL-Spatially Queryable Sector Flow Fields), a framework that predicts a language-conditioned flow field over the robot's local visible sector to generate continuous trajectories.

1. Sector Flow Field Representation

Instead of regressing a single trajectory or a discrete action, CoFL-S predicts a 2D flow field $v_\phi$ over the robot's local ground-plane sector.

Input: An ego-centric RGB-D observation $(I, D)$ and a low-level local sub-instruction $\ell$ .
Output: A spatially queryable velocity field. The field is queried in normalized polar sector coordinates $q = (\tilde{\theta}, \tilde{r})$ aligned with the camera's field of view, but outputs Cartesian velocities $(v_{fwd}, v_{lft})$ in the robot-centric ground-plane frame.
Generation: The trajectory $\tau(t)$ is generated by rolling out the predicted field using a first-order motion model: $\dot{\tau}(t) = v_\phi(\Pi_{polar}(\tau(t)) | I, D, \ell)$ .

2. Architecture

The model consists of three main components (Fig. 2):

Vision-Language Encoder: Uses a frozen SigLIP2 encoder to map RGB and text to shared embeddings. Depth information is injected into visual tokens via a convolutional patchifier and a FiLM (Feature-wise Linear Modulation) pathway to provide geometric awareness.
Sector Decoder: A query-based Transformer decoder that takes coordinate queries (embedded via Gaussian Fourier features) and the fused context to predict velocity vectors for each sector location.
Action Module: A lightweight auxiliary branch using learnable action queries to predict a discrete termination probability (STOP), complementing the continuous field rollout.

3. Continuous-Control Supervision

To train this representation, the authors convert standard VLN-CE episodes (R2R and RxR) into frame-level local supervision:

Fine-Grained Alignment: Each RGB-D frame is paired with a local sub-instruction derived from fine-grained annotations (Fine-Grained R2R, Landmark RxR).
Instruction Expansion: To increase diversity, each frame is expanded into $K$ instruction slots, including the human anchor, object-grounded commands, region-grounded commands, and terminal commands.
Target Generation: For each instruction slot, a dense sector flow field target is generated by combining geodesic attraction toward a grounded goal with obstacle/invisible-region repulsion. This creates a potential field whose negative gradient serves as the target velocity field.
Loss Function: The model is trained with a combination of direction loss (cosine similarity), magnitude loss (distance-to-go scaling), and action termination loss.

4. Evaluation Benchmark

The authors introduce a continuous-time Habitat benchmark to isolate low-level action interfaces:

Protocol: All methods (CoFL-S, Action Token, Action Chunk) are evaluated under the same continuous-time simulation where the agent state updates at 60 Hz and observations at 30 Hz.
Controller: A shared low-level velocity controller executes the outputs of all methods, ensuring that performance differences arise from the action representation rather than the controller.
Decomposition Independence: At each step, the agent's position is projected onto the reference path to retrieve the corresponding local sub-instruction, removing the variable of high-level instruction decomposition.

Key Contributions

CoFL-S Framework: A novel sector flow-field representation for first-person, language-conditioned local navigation that maps observations and instructions to a spatially queryable control field.
Supervision Conversion: A method to convert whole-episode VLN-CE traces into frame-level local supervision, pairing observations with aligned sub-instructions and dense flow-field targets.
Continuous-Time Benchmark: A new evaluation protocol that compares action-token, action-chunk, and flow-field interfaces under matched encoders, instructions, and controllers, enabling decomposition-independent comparison across planner frequencies.

Experimental Results

The paper evaluates CoFL-S against Action Token and Action Chunk baselines in both simulation and real-world zero-shot settings.

Simulation (R2R-CE and RxR-CE): CoFL-S consistently outperforms both baselines across all tested planner frequencies (2 Hz, 5 Hz, 10 Hz) in terms of Navigation Error (NE), Oracle Success (OS), Success Rate (SR), and Success weighted by Path Length (SPL). Notably, it achieves higher task completion rates than Action Chunk despite the latter having a larger prediction head, and maintains competitive Blocked-Step Rate (BSR) and Heading Smoothness (HS) metrics.
Real-World Zero-Shot: Deployed on a physical mobile robot without fine-tuning, CoFL-S achieved a Success Rate of 0.75 compared to 0.46 for Action Token and 0.33 for Action Chunk. It demonstrated fewer collisions, shorter completion times, and more stable navigation, particularly in narrow passages where discrete baselines tended to oscillate or get stuck.

Significance and Claims

The paper claims that the low-level action representation is a critical, often overlooked component of VLN. By shifting from discrete or finite-horizon action predictions to a spatially queryable flow field, CoFL-S provides a "family of corrective motion targets" rather than a single current-state decision. This dense workspace-level supervision allows the policy to better couple local instruction semantics with geometric constraints, resulting in more robust continuous control.

The authors emphasize that while CoFL-S assumes the availability of a local sub-instruction (relying on a high-level planner for decomposition), it effectively solves the execution layer problem. The results suggest that flow-field policies with dense supervision offer superior geometric awareness and stability compared to traditional action interfaces, particularly in zero-shot real-world deployments where latency and environmental variability are significant factors.