⚡ electrical engineering

Seq-DeepIPC: Sequential Sensing for End-to-End Control in Legged Robot Navigation

This paper introduces Seq-DeepIPC, a sequential end-to-end perception-to-control model for legged robot navigation that integrates multi-modal RGB-D and GNSS data with temporal fusion to achieve robust, real-time performance on edge devices across diverse terrains.

Original authors: Oskar Natan, Jun Miura

Published 2026-06-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Oskar Natan, Jun Miura

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a robot dog trying to navigate a campus that has smooth sidewalks, bumpy grass, and even a set of stairs. For a robot, this is like a human trying to walk through a crowded market while wearing blinders and a compass that spins wildly whenever they get near a building.

The paper introduces Seq-DeepIPC, a new "brain" for robot dogs that solves three major problems: shaky vision, confusing direction, and slow thinking.

Here is how it works, broken down into simple concepts:

1. The "Shaky Camera" Problem (Sequential Sensing)

The Analogy: Imagine trying to read a street sign while riding a bumpy horse. If you look at just one split-second photo, the sign might be blurry or cut off. But if you look at a short video clip, your brain can piece together the blurry parts to understand what the sign says.

The Solution: Older robot models looked at the world one frozen frame at a time. Because robot dogs bounce up and down when they walk, their cameras shake, making single frames look messy.
Seq-DeepIPC doesn't just look at one photo; it looks at a short video clip (3 frames) at a time. It uses a special memory unit (called a GRU) to "stitch" these frames together. This smooths out the camera shake, allowing the robot to see the road clearly even while it's bouncing. The paper found that this "video clip" approach worked much better for robot dogs than for wheeled robots, which don't bounce as much.

2. The "Confused Compass" Problem (Magnetometer-Free Heading)

The Analogy: Imagine trying to use a magnetic compass inside a house full of microwaves and steel beams. The needle would spin wildly, pointing you in the wrong direction. This is what happens to robot compasses near tall buildings.

The Solution: Most robots use a magnetic sensor (like a compass) to know which way is "North." The authors realized this sensor is too noisy in cities. Instead, they taught the robot to figure out its direction by looking at two GPS points from a second ago and a second ago.
Think of it like this: If you know where you were 5 seconds ago and where you are now, you can draw a straight line to know which way you are facing. By doing the math on these GPS points, the robot gets a "North" that doesn't get confused by metal buildings. It's not perfect near skyscrapers (where GPS signals get blocked), but in open areas, it's much more reliable than a spinning magnetic needle.

3. The "Two-Brain" Problem (Multi-Task Learning)

The Analogy: Imagine a student taking a driving test. If they only study "how to steer," they might crash into a tree. But if they study "how to steer" AND "how to judge distance," they become a much safer driver.

The Solution: The robot's brain is trained to do two things at once:

Identify objects: "Is that grass? Is that a road? Is that a wall?" (Semantic Segmentation).
Judge distance: "How far away is that wall?" (Depth Estimation).

By forcing the robot to learn both at the same time, the "brain" becomes smarter at understanding the 3D shape of the world. The paper shows that even though this robot uses a smaller, lighter computer chip (to save battery and weight), it performs just as well as much heavier, more complex systems because it is learning these two skills together.

4. The "Bird's-Eye View" (BEV Projection)

The Analogy: Imagine looking at a map from a drone. You can see the whole path ahead clearly. Now imagine looking at a map from the ground; you can only see what's directly in front of your nose.

The Solution: The robot takes its camera view and mathematically flattens it into a Bird's-Eye View (BEV) map. This helps the robot plan its steps like a chess player looking at the whole board, rather than just the square right in front of them.

The Results: What Actually Happened?

The researchers tested this on a real robot dog (a Unitree Go2) on a university campus.

Success: The robot successfully walked on asphalt roads, climbed grassy hills, and even navigated up a set of stairs embedded in the grass. The "video clip" method kept the robot from stumbling when the camera shook.
Failure: The robot got confused near tall buildings. Because the GPS signal got blocked by the buildings, the robot couldn't calculate its direction correctly. The paper notes that the robot's vision was fine, but its sense of direction was lost due to the GPS signal issues.

Summary

Seq-DeepIPC is a new way to teach robot dogs to walk. Instead of looking at the world in single, shaky snapshots with a faulty compass, it watches short video clips to smooth out the bumps and uses GPS math to find its direction. This allows a small, lightweight robot to navigate complex, bumpy terrain like stairs and grass much better than previous models.

Technical Summary: Seq-DeepIPC: Sequential Sensing for End-to-End Control in Legged Robot Navigation

Problem Statement
Modern robotic navigation increasingly relies on end-to-end learning architectures that map raw sensory data directly to control commands. However, deploying these systems on legged robots in unstructured, real-world environments presents significant challenges. Existing approaches often suffer from: (1) Temporal Inconsistency, where single-frame perception is susceptible to aliasing and noise, particularly given the high-frequency camera pitch oscillations inherent to legged gaits; (2) Heading Instability, where reliance on IMU-based magnetometers introduces drift due to magnetic interference in urban settings; and (3) Limited Generalization, as most validated systems are designed for wheeled robots on structured roads, failing to generalize to mixed terrains like grass, stairs, and uneven surfaces.

Methodology
The authors propose Seq-DeepIPC, a sequential, multi-task end-to-end perception-to-control framework designed specifically for legged robot navigation. The architecture consists of two primary components:

Perception and BEV Construction:
- Input: The model ingests short sequences ( $K$ ) of RGB-D frames.
- Encoder: A lightweight EfficientNet-B0 encoder processes the sequential RGB inputs to extract latent features.
- Multi-Task Heads: The encoder drives two prediction heads: semantic segmentation (19 classes) and depth estimation. The depth prediction acts as an auxiliary task to regularize the encoder and enforce geometry-aware features.
- BEV Projection: Ground-truth depth maps are combined with predicted segmentation to generate Bird's-Eye-View (BEV) semantic maps. These maps are encoded by a second EfficientNet-B0.
- Temporal Fusion: A Gated Recurrent Unit (GRU) fuses the RGB latent features and the BEV latent features over time. This temporal window is empirically tuned to mitigate camera jitter caused by the robot's gait, stabilizing the BEV projection without mechanical stabilization.
Planning and Control:
- Heading Estimation: To eliminate IMU drift, the system derives global heading solely from differential GNSS coordinates using a geodesic formula, discarding the magnetometer.
- Control Policy: The fused latent state, concatenated with transformed route points and velocity, drives two parallel control pathways:
  - A PID-based pathway that predicts future waypoints to generate lateral and longitudinal control signals.
  - A Command-specific MLP pathway that infers control actions directly from the latent state based on discrete commands (straight, left, right).
- Blending: A confidence-gating mechanism blends the outputs of the PID and MLP controllers to produce the final position-orientation control action $(x, y, \theta)$ .

Key Contributions
The paper identifies three primary novelties:

Locomotion-Aware Temporal Perception: A framework integrating sequential RGB-D inputs via a GRU, specifically tuned to stabilize BEV projections against the high-frequency oscillations of legged robot gaits.
Magnetometer-Free Global Heading: A method for deriving bearing solely from differential GNSS fixes, effectively eliminating magnetic drift common in urban environments.
Comprehensive Legged Evaluation: A validation of the system on a Unitree Go2 legged robot across mixed terrains (asphalt, grass, stairs), demonstrating that sequential fusion significantly outperforms single-frame baselines in handling chaotic camera motion.

Results
The authors evaluated Seq-DeepIPC against three baselines (Huang et al., AIM-MT, and the previous DeepIPC model) using a new dataset collected on a university campus.

Quantitative Performance: In offline tests, Seq-DeepIPC achieved competitive or superior results in segmentation IoU, depth MAE, waypoint MAE, and control MAE compared to baselines. Notably, increasing the sequence length ( $K$ ) from 1 to 3 yielded monotonic improvements for Seq-DeepIPC and the DeepIPC baseline, while non-recurrent models (Huang, AIM-MT) showed no benefit or degradation from sequential inputs.
Model Efficiency: Seq-DeepIPC utilizes a significantly lighter encoder (EfficientNet-B0) compared to baselines (ResNet-50, EfficientNet-B3), resulting in a smaller model size (47.5 MB) while maintaining high accuracy.
Qualitative Deployment: Online tests on the legged robot demonstrated successful traversal of roads, grass, and stairs. The temporal fusion successfully smoothed waypoint transitions and control commands.
Limitations: The system exhibited navigation drift near tall buildings due to GNSS signal blockage and multipath interference affecting the bearing estimation. However, visual perception remained robust, isolating the error source to localization rather than the neural network.

Significance
The paper claims that Seq-DeepIPC extends the scope of end-to-end navigation beyond wheeled robots to versatile, temporally-aware legged systems. By tightly integrating multi-modal perception (RGB-D + GNSS) with temporal fusion and multi-task learning, the framework addresses the specific challenges of legged locomotion in unstructured environments. The authors emphasize that their approach bridges the gap between intelligent sensing and autonomous decision-making, providing a robust, lightweight solution for real-world deployment. The work suggests that while GNSS-only heading has limitations in dense urban canyons, the system is highly effective in open areas and mixed terrains, offering a practical pathway for legged robot autonomy.