💻 computer science

InFeR: Informed Failure Resilience in Learned Visual Navigation Control

The paper proposes InFeR, a general framework that enhances imitation learning policies for visual navigation by using a Variational Information Bottleneck for out-of-distribution failure detection and Grad-CAM for localizing failure sources to enable autonomous recovery, all without requiring additional failure or recovery training data.

Original authors: Zishuo Wang, Joel Loo, David Hsu

Published 2026-05-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zishuo Wang, Joel Loo, David Hsu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to walk through a house by showing it a video of a human doing it perfectly. This is called Imitation Learning. The robot watches the video, learns the pattern, and tries to copy it.

However, there's a big problem: What happens when the robot encounters something it hasn't seen before?

Maybe a cat suddenly runs in front of it, or a door is closed that was open in the training video, or the camera gets covered in black paint. In these "out-of-distribution" (OOD) moments, a standard robot policy just guesses blindly. It might keep walking into the cat, crash into the door, or spin in circles, often without the robot even realizing it's in trouble.

InFeR is a new framework designed to fix this. Think of it as giving the robot a "sixth sense" and a "recovery plan" without needing to show it any videos of crashes or failures.

Here is how InFeR works, broken down into simple steps:

1. The "Stress Test" Training (VIB)

Usually, robots are trained only on "good" days. InFeR changes how the robot thinks during training. It uses a technique called a Variational Information Bottleneck (VIB).

The Analogy: Imagine you are studying for a test. A normal student memorizes the exact answers. A student using InFeR is taught to understand the core concepts so well that they can instantly tell if a question is "weird" or "nonsense" compared to what they studied.
How it works: InFeR forces the robot to compress its visual information into a very specific, clean mental map. If the robot sees something strange (like a black screen or a sudden obstacle), this mental map gets "scrambled" or doesn't fit the pattern. The robot immediately knows, "Hey, this doesn't look like the training data!"

2. The "Spotlight" (Grad-CAM)

Once the robot knows something is wrong, it needs to know what is wrong. Is it a cat? A wall? A broken camera?

The Analogy: Imagine the robot is looking at a messy room. Instead of just saying "I'm confused," InFeR puts a glowing red spotlight on the specific object causing the confusion.
How it works: The system uses a tool called Grad-CAM to look at the robot's internal "thoughts" and highlight the exact part of the image that is causing the problem. If a chair is blocking the path, the robot highlights the chair. If the camera is covered, it highlights the whole black screen.

3. The "Recovery Plan" (Heuristic Policy)

Now that the robot knows it's in trouble and knows where the trouble is, it needs to fix it. Since the robot was never shown videos of how to recover from a crash, it can't just "learn" a new trick. Instead, InFeR gives it a simple, smart set of rules.

The Analogy: Think of this like a "Get Out of Jail Free" card with a few specific moves.
- If the spotlight is on the left: The robot knows to turn right to get away from the obstacle.
- If the spotlight is on the right: The robot turns left.
- If the robot is stuck in a corner: It tries to back up.
- If the camera is broken (black screen): It knows it can't fix this alone and stops, waiting for a human to help.

The Results

The researchers tested this on a real robot (a Boston Dynamics Spot) in the real world. They didn't show the robot any videos of it crashing. They just taught it the "InFeR" way.

The Test: They made the robot walk through a 300-meter route (indoors and outdoors) and introduced problems like blocked paths, sudden moving obstacles, and even covered the camera.
The Outcome: The robot successfully detected these problems, figured out what was causing them, and fixed itself. It could back up, turn away from obstacles, or ask for help if the situation was impossible. It completed the long journey without a human needing to take over the controls.

Why This Matters

Most previous methods could only say, "I think I'm failing," and then just stop and wait for a human. InFeR is special because it says, "I'm failing, I see why (the cat is there), and I know what to do (turn left)."

It turns a robot that blindly follows a script into a robot that can adapt, recognize danger, and save itself, all without needing extra training data of disasters.

Technical Summary: InFeR – Informed Failure Resilience in Learned Visual Navigation Control

Problem Statement
Imitation learning (IL) has achieved significant success in visual navigation but remains inherently limited by its reliance on training data. Policies trained on successful demonstrations often fail unpredictably when encountering out-of-distribution (OOD) scenarios (e.g., sensor failures, dynamic obstacles, or blocked paths). Current approaches to failure handling suffer from two primary limitations: they often focus solely on detection, triggering human intervention rather than autonomous recovery; and they frequently rely on external detectors that lack access to the policy's internal representations, failing to capture true failure modes or provide actionable insights for recovery. The paper argues that true failure resilience requires not only detecting failures but also autonomously recognizing their sources and recovering from them without additional failure or recovery demonstrations.

Methodology: The InFeR Framework
InFeR is a general framework designed to augment standard IL policies with "informed failure resilience" using only a single static dataset of successful demonstrations. It operates in two main phases:

OOD-Aware Policy Training ( $\pi_{IL-OOD}$ ):
- Variational Information Bottleneck (VIB): The core innovation involves retraining the IL policy with a VIB loss. This regularizes the latent space between the visual encoder and the action decoder. The objective is to compress input observations while maximizing the retention of information crucial for action prediction.
- Latent Space Structuring: By enforcing the latent representation to stay close to a prior (unit Gaussian), the KL divergence between the encoded latent features and the prior serves as a scalar OOD score. High divergence indicates an OOD input.
- Conformal Prediction (CP): To handle time-varying dynamics and avoid arbitrary thresholding, InFeR applies functional conformal prediction on a calibration set of trajectories. This generates time-varying upper bounds for the KL scores, providing a statistically principled method for binary OOD detection ( $b_t$ ).
- Grad-CAM for Recognition: To identify the source of the failure, InFeR applies Grad-CAM to the visual encoder's feature maps using the gradients of the KL-based OOD score. This produces a heatmap ( $M_t$ ) localizing the image regions responsible for the OOD detection.
Heuristic Recovery Policy ( $\pi_{Rec}$ ):
- Since failure and recovery data are absent, InFeR employs a heuristic recovery policy guided by the OOD recognition heatmap.
- The policy selects from a discrete set of macro-actions: backtrack, rotate left, rotate right, or get help.
- Decision Logic: The heatmap is discretized into horizontal bins. If OOD features are detected in the left or right bins, the robot rotates away from them. If the robot is blocked or the OOD region is central, it backtracks (reversing a cached sequence of past actions). If recovery attempts fail after a maximum number of tries ( $t_{max}$ ), the robot halts and requests human assistance.

Key Contributions

Informed Failure Resilience: The paper defines and implements a framework that moves beyond simple failure detection to include source recognition and autonomous recovery, grounded in the policy's internal latent representations.
Data-Efficient Training: InFeR achieves OOD awareness and recovery capabilities without requiring any additional failure or recovery demonstrations, relying solely on the original successful demonstration dataset.
Architecture Agnosticism: The framework is designed to be general, capable of augmenting different IL policy architectures (e.g., CNN-based image-goal policies and recurrent path-following policies) without modifying their core structures.
Integration of Detection and Recognition: Unlike external detectors, InFeR embeds both detection and recognition directly into the policy, utilizing the latent space to distinguish in-distribution from OOD inputs and localizing the specific visual features causing the anomaly.

Experimental Results
The authors evaluated InFeR on a Boston Dynamics Spot robot in real-world indoor and outdoor environments, testing against four failure types: sensor failure, blocked pathways (dead-ends and local minima), close-range dynamic obstacles, and infeasible image goals.

Detection and Recognition Performance: InFeR outperformed state-of-the-art baselines (including density-based methods, one-class discriminators, and reconstruction-based autoencoders) in both OOD detection and recognition. The use of KL scores and Grad-CAM on latent vectors proved significantly more effective and less noisy than reconstruction error metrics on high-dimensional images.
Recovery Success: InFeR-enabled policies successfully detected and recovered from diverse failures. For instance, InFeR-DEC (based on the DECISION policy) autonomously completed a 300m route involving complex terrain, recovering from local minima and dynamic obstacles.
Importance of Recognition: Comparisons with a "Blind" variant (which ignores the heatmap and selects random corrective actions) showed that informed recovery is significantly more efficient and effective, particularly in dynamic scenarios. InFeR-DEC recovered up to 2x faster than the blind variant.
Generalizability: The framework demonstrated consistent performance across two distinct policy architectures (GNM and DECISION), validating its applicability to different visual navigation paradigms.

Significance
The paper claims that InFeR represents a significant step toward safe and robust visual navigation with imitation learning. By enabling policies to autonomously detect, recognize, and recover from failures without external supervision or additional training data, InFeR addresses a critical gap in deploying learned navigation systems in open-world, unstructured environments. The authors posit that this approach allows learned controllers to handle long-range navigation in complex scenarios where traditional IL policies would otherwise fail unpredictably.