Dynamic Control Barrier Function Regulation with Vision-Language Models for Safe, Adaptive, and Realtime Visual Navigation

1. Problem Statement

Autonomous robots operating in dynamic, unstructured environments face a critical trade-off between safety and efficiency.

• The Challenge: Traditional Control Barrier Functions (CBFs) provide mathematically guaranteed collision avoidance but typically rely on fixed parameters. This leads to suboptimal behavior:
  • Over-conservative: In benign scenes (e.g., open hallways), fixed parameters cause robots to move too slowly or hesitate unnecessarily.
  • Over-permissive: Near complex hazards, fixed parameters may fail to enforce sufficient caution, leading to collisions.
• The Latency Bottleneck: While Vision-Language Models (VLMs) offer superior semantic understanding of scenes, their inference latency is high. Using them synchronously (waiting for a response before acting) makes real-time control impossible. Naively applying asynchronous, delayed VLM risk estimates can lead to safety violations if the robot acts on outdated information.

2. Methodology: AlphaAdj

The authors propose AlphaAdj, a framework that adaptively tunes the conservativeness of a CBF safety filter in real-time using only egocentric RGB inputs and a VLM, while mitigating inference latency.

A. Core Control Architecture

• CBF Safety Filter: The robot uses a standard CBF-QP (Quadratic Program) to filter nominal goal-seeking commands. The safety constraint is governed by a parameter $\alpha$.
  • Small $\alpha$: Enforces strict safety (conservative behavior, slower approach to obstacles).
  • Large $\alpha$: Allows more aggressive motion (closer approach, faster speeds).
• Dynamic Adaptation: Instead of a fixed $\alpha$, the system calculates a time-varying $\alpha(t)$ based on the current scene's risk.

B. Vision-Language Model (VLM) Integration

• Risk Estimation: Every $N$ steps (e.g., 0.5s), the system captures an RGB frame and queries a VLM (LLaVA-1.5 7B) to output a scalar risk score $r \in [0, 1]$.
• Prompt Engineering: The VLM is prompted with the image and context (current speed, minimum barrier distance) to output a strict JSON risk score.
• Risk-to-Parameter Mapping: The risk score $r$ is mapped to a candidate $\alpha_{vlm}$ using a monotone decreasing power-law function:
  $$\alpha_{vlm}(r) = \alpha_{min} + (\alpha_{max} - \alpha_{min})(1 - r)^\gamma$$
  This ensures higher risk yields lower $\alpha$ (more conservative).

C. Latency-Aware Robustness Mechanisms

To handle the asynchronous nature of VLM inference (average latency ~695ms) and prevent safety failures due to stale data, AlphaAdj employs two key mechanisms:

1. Staleness-Gated Fusion:

   • The system tracks the time since the last valid VLM update ($t_{vlm}$).
   • If the update is fresh, the system uses a "Soft Cap" fusion: $\alpha_{final} = \min(\alpha_{vlm}, \alpha_{cap})$.
   • If the update is stale (delayed beyond a threshold), the system ignores $\alpha_{vlm}$ and defaults to a "Hard Cap" (more conservative fallback).

2. Geometric Dynamic Cap ($\alpha_{cap}$):

   • Even if the VLM suggests a low risk, a geometric safety envelope prevents the robot from being too aggressive when physically close to obstacles.
   • The cap is a function of clearance margin ($m$) and speed ($v$).
   • Distance Component: As the robot gets closer to an obstacle, the cap tightens (lowers $\alpha$).
   • Speed Component: Higher speeds further tighten the cap.
   • Fusion Logic:
     • Fresh VLM: $\alpha_{final} = \min(\alpha_{vlm}, \alpha_{soft\_cap})$ (Allows VLM to be less conservative if safe, but never exceeds the soft cap).
     • Stale VLM: $\alpha_{final} = \alpha_{hard\_cap}$ (Forces maximum conservatism).

D. Implementation Optimizations

• Asynchronous Execution: The control loop runs at 60 Hz continuously. The VLM query runs in the background (every 30 steps) without blocking the robot.
• Latency Reduction: Used 4-bit quantization, strict JSON output constraints, and local hosting (FastAPI) to reduce end-to-end latency from ~1.4s to 695ms.

3. Key Contributions

1. Real-time Adaptive Mobility from RGB: Demonstrated a method to modulate low-level CBF safety parameters directly from high-level semantic VLM risk estimates, improving the safety-efficiency trade-off without changing the controller structure.
2. Latency-Aware Robustness: Introduced a novel fusion policy combining a geometric dynamic cap and staleness detection. This ensures safety even when VLM updates are delayed or anomalous, a critical gap in existing VLM-based navigation.
3. Practical Risk-to-$\alpha$ Mapping: Proposed a specific geometric and speed-aware mapping strategy that allows the system to be aggressive in open spaces but automatically conservative in tight or high-speed scenarios.

4. Experimental Results

The framework was evaluated in NVIDIA IsaacSim across 8 scenarios (4 environments $\times$ 2 scenario types: static/dynamic obstacles).

• Safety: AlphaAdj achieved 100% success rate (8/8) with 0 collisions.
  • Comparison: Fixed $\alpha_{max}$ (aggressive) resulted in 6 collisions. Fixed $\alpha_{min}$ (conservative) failed to reach goals in dynamic scenarios due to excessive hesitation.
• Efficiency:
  • AlphaAdj improved path length and time-to-goal by up to 18.5% compared to the fixed conservative baseline ($\alpha_{min}$).
  • Compared to the "No Cap" adaptive baseline, AlphaAdj improved time-to-goal by 4.5% on average while significantly reducing collision risk.
• Latency Handling: The system maintained a 60 Hz control loop without stalling, unlike synchronous approaches which would require the robot to stop and wait for VLM inference (often >1 second).

5. Significance

• Bridging the Gap: AlphaAdj successfully bridges the gap between high-level semantic understanding (VLMs) and low-level safety-critical control (CBFs), addressing the "latency vs. safety" dilemma that has hindered VLM adoption in robotics.
• Practical Deployment: By proving that asynchronous VLM updates can be safely integrated via geometric fallbacks, the paper provides a blueprint for deploying large foundation models on real-world robots without requiring expensive, low-latency cloud infrastructure or sacrificing safety.
• Adaptive Behavior: It moves beyond static safety policies, enabling robots to dynamically adjust their "personality" (cautious vs. assertive) based on the immediate visual context, mimicking human-like adaptability in navigation.