VISION-SLS: Safe Perception-Based Control from Learned Visual Representations via System Level Synthesis

1. Problem Statement

The paper addresses the challenge of safe, nonlinear output-feedback control for robots operating under partial observability with high-dimensional visual inputs (e.g., RGB images).

• Core Difficulty: Robots must stabilize nonlinear dynamics and satisfy safety conditions (e.g., collision avoidance) while relying exclusively on noisy, high-dimensional sensor data.
• Limitations of Existing Methods:
  • Reinforcement Learning (RL): Scales to high dimensions but offers no safety guarantees and lacks interpretability.
  • Model-Based Control (MPC, SOS): Provides safety guarantees but typically relies on the separation principle (estimation and control are decoupled). This prevents "information-gathering" behaviors (active sensing), where the robot moves to reduce uncertainty, which is crucial for robustness against occlusions.
  • Belief Space Planning: Handles uncertainty but is often intractable for high-dimensional systems or fails to enforce hard constraints under nonlinear dynamics.

The goal is to develop a scalable framework that couples perception and control to provide robust guarantees for constraint satisfaction while enabling information-gathering behavior directly from raw pixels.

2. Methodology: VISION-SLS

The proposed method, VISION-SLS, integrates learned visual representations with System Level Synthesis (SLS). It avoids the separation principle by jointly optimizing a nominal trajectory and a feedback controller that actively reduces state uncertainty.

A. Key Components

1. Learned Low-Dimensional Observation Mapping:

   • Instead of planning in pixel space ($10^5+$ dimensions), the method maps high-dimensional images to a low-dimensional reduced observation space ($y_r$).
   • Architecture: Uses pre-trained DINO features (self-supervised visual foundation model) followed by a learned projection network $p(\cdot)$.
   • Training Objective: Minimization of the error between projected features and a state-based reduced model $h_r(x)$ while simultaneously maximizing the observability of the underlying state (ensuring the reduced features contain control-relevant information).
   • Calibration: A state-dependent error bound (coverage) is empirically calibrated to limit the residual error between the learned mapping and the true state. This allows the system to treat perception error as a bounded disturbance.

2. Nonlinear System Level Synthesis (SLS):

   • The control strategy is parameterized as a causal affine time-varying output-feedback controller around a jointly optimized nominal trajectory $(z, v)$.
   • SLS Parameterization: Instead of optimizing feedback gains directly, SLS optimizes the closed-loop response maps ($\Phi$), which map disturbances (process noise, measurement noise, and perception error) to state and input deviations.
   • Robust Constraint Satisfaction: The method enforces "tube tightening." It ensures that for all possible disturbances within the calibrated bounds, the actual trajectory remains within an "achievable tube" that satisfies safety conditions. This is achieved by tightening the constraints for the nominal trajectory based on the worst-case propagation of errors.

3. Efficient Solver (Double-Riccati + SCP):

   • The resulting optimization problem is non-convex due to nonlinear dynamics and the coupling of trajectory and controller.
   • Sequential Convex Programming (SCP): The problem is solved iteratively by linearizing the dynamics and convexifying the constraints around the current nominal trajectory.
   • Double-Riccati Structure: A key theoretical contribution is the proof that the unconstrained subproblem (LQG in SLS form) admits a Double-Riccati solution. This enables the solver to compute optimal closed-loop responses using parallel backward Riccati recursions (for control) and Kalman recursions (for estimation) followed by forward passes. This structure makes the solver significantly faster than generic convex solvers and scales to large systems.

3. Main Contributions

1. Novel Controller Parameterization: The first method coupling nonlinear trajectory optimization with output-feedback SLS for nonlinear systems, thereby enabling robust information-gathering control with safety guarantees.
2. Dynamics-Aware Visual Abstraction: A method for learning low-dimensional observation mappings from foundation model features (DINO) optimized for control observability, accompanied by a calibrated, state-dependent error bound.
3. Efficient Solver: The derivation of a Double-Riccati solution for unconstrained output-feedback SLS, enabling a scalable solver that utilizes parallel recursions to handle high-dimensional robotics problems.
4. Empirical Validation: Successful deployment on diverse systems, including a 4D car, a 10D quadcopter, a 59D humanoid, and a physical TurtleBot, all using high-resolution RGB images.

4. Results

The method was evaluated on four simulated tasks and one hardware experiment, compared against Certainty Equivalent (CE) and Non-Robust (NR) baselines.

• Performance Metrics:
  • Success Rate (SR): VISION-SLS achieved a 100% success rate across all simulated tasks (Light-Dark, 4D Car, Quadcopter, Humanoid).
  • Constraint Violation Rate (CVR): VISION-SLS maintained 0% (or near 0%) constraint violations. In contrast, baselines failed significantly:
    • 4D Car: Baselines had violation rates of 6.5%–55%.
    • Quadcopter: The non-robust baseline had a violation rate of 92.8%.
    • Humanoid: The non-robust baseline had a violation rate of 98%.
• Information Gathering: In "Light-Dark" and "Occluded Parking Lot" scenarios, VISION-SLS actively steered the robot into areas of lower perception uncertainty (e.g., avoiding heavy occlusions) to reduce the size of the achievable tubes, whereas CE baselines drove through areas of high uncertainty, leading to larger tubes and constraint violations.
• Scalability & Speed:
  • The method successfully controlled a 59-state Humanoid (Unitree G1) performing a backflip.
  • Due to the Double-Riccati structure, the solver was significantly faster than generic SLS solvers (e.g., MOSEK), enabling planning horizons of 30–60 steps in seconds.
• Hardware Validation: The method was deployed on a TurtleBot with onboard cameras, successfully navigating an office environment and avoiding obstacles, outperforming baselines in both safety and solution times.

5. Significance

• Bridging Learning and Control: VISION-SLS demonstrates that learned visual abstractions can be rigorously coupled with model-based control to provide hard safety guarantees, moving beyond the "black-box" nature of pure RL.
• Active Sensing: By rejecting the separation principle, the method enables robots to make decisions that improve their own state estimation (information gathering), a critical capability for robust operation in real-world, partially observable environments.
• Scalability: The use of the Double-Riccati structure within an SCP framework makes safe, nonlinear, vision-based control practical for high-dimensional systems (e.g., humanoids), which were previously intractable for rigorous, safety-certified control.
• Practicality: Successful hardware deployment proves that theoretical guarantees hold even in the presence of real-world noise, latency, and unmodeled dynamics.

In summary, VISION-SLS offers a scalable, theoretically grounded pipeline for safe visuomotor control that effectively manages the trade-off between the complexity of high-dimensional perception and the stringent requirements of safety-critical robotics.