Technical Summary: Adversarial Stress Testing of SPARK Humanoid Safety Filters

Problem Statement

Humanoid robots present unique challenges for safe deployment due to their high-dimensional bodies, complex collision constraints, and necessity to operate in close proximity to humans and obstacles. While safety filters are employed to modify nominal control actions when collision-avoidance constraints are threatened, standard benchmark scores often fail to reveal how these filters behave in difficult or adversarial environments. A method that performs well in a clean, nominal benchmark may degrade significantly when faced with noisy obstacle estimates, delayed sensor information, or crowded environments. This paper addresses the gap between nominal performance and robustness by investigating the behavior of safety filters under stress.

Methodology

The authors conducted a focused study replicating and stress-testing the SPARK (Safe Humanoid Control) benchmark, specifically the case G1SportMode_D1_WG_SO_v1. The experimental pipeline consisted of three main components:

1. Replication in MuJoCo: The study was executed in a local Ubuntu environment using the MuJoCo physics engine and the SPARK codebase. Six safety filter variants were evaluated under a shared benchmark setting with fixed time horizons and controlled random seeds (20, 21, and 22):

   • RSSA (Randomized Safe Set Algorithm)
   • RSSS (Randomized Safe Set Strategy)
   • SSA (Safe Set Algorithm)
   • CBF (Control Barrier Functions)
   • PFM (Potential Field Method)
   • SMA (Safety Metric Algorithm)

2. Post-Processing Pipeline: A custom pipeline was developed to parse raw .npz simulation logs into interpretable metrics. This involved extracting high-dimensional trajectory data to compute:

   • Goal Tracking: Final distance between the robot's arm and the target (dist_goal_arm).
   • Safety Metrics: Minimum distance between the robot's collision volumes and obstacles (d_env_min), and the count of environment-collision steps (C_env), defined as the number of timesteps where the minimum distance is negative.

3. Adversarial Stress Testing: Instead of altering the MuJoCo environment directly, the authors injected attacks into the safety filter's perception path via monkey-patching. Three stress dimensions were tested:

   • Obstacle Crowding: Increasing the number of static obstacles from 5 to 15 and 30.
   • Perception Noise: Adding Gaussian noise (standard deviations 0.02–0.10) to perceived pairwise distances.
   • Sensor Latency: Buffering and returning stale obstacle information with delays up to 10 simulation steps.

Key Results

Baseline Replication

Under nominal conditions (fixed seeds, standard obstacle count), a clear safety–performance trade-off emerged:

• PFM tracked the goal most closely but resulted in the highest number of collision steps.
• SMA achieved the lowest average environment-collision count but tracked the goal less precisely.
• SSA, RSSA, and RSSS exhibited more balanced behavior between safety and goal tracking.
• Several runs produced repeated "No Solution" outputs, indicating feasibility limits when constraints became difficult to satisfy.

Obstacle Crowding

As obstacle density increased, the relative performance of the filters shifted:

• With 5 obstacles, most methods avoided collisions.
• With 15 obstacles, CBF demonstrated the fewest collision steps.
• With 30 obstacles, SMA performed best, while PFM degraded most significantly, showing a sharp increase in collisions and task inefficiency.
• The results indicate that superior goal tracking in nominal settings does not guarantee safer behavior in crowded environments.

Perception Noise and Latency

• Perception Noise: PFM degraded sharply even at low-to-medium noise intensities, showing large increases in collision steps and goal error. SSA remained relatively safe, while CBF and SSS stayed stable. SMA showed moderate degradation.
• Sensor Latency: While collision counts were more similar across methods under latency, stale information still negatively impacted both safety and tracking. PFM maintained the lowest collision count, whereas SSA suffered the highest collision count at high latency.

Significance and Claims

The paper claims three primary contributions:

1. Replication: Successfully replicating a specific SPARK humanoid safety benchmark in MuJoCo, allowing for a controlled comparison of six safety filters under identical task structures and seeds.
2. Tooling: Developing a post-processing pipeline that converts raw, high-dimensional simulation logs into standardized metrics (goal distance, minimum distance, collision counts), facilitating clearer analysis of safety–performance trade-offs.
3. Robustness Insight: Demonstrating through adversarial stress testing that nominal safety–performance behaviors are not static. Obstacle crowding, perception noise, and sensor latency can expose failure modes that are invisible in standard benchmark evaluations.

The authors conclude that humanoid autonomy evaluation must extend beyond nominal performance. They advocate for using metrics that expose failure modes under harder sensing and environmental conditions before deployment. The study is modest in scope, focusing on a single SPARK case (G1SportMode_D1_WG_SO_v1) rather than the entire benchmark suite, and acknowledges limitations such as high runtime costs and a lack of systematic measurement for infeasibility frequencies ("No Solution" events). Future work is suggested to expand across more benchmark cases, seeds, and systematic stress-test settings, including direct comparisons with original SPARK metrics and recovery time analysis.