Collaborative Multi-Agent Testing for Emergent Failure Discovery in Autonomous Driving Systems

Technical Summary: CREAD – Collaborative Multi-Agent Testing for Emergent Failure Discovery in Autonomous Driving Systems

Problem Statement
Autonomous Driving Systems (ADS) are safety-critical, open-world systems where failures arise not only from faults within individual modules (perception, planning, control) but also from complex interactions between them. Current testing research often treats critical functions—perturbation generation, behavioral assessment, and test case selection—as loosely coupled, sequential steps rather than coordinated roles. This separation limits the ability to consistently uncover rare, high-consequence failures, particularly those emerging from the "long tail" of safety-critical scenarios. Furthermore, existing approaches, including LLM-guided synthesis and feedback-guided fuzzing, often struggle to steer exploration toward diverse, safety-relevant scenario families without generating redundant test cases, and they lack mechanisms to effectively reuse intermediate evidence between generation and validation stages.

Methodology: The CREAD Framework
The authors propose CREAD (CollaboRative for Emergent ADS behaviour Discovery), a collaborative multi-agent testing framework designed to coordinate perturbation generation, validation, and search exploration through a shared blackboard architecture and an Orchestrator Agent. Unlike prior frameworks that focus primarily on realistic scenario construction, CREAD treats ADS testing as a closed-loop collaborative process.

The framework currently instantiates three coordinated roles:

1. Perception Fuzzer Agent: Generates structured, perception-oriented perturbations and scenario variants. In the current implementation, it uses an LLM (GLM-5-Turbo) to refine seeded scenario templates into specific test cases involving traffic configurations, ego states, and perception presets (e.g., fog, rain, night). It executes scenarios in two passes: a baseline run and a perturbed run.
2. Metamorphic Validator Agent: Assesses behavioral consistency by comparing the baseline and perturbed executions. It detects safety-relevant inconsistencies (e.g., collisions, near-misses, lane departures, or perception faults) where the perturbed run exhibits disproportionate behavioral changes relative to the baseline.
3. Orchestrator Agent: Coordinates global testing priorities using a coarse Quality-Diversity (QD)-inspired archive. It does not generate scenarios itself but selects which baseline scenarios to revisit or mutate based on heuristic scores derived from failure yield, severity, novelty, and diversity. It prioritizes underexplored but promising scenario families.

Key Mechanism: Blackboard Collaboration
Agents interact asynchronously via a shared blackboard that stores baseline scenarios, perturbed scenarios, execution traces, validation outcomes, and coverage summaries. This design allows the Orchestrator to use feedback from the Validator to guide the Fuzzer toward specific scenario families, creating a closed loop where complementary testing functions inform one another during the search process.

Experimental Evaluation
The authors evaluated CREAD using the HighwayEnv simulator in two environments: Highway (structured straight-road) and Roundabout (denser, richer interactions). The study employed an ablation design comparing three configurations over 50 iterations each:

• Collaborative (Full System): Both agents with blackboard and orchestration.
• Non-collaborative: Both agents executed sequentially without blackboard or orchestration.
• Single-Agent (Baseline): Perception Fuzzer only.

The ADS under test was a deterministic heuristic policy to isolate the testing architecture's effect, and perception degradation was modeled as structured state corruption rather than full sensor simulation.

Results

• Impact of Collaboration (RQ1): In the Highway environment, the collaborative configuration significantly improved failure discovery, increasing failures from 14 to 52 per 100 scenarios compared to the non-collaborative baseline, while maintaining failure diversity and increasing the perception fault rate. In the Roundabout environment, the collaborative configuration yielded 52 failures per 100 scenarios, slightly underperforming the non-collaborative baseline (58 failures) but showing a marginal improvement in failure diversity.
• Impact of Multi-Agent Architecture (RQ2): The non-collaborative multi-agent configuration improved failure discovery over the single-agent baseline in both environments. In Highway, failures rose from 12 to 14 per 100 scenarios; in Roundabout, the increase was more substantial, from 38 to 58 per 100 scenarios.

Significance and Claims
The paper claims that CREAD introduces a new paradigm for ADS testing by organizing distinct testing roles (fuzzing, validation, coordination) as interacting agents rather than isolated steps. The primary contribution is empirical evidence that collaborative multi-agent testing can improve failure discovery, particularly in specific environments like the Highway setting, where it outperforms both single-agent and non-collaborative multi-agent baselines.

The authors modestly frame the current work as a proof-of-concept instantiation focused on perception fuzzing, noting that the architecture is extensible to planning and control modules. They acknowledge limitations, including the use of a lightweight simulator (HighwayEnv) rather than high-fidelity platforms like CARLA, the use of a heuristic ego policy rather than a learned controller, and the abstraction of perception degradation. The paper concludes that while collaborative testing is a promising direction for emergent failure discovery, its benefits are environment-dependent, and further work is required to validate the framework in higher-fidelity simulations and with more complex ADS controllers.