💻 computer science

Simulator Ensembles for Trustworthy Autonomous Driving Systems Testing

This paper introduces MultiSim, a search-based testing approach that leverages an ensemble of simulators to prioritize and identify simulator-agnostic failure scenarios for autonomous driving systems, demonstrating significantly higher effectiveness and efficiency compared to single-simulator or independent multi-simulator testing methods.

Original authors: Lev Sorokin, Matteo Biagiola, Andrea Stocco

Published 2026-07-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lev Sorokin, Matteo Biagiola, Andrea Stocco

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are training a robot car to drive itself. You can't just let it loose on real highways immediately; that would be dangerous and expensive. So, you build a digital playground—a video game world where the car can crash, spin out, and learn from its mistakes. This is what engineers call a simulator.

But here's the tricky part: there isn't just one video game engine. There are many, like different brands of game consoles. And just like how a game might look or behave slightly differently on a PlayStation versus an Xbox, these driving simulators sometimes disagree.

The Problem: The "Flaky" Game Console

The paper points out a frustrating issue: sometimes, a robot car will drive perfectly in one simulator but crash in another, even if the road looks exactly the same. This is called flakiness. It's like if you played a level in a video game and it said "Game Over," but when you tried the exact same move on a different console, it said "Level Complete."

If you only test your robot car on one simulator, you might think you found a dangerous flaw, but it turns out it was just a glitch in that specific game engine. Or worse, you might think the car is safe because it passed in one engine, but it would have crashed in another. This makes it hard to trust the testing.

The Solution: The "Council of Simulators"

The authors, Lev Sorokin, Matteo Biagiola, and Andrea Stocco, came up with a new way to test called MultiSim. Instead of asking just one simulator, "Is this road safe?", they ask a whole team of them at the same time.

Think of it like a panel of judges at a talent show.

The Old Way (Single Simulator): You ask one judge. If they say "You're fired," you fire the act. But maybe that judge just hates your music style.
The Old Multi-Way (DSS): You ask two judges separately. If both say "fired," you fire the act. But you have to wait for both to finish their separate shows before you decide.
The New Way (MultiSim): You put the act on stage in front of all the judges simultaneously. You only keep the act if all the judges agree it's bad. If one judge says "Great!" and another says "Terrible," the system ignores that act for now because the disagreement suggests the problem might be with the judges (the simulators), not the act (the car).

By running the test on multiple simulators at once, MultiSim filters out the "fake" failures caused by simulator glitches. It only keeps the "real" failures that happen everywhere.

What They Found

The team tested this idea on three different robot cars (using different brain architectures like CNNs and Transformers) and three different simulators: BeamNG, Donkey, and Udacity.

Here is what the numbers say:

Better at finding real trouble: MultiSim found 66% more "simulator-agnostic" failures (real problems that happen in all simulators) compared to testing with just one simulator.
Beating the competition: Compared to the previous best method (called DSS), MultiSim found up to 3.4 times more valid failures.
The Success Rate: Across all their tests, MultiSim identified 70% of its findings as valid failures. In comparison, the single-simulator method only got 47%, and the old multi-simulator method (DSS) got 65%.
Speed: It didn't slow things down. Finding the first real failure took about the same amount of time as the other methods, though the results varied a bit more.

The "Magic Crystal Ball" Upgrade

The researchers also tried to make MultiSim even smarter. They trained a machine learning model (a "surrogate") to act like a crystal ball. Before running a test on the heavy, slow simulators, this crystal ball predicts: "Hey, these two simulators are going to disagree on this road. Don't bother running it yet!"

This trick worked well. It helped the system skip the wasted tests and focus on the interesting ones. In their tests, this upgrade increased the number of valid failures found by about 38% on average and made the search more consistent.

What They Didn't Prove

It's important to know what this paper doesn't claim.

It's not a magic fix for everything: The paper explicitly rules out the idea that you can just ignore simulator differences. You must check for them.
It's not a solved problem for all cars: They tested this on three specific types of lane-keeping cars. They suggest it might work for other systems like emergency braking, but they haven't proven it yet.
It doesn't fix the simulators: They didn't go into the code of the simulators to fix the bugs causing the disagreements. They just built a system that works around the bugs by ignoring the ones that don't agree.
It's based on simulations: The results are based on running tests in these digital worlds. They haven't proven this works on a real car on a real road yet, though the goal is to make the real-world testing safer.

The Bottom Line

The paper suggests that if you want to find the real dangerous flaws in a self-driving car, don't trust just one video game engine. Trust a team. By running tests in a group of simulators and only listening to the ones that agree, you can find the true dangers much faster and with more confidence. It's a smarter way to play the "what if" game before the car ever hits the street.

Technical Summary: Simulator Ensembles for Trustworthy Autonomous Driving Systems Testing

1. Problem Statement

Scenario-based testing using driving simulators is the standard method for validating Automated Driving Assistance Systems (ADAS) and reducing the need for costly in-field road testing. However, existing research indicates that test execution results are often inconsistent. These inconsistencies arise from two primary sources:

Test Flakiness: Non-deterministic behavior where the same test scenario yields different pass/fail outcomes even within the same simulator, often due to timing, synchronization, or environmental uncertainties.
Simulator Disagreement: Different simulators (e.g., BeamNG, Udacity, Donkey) may produce divergent outcomes for the same test input due to differences in physics engines, rendering, or implementation details.

Current approaches to mitigate these issues are limited. Some discard flaky tests, while others, such as the "Digital Siblings" (DSS) framework, run independent test generation campaigns on different simulators and merge results as a post-processing step. This post-hoc merging is inefficient because the search algorithm evolves test cases based on single-simulator feedback, potentially wasting resources on simulator-specific (invalid) failures before they are filtered out.

The core problem addressed is the lack of a testing methodology that can identify simulator-agnostic failures (valid failures that represent true defects in the ADAS) directly during the search process, rather than after it, while maintaining efficiency.

2. Methodology: MultiSim

The authors propose MultiSim, a novel search-based software testing (SBST) approach that treats multiple simulators as an ensemble during the test generation process.

2.1 Multi-Objective Optimization Formulation

MultiSim reformulates the testing problem as a multi-objective optimization (MOO) problem. Instead of evaluating a test input in a single simulator, the system evaluates it across an ensemble of simulators simultaneously. The fitness vector for a test input $x$ is defined as:
$F(x) = (f_1(x), \dots, f_j(x), f_d(x), f_a(x, A))$
Where:

$f_k(x)$ : The fitness value (e.g., cross-track error) obtained from simulator $S_k$ .
$f_d(x)$ : The average distance between the fitness values of all simulators. Minimizing this term drives the search toward scenarios where simulators agree.
$f_a(x, A)$ : The distance of the current test input to previously found tests in an archive $A$ . Maximizing this term ensures diversity in the search space.

By minimizing the fitness values (indicating failure) and the distance between simulators, the algorithm prioritizes simulator-agnostic failures—scenarios where the ADAS fails consistently across different simulation environments.

2.2 Test Representation and Operators

Representation: Road scenarios are modeled using control points interpolated via Catmull–Rom splines. The search space is parameterized by segment angles and lengths.
Genetic Operators: The approach employs standard genetic operators (selection, crossover, mutation) adapted for road geometry. Crossover exchanges road segments between parent roads, while mutation randomly alters segment angles or lengths. Invalid roads (e.g., those intersecting map boundaries) are regenerated.
Oracle Function: A test is considered a failure if the maximum cross-track error (XTE) exceeds a predefined threshold (e.g., 2.2 meters), indicating the vehicle has left its lane.

2.3 Validation Strategy

To confirm that identified failures are truly simulator-agnostic, the authors employ a validation phase:

Feature Mapping: Generated failing tests are mapped to a feature space defined by road curvature and number of turns.
Re-execution: Failing tests are re-executed multiple times (to account for flakiness) on simulators not used during the search phase.
Thresholding: A test is classified as a "valid failure" only if it fails consistently (e.g., 100% failure rate) across all validation simulators.

2.4 Surrogate Model Enhancement (RQ3)

To improve efficiency, the authors propose an enhancement using a surrogate model (a machine learning classifier). This model predicts whether a candidate test input will result in a simulator disagreement. If the predicted probability of disagreement exceeds a threshold, the candidate is discarded before running the expensive full simulation. This allows the search to focus on high-value regions of the input space.

3. Empirical Study and Results

The study evaluated MultiSim against SingleSim (single-simulator testing) and DSS (Digital Siblings) using three lane-keeping ADAS models of increasing complexity:

DAVE-2: A convolutional neural network (CNN).
ViT: A Vision Transformer-based architecture.
TCP: A hybrid trajectory-control policy network.

The experiments utilized three simulators: BeamNG, Udacity, and Donkey, tested in various pairings (BD, BU, UD).

3.1 Effectiveness (RQ1)

Valid Failure Rate: MultiSim achieved an average valid failure rate of 70%, outperforming DSS (65%) and SingleSim (47%).
Configuration Performance: The BD (BeamNG + Donkey) configuration yielded the highest validity rates for DAVE-2 (99%) and ViT (98%). The BU (BeamNG + Udacity) configuration performed best for TCP (80%).
Statistical Significance: Wilcoxon rank-sum tests confirmed that MultiSim significantly outperforms SingleSim and DSS in generating valid failures, with large effect sizes in most comparisons.

3.2 Efficiency (RQ2)

Search Budget: MultiSim demonstrated efficiency comparable to SingleSim, requiring a similar search budget to find the first valid failure.
Comparison with DSS: MultiSim generally showed better efficiency than DSS, particularly in the BD configuration, where it found valid failures faster than the post-processed DSS approach.

3.3 Prediction and Surrogate Models (RQ3)

Classifier Performance: A Random Forest classifier achieved the best performance in predicting simulator disagreements (F-1 scores of 74–83%, AUC of 0.81–0.89).
Impact: Integrating the surrogate model (BD-P) increased the number of valid failures by approximately 38% on average and reduced the variability in search budget required to find the first valid failure. It successfully bypassed low-value simulations without sacrificing the discovery of valid faults.

4. Key Contributions

MultiSim Approach: A novel search-based testing framework that integrates multiple simulators directly into the optimization loop, treating simulator agreement as a first-class objective to identify simulator-agnostic failures.
Empirical Validation: An extensive study demonstrating that MultiSim outperforms both single-simulator testing and the state-of-the-art DSS framework in terms of the rate of valid failures identified.
Surrogate Integration: A demonstration that machine learning predictors can be used to filter simulator disagreements during the search, improving both effectiveness and efficiency.
Open Replication: The authors provide the source code, experimental data, and test generation approach to encourage open research.

5. Significance and Claims

The paper claims that combining an ensemble of simulators during the search is a promising approach for automated cross-replication in ADAS testing. The significance lies in shifting the paradigm from post-hoc validation of simulator results to in-process consensus building.

Reliability: By prioritizing scenarios where simulators agree, MultiSim reduces the risk of identifying failures that are artifacts of a specific simulator's implementation (simulator-dependent failures).
Efficiency: The approach avoids the inefficiency of generating and then discarding simulator-specific failures, a common issue in post-processing methods like DSS.
Generalizability: While the study focuses on lane-keeping, the authors note that the modular architecture (based on OpenSBT) and the concept of simulator coupling can be generalized to other ADAS functions (e.g., emergency braking) and higher levels of autonomy, provided that consistent scenario representations can be established.

The authors remain modest regarding the scope, acknowledging that the optimal simulator pairing depends on the specific System Under Test (SUT) and its dominant failure modes (e.g., control-related vs. perception-related). They also note that while the approach mitigates the symptoms of simulator disagreement, it does not inherently solve the root causes of the discrepancies, which would require deeper inspection of simulator code.