Declarative Scenario-based Testing with RoadLogic

This paper introduces RoadLogic, an open-source framework that bridges declarative OpenSCENARIO specifications and executable simulations by combining Answer Set Programming, motion planning, and specification-based monitoring to automatically generate diverse, realistic, and compliant autonomous vehicle testing scenarios.

The Gist

Imagine you are a director trying to film a complex car chase scene for a movie.

The Problem:
In the old days, to get the perfect shot, you would have to write a very specific, step-by-step script for every single car. "Car A moves 5 feet left, then Car B speeds up, then Car A turns right." If you wanted to see 100 different versions of this chase to make sure the actors (the autonomous cars) could handle it, you'd have to write 100 different scripts. That takes forever and is incredibly boring.

Then, someone invented a new way of writing scripts called OpenSCENARIO DSL (OS2). Instead of writing every tiny move, you just write the story: "Car A starts behind Car B, then Car A speeds up and passes Car B, then Car A ends up in front." It's like giving the director a plot summary instead of a shot-by-shot storyboard.

The Catch:
While this new "plot summary" style is great for humans, computers don't know how to act it out. They need the step-by-step instructions. Until now, there was no free, open tool that could take that high-level story and automatically turn it into a realistic, working movie (simulation) where the cars actually drive themselves safely.

The Solution: RoadLogic
This paper introduces RoadLogic, a new open-source tool that acts as the ultimate "AI Director's Assistant." It bridges the gap between the high-level story and the actual driving simulation.

Here is how RoadLogic works, using a simple analogy:

1. The Translator (OS2 to Symbolic Automata)

Imagine the OS2 story is written in a foreign language. RoadLogic first translates this story into a flowchart (called a "Symbolic Automaton").

• Analogy: Think of this as turning the plot summary into a "Choose Your Own Adventure" book. It maps out all the possible checkpoints: "Start here," "Pass the other car," "End up in front." It doesn't say how to drive, just what needs to happen.

2. The Puzzle Solver (Answer Set Programming)

Next, RoadLogic uses a powerful logic engine called ASP (Answer Set Programming). This is like a super-smart puzzle solver.

• Analogy: Imagine you have a jigsaw puzzle where the pieces are "drive forward," "change lane," or "slow down." The puzzle solver looks at the flowchart and tries to fit the pieces together to find a valid path. It asks, "If I move Car A here, can Car B still pass safely?" It finds a high-level plan that satisfies the story without worrying about the physics yet.

3. The Choreographer (Motion Planning)

Now that we have a plan, we need to make it look real. RoadLogic passes the plan to a Motion Planner (FrenetiX).

• Analogy: The puzzle solver gave the choreographer a list of dance moves. The choreographer now figures out the exact footwork. How fast should the car turn? How hard should it brake? It turns the abstract "change lane" instruction into a smooth, realistic curve that a real car could actually drive without crashing.

4. The Critic (Runtime Monitoring)

Finally, as the simulation runs, RoadLogic has a "Critic" watching the whole time.

• Analogy: This is like a strict film critic sitting in the theater. If the actors (cars) deviate from the story—for example, if Car A passes Car B but then immediately crashes into a tree—the Critic yells "Cut!" and rejects that version. Only the simulations that perfectly match the original story get to stay in the final movie.

Why is this a big deal?

• Safety: It allows engineers to test self-driving cars in thousands of different "what-if" scenarios (like bad weather, sudden obstacles, or tricky lane changes) without ever risking a real car on the road.
• Variety: Because the tool is smart, it can generate many different versions of the same story. One time, the car might pass slowly; another time, it might pass quickly. This helps find hidden bugs that a human might miss.
• Open Source: Unlike expensive, secret tools used by big car companies, RoadLogic is free for anyone to use and improve.

In a nutshell:
RoadLogic takes a simple, human-readable story about a car driving, uses logic puzzles to figure out the steps, uses physics to make the driving look real, and double-checks the result to ensure it matches the story. It turns a "what if" idea into a "let's test it" reality, making self-driving cars safer and faster to develop.

Technical Summary

Here is a detailed technical summary of the paper "Declarative Scenario-based Testing with RoadLogic".

1. Problem Statement

Testing Autonomous Vehicles (AVs) requires validating them across a vast range of driving scenarios, including rare and critical edge cases. While Scenario-based testing is a standard methodology, current approaches face significant limitations:

• Imperative vs. Declarative: Existing tools (e.g., OpenSCENARIO XML, Scenic) rely on imperative definitions, requiring developers to manually enumerate specific execution steps and variants. This is labor-intensive and limits coverage.
• The "Instantiation Gap": OpenSCENARIO DSL (OS2) is a declarative language that allows high-level specification of what should happen (intent) rather than how to execute it. However, there is currently no open-source solution capable of systematically instantiating these abstract OS2 specifications into concrete, executable, and specification-compliant simulations.
• Lack of Verification: Even if a simulation is generated, there is often no systematic way to verify that the concrete execution strictly adheres to the original abstract declarative intent.

2. Methodology: The RoadLogic Framework

The authors propose RoadLogic, an open-source framework that bridges the gap between declarative OS2 specifications and executable AV simulations. The framework operates through a five-stage pipeline:

A. Translation to Symbolic Automata

The first step translates the input OS2 scenario into a Symbolic Finite Automaton (SFA).

• Atomic Behaviors: Driving segments (e.g., drive) are mapped to automaton locations.
• Composition: Sequential (·), parallel (||), and choice (∪) operators in OS2 are translated into corresponding automaton transitions (serial composition, Cartesian product, or union of automata).
• Role: The SFA serves as a dual-purpose intermediate representation: it guides the planning process and acts as a runtime monitor.

B. Answer Set Programming (ASP) Encoding

The SFA is encoded into a discrete-time planning problem using Answer Set Programming (ASP) (specifically the clingo solver).

• Discretization: The continuous environment is abstracted into a grid (time steps, longitudinal segments, lateral lanes).
• Planning: The ASP solver generates high-level abstract plans. These plans define a sequence of discrete states (waypoints) and maneuvers (lane changes, speed adjustments) that satisfy the OS2 constraints.
• Incremental Solving: The framework uses multi-shot solving to incrementally extend the time horizon until a valid plan satisfying all goals is found.
• Constraints: The encoding includes domain constraints (e.g., no collisions, lane change limits, vehicle dynamics) to ensure physical plausibility.

C. Concretization via Motion Planning

The abstract discrete plan from the ASP solver is converted into a continuous, physically feasible trajectory.

• Integration: RoadLogic integrates with the CommonRoad simulation environment and the FrenetiX motion planner.
• Multi-Goal Handling: Standard motion planners typically handle single goals. RoadLogic extends the planner to handle intermediate goals sequentially. The vehicle plans a path to the first intermediate goal, reaches it, and then re-plans for the next, ensuring the continuous trajectory respects the high-level ASP plan.

D. Runtime Monitoring

To ensure the concrete simulation faithfully represents the abstract OS2 specification:

• Trace Monitoring: The execution trace generated by the simulator is fed into a runtime monitor derived from the original SFA.
• Verification: If the trace violates the SFA (i.e., the simulation deviates from the intended scenario), the instantiation is rejected, and the system generates a new plan.

E. Implementation

RoadLogic is implemented in Python 3.10, utilizing:

• ANTLR4 for parsing OS2.
• Clingo for ASP solving.
• FrenetiX and CommonRoad for simulation and motion planning.

3. Key Contributions

1. First Open-Source Instantiation Engine: RoadLogic is the first framework to automatically generate executable simulations from declarative OS2 specifications.
2. End-to-End Pipeline: It provides a complete flow from high-level intent (OS2) to low-level execution (simulation), ensuring that only specification-compliant simulations are retained.
3. Formal Conformance: By using symbolic automata for runtime monitoring, the system guarantees that the generated concrete scenarios strictly adhere to the abstract logical constraints.
4. Diversity Generation: The framework supports parameter sampling to generate diverse behavioral variants from a single abstract specification.

4. Evaluation Results

The authors evaluated RoadLogic on 7 distinct OS2 scenarios (e.g., overtaking, lane changing, obstacle avoidance) using two strategies: a Base Strategy (ASP finds all variations) and a Refined Strategy (sampling input parameters).

• Effectiveness (RQ1):

  • RoadLogic successfully generated 100% compliant simulations for the base strategy across all scenarios.
  • The full pipeline (planning, execution, monitoring) typically completed within minutes (e.g., ~3–5 minutes for most scenarios).
  • The execution (simulation) phase dominated the runtime, except for complex scenarios requiring long time horizons where ASP planning became the bottleneck.

• Diversity (RQ2):

  • The Base Strategy produced low diversity due to the depth-first search nature of the ASP solver (generating plans that differ only in the final steps).
  • The Refined Strategy (sampling parameters) significantly increased diversity, producing scenarios with low Jaccard similarity (occupancy grid overlap), indicating distinct trajectories.
  • Trade-off: The refined strategy introduced a success rate drop (some simulations timed out or failed to conform) and increased computation time, but successfully demonstrated the ability to stress-test different behavioral variants.

• Case Study: The system successfully detected a "false positive" where a vehicle satisfied the overtaking constraints but did so in a way that violated the intent (e.g., the lead vehicle changed lanes). This highlighted the framework's ability to refine specifications by adding constraints (e.g., "lead vehicle must stay in lane").

5. Significance and Impact

• Bridging Abstraction Levels: RoadLogic solves a critical bottleneck in AV testing by making high-level declarative specifications (OS2) actionable. This allows engineers to define intent without micromanaging execution details.
• Systematic Testing: By enabling the automatic generation of diverse, compliant scenarios, the framework facilitates systematic exploration of the scenario space, potentially uncovering edge cases that manual testing would miss.
• Open Science: As an open-source tool, it lowers the barrier for researchers to adopt declarative scenario-based testing, promoting reproducibility and standardization in AV validation.
• Future Directions: The authors plan to extend OS2 support to include non-highway networks, events, and optimize the planning efficiency using alternative methods like RRTs, as well as integrating with other simulators like CARLA and BeamNG.