💻 computer science

JTA: Joint Testability Architecture for Scenario-Based Validation of Safety-Critical Software

This paper introduces the Joint Testability Architecture (JTA), a novel framework that unifies the scenario, test system, and system under test into a single design object characterized by controllability, observability, and isolability to enhance the validation adequacy of safety-critical software through scenario contracts, capability assessment, and bridge-oriented design actions.

Original authors: Wenyao Xue, Jiandi Wang, Yichen Wang

Published 2026-08-07

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Wenyao Xue, Jiandi Wang, Yichen Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to prove that a self-driving car is safe enough to hit the road. You can't just write a list of "what-if" questions and hope the car answers them correctly. You need a whole team working together: the car itself (the software), the testers (the people and computers running the tests), and the scenarios (the specific, tricky situations you want to test, like a sudden rainstorm or a pedestrian jumping out).

In the world of safety-critical software—like the brains behind airplanes, trains, and autonomous vehicles—this team often gets out of sync. The car might be ready, but the testers can't create the exact rainstorm needed. Or, the testers can create the storm, but the car doesn't "speak up" clearly enough to tell them why it stopped. This paper, written by researchers at Beihang University, tackles a big question: How do we make sure the car, the testers, and the test scenarios are all on the same page? They introduce a new way of thinking called Joint Testability Architecture (JTA). Instead of looking at the software code in isolation, JTA treats the whole trio as a single, connected system. It asks three simple but powerful questions for every test: Can we control the situation? Can we see what's happening? And if something goes wrong, can we pinpoint exactly who or what is responsible?

The Problem: A Broken Chain of Trust

Think of testing safety-critical software like trying to solve a mystery in a dark room. You have a detective (the Test System), a suspect (the System Under Test, or the software), and a specific crime scene you need to recreate (the Scenario).

In the past, researchers mostly focused on the suspect. They asked, "Is the code written in a way that makes it easy to test?" But the authors of this paper argue that this is like asking if a suspect is easy to interrogate without checking if the detective has a flashlight or if the crime scene is even set up correctly. If the detective can't turn on the lights (Observability), or if the crime scene is too chaotic to recreate (Controllability), the best code in the world won't help.

The paper suggests that "testability" isn't just a property of the code; it's a property of the relationship between the code, the tools, and the scenario. If any one of these three links is weak, the whole validation process fails.

The Solution: The "Three Bridges"

To fix this, the authors propose a blueprint called Joint Testability Architecture (JTA). Imagine the Scenario, the Test System, and the Software as three islands. To make them work together, you need three bridges connecting them.

The Control Bridge: This connects the Test System to the Software. It asks, "Can we actually force the software into this specific situation?" If you want to test what happens when a drone loses its remote signal, can the test system reliably cut that signal at the exact right moment? If the bridge is broken, you can't even start the test.
The Evidence Bridge: This connects the Software back to the Test System. It asks, "Can we see what's happening?" When the drone loses the signal, does it scream for help in a way the test system can understand? Does it leave a clear trail of logs, or just a confusing mess of data?
The Attribution Bridge: This is the most crucial one. It asks, "If things go wrong, do we know why?" If the drone crashes, was it because the signal was cut (a real problem), or because the test system accidentally cut the signal too early (a fake problem)? This bridge ensures we can distinguish between a real failure and a testing mistake.

The Secret Weapon: The "Scenario Contract"

The paper introduces a clever tool called a Scenario Contract. Think of this as a strict checklist or a rulebook for every single test. Before you even run a test, you write down exactly what you need:

What are we testing? (The goal)
How do we trigger it? (The control)
What proof do we need to see? (The evidence)
Who is responsible if it fails? (The attribution)

By filling out this contract first, you can spot "blind spots" before you waste time running tests. If the contract says you need to distinguish between two types of failures, but your software doesn't have a way to tell them apart, the contract reveals the gap immediately.

The Case Study: The ArduPilot Drone

To see if this idea works, the authors tested it on ArduPilot, a popular open-source flight control system used in drones and robots. They looked at three specific "disaster" scenarios:

Lost Remote Control: The drone loses connection to its pilot.
Lost Ground Station: The drone loses connection to the computer on the ground.
Confused Brain: The drone's internal sensors (which guess where it is) start giving bad data.

What they found:

The Good News: The "Lost Remote Control" scenario was actually doing pretty well. The test system could easily cut the signal, and the drone had clear logs to show it happened. The "Control" and "Evidence" bridges were strong.
The Bad News: The "Confused Brain" scenario was a mess. The test system struggled to create a realistic "confused brain" situation (weak Control bridge), and even when it did, the drone's logs were too vague to tell if the confusion came from a sensor error or a GPS glitch (weak Attribution bridge).

The authors calculated a "safety score" for the whole system. Because the "Confused Brain" scenario is so dangerous (high criticality), its failure dragged the entire system's score down to just 28.6%. This means that while the drone is great at handling simple signal loss, it is currently very hard to prove it's safe against complex sensor errors.

The Takeaway

The paper doesn't claim to have "solved" drone safety or fixed the ArduPilot code. Instead, it offers a new way to diagnose the problem. It suggests that the difficulty isn't just that the code is hard to write; it's that the whole system of testing is misaligned.

By using the "Three Bridges" and the "Scenario Contract," engineers can stop guessing why a test failed. They can look at their checklist and say, "Ah, we have a gap in the Attribution Bridge. We need to add a specific code label to tell the difference between a sensor error and a GPS error."

In short, JTA turns the vague feeling of "this is hard to test" into a specific, actionable to-do list. It moves the conversation from "Is the code good?" to "Is our entire testing team set up to prove the code is safe?" For anyone building software that keeps people alive, that's a pretty big shift.

Technical Summary: Joint Testability Architecture (JTA) for Safety-Critical Software

1. Problem Statement

In safety-critical software systems (e.g., flight control, automated driving), validation adequacy is often hindered not by a lack of test cases, but by the inability to stably construct critical scenarios, organize execution evidence into verdict-ready forms, or attribute abnormal outcomes to actionable engineering boundaries. Traditional testability research focuses primarily on the software artifact itself, treating testability as a static property of code (controllability and observability). However, this artifact-centric view is insufficient for safety-critical scenarios involving failsafe activation, mode migration, or multi-component coordination. It fails to address whether the combined capability of the scenario, the test system, and the system under test (SUT) is sufficient to support adequate validation. Consequently, "code being testable" does not guarantee that "scenario-based validation is adequate."

2. Methodology: The Joint Testability Architecture (JTA)

JTA reframes testability as a scenario-conditioned capability of a joint system. It proposes a formal architecture and a five-step analysis workflow to identify and bridge capability gaps.

2.1 Formal Model

The architecture is defined as $JTA = \langle D_\Sigma, D_T, D_S, BC, BO, BI, L \rangle$ , comprising:

Three Domains:
- Scenario Domain ( $D_\Sigma$ ): Organizes validation objectives, constraints, and criticality.
- Test-System Domain ( $D_T$ ): Encompasses stimulation, evidence collection, and verdict generation capabilities.
- System-Under-Test Domain ( $D_S$ ): Includes the functional behavior and the exposed validation hooks (control points, evidence points, responsibility boundaries).
Three Bridges: Architectural patterns that connect the domains to resolve capability mismatches:
- Control Bridge ($BC$): Translates scenario preconditions into reproducible parameter settings and fault injections.
- Evidence Bridge ($BO$): Organizes raw logs and states into structured, temporally aligned evidence for automated verdicts.
- Attribution Bridge ($BI$): Maps anomalous outcomes to specific trigger sources and responsibility boundaries to enable remediation.
Closed Loop ( $L$ ): An iterative process of analysis, design, evaluation, and refinement.

2.2 Core Concepts

Joint Testability System ( $JTS_\sigma$ ): The unit of analysis is not the software alone, but the triplet $\langle \sigma, T_\sigma, S_\sigma \rangle$ representing a specific scenario, the participating test capabilities, and the relevant SUT boundary.
Scenario Contracts ( $SC(\sigma)$ ): An eight-tuple specification $\langle g, p, u, e, o, r, \kappa, \Theta \rangle$ that explicitly defines the validation objective, preconditions, stimulation mode, minimum evidence, verdict rules, a set of distinguishable fault sources ( $r$ ), criticality, and minimum thresholds for controllability, observability, and isolability.
Joint Capability Assessment:
- Controllability ( $C$ ), Observability ( $O$ ), and Isolability ( $I$ ): Measured for both the SUT ($CS, OS, IS$) and the Test System ($CT, OT, IT$).
- Joint Capability ($JTCap$): Defined as the minimum of the SUT and Test System capabilities for each dimension (e.g., $JC = \min(CS, CT)$ ), reflecting the "weakest link" principle.
- Adequacy ($Adeq$): A scenario is adequate only if all three joint dimensions meet the minimum thresholds defined in the contract.
- Blind Spots: Scenarios where $Adeq(\sigma) = 0$ .

2.3 Five-Step Workflow

Identify Critical Scenarios: Determine the set $\Sigma$ and assign criticality weights.
Instantiate Scenario Contracts: Explicitly define objectives, evidence requirements, and the set of fault sources to be distinguished.
Assess Bilateral Capabilities: Evaluate $Cap_S$ and $Cap_T$ against the contract requirements.
Compute Gaps and Identify Blind Spots: Calculate the capability gap vector $Gap(\sigma)$ and identify scenarios failing adequacy.
Bridge Design and Re-evaluation: Map gaps to specific design actions on the Control, Evidence, or Attribution bridges, then re-evaluate.

3. Key Contributions

Joint Testability System Model ( $JTS_\sigma$ ): Expands the object of validation from an isolated artifact to a scenario-instantiated joint entity comprising the scenario, test system, and SUT.
Scenario Contracts with Fault-Source Sets: Introduces a formal contract that includes a set of distinguishable fault sources ( $r$ ), transforming isolability from a generic concept into a concrete design constraint for attribution.
Bridge-Oriented Design Method: Proposes a "three domains, three bridges, and a closed loop" framework that maps capability gaps directly to executable architectural design actions (e.g., adding a timing controller to the Control Bridge or cause-code extensions to the Attribution Bridge).
Illustrative Analysis of ArduPilot: Demonstrates the application of JTA on the ArduPilot failsafe validation, identifying specific blind spots in link-loss and state-estimation anomaly scenarios.

4. Results: ArduPilot Case Study

The authors applied JTA to three failsafe scenarios in the open-source ArduPilot project: RC link loss (S1), GCS heartbeat timeout (S2), and EKF state-estimation anomalies (S3).

S1 (RC Link Loss): Rated as Adequate (1). The test system and SUT jointly support the scenario, though some bottlenecks exist in evidence structuring.
S2 (GCS Heartbeat Timeout): Rated as Inadequate (0). While the scenario can be triggered, the Attribution Bridge is weak; current logs cannot distinguish between genuine timeouts and test-toolchain errors.
S3 (EKF Anomalies): Rated as Inadequate (0). This scenario reveals bottlenecks in both Controllability (difficulty in decoupling perturbation strength and timing) and Isolability (ambiguity in distinguishing between EKF health failures and concurrent GPS degradation).

System-Level Metric: The criticality-weighted adequacy coverage ($JT(JTS)$) for the analyzed scenarios was calculated at 28.6%, highlighting that high-criticality scenarios (like EKF anomalies) remain significant validation blind spots despite mature infrastructure for simpler scenarios.

Design Recommendations: The analysis yielded specific bridge-level actions, such as:

Adding a "triadic attribution view" (timeout event–cause code–link status) for S2.
Implementing parameterized perturbation templates and injection-enabled replay adapters for S3 to improve controllability.

5. Significance and Claims

The paper claims that JTA provides an architectural basis for modeling, designing, and assessing scenario-based validation, rather than replacing existing testing or safety-analysis techniques.

Shift in Perspective: It moves the focus from "is the code testable?" to "is the joint system capable of validating this specific scenario?"
Actionable Output: It translates vague engineering impressions of "hard to test" into dimension-traceable, object-localizable, and action-executable design gaps.
Proactive Validation: It advocates for treating validation capability as an architectural quality attribute that should be designed alongside functional capability, rather than checked only after implementation.
Complementarity: JTA complements existing methods like STPA (which identifies scenarios) and fault injection (which constructs abnormalities) by providing the missing link for assessing joint adequacy and attribution convergence.

The authors maintain modesty regarding their claims, noting that the case study is illustrative and relies on ordinal (H/M/L) assessments rather than quantitative performance metrics, and that the applicability to closed industrial systems with restricted interfaces requires further validation.