🤖 machine learning

From Failure to Alignment: A Requirements Engineering Framework for Machine Learning Systems

This paper proposes REAL, a model-based Requirements Engineering framework that integrates data, model, and system requirements while leveraging failure analysis and iterative refinement to ensure machine learning systems align with stakeholder needs and are trustworthy.

Original authors: Amel Bennaceur, Gopi Krishnan Rajbahadur, Prince Mercy, Bashar Nuseibeh, Faeq Alrimawi

Published 2026-07-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Amel Bennaceur, Gopi Krishnan Rajbahadur, Prince Mercy, Bashar Nuseibeh, Faeq Alrimawi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to drive a car. You tell it, "Never hit a pedestrian." Simple, right? But in the real world, things get messy. The robot might fail to see a small child in the rain, or it might brake too hard for a plastic bag blowing across the road.

This paper, titled "From Failure to Alignment," introduces a new way to build these smart systems. The authors call their method REAL (Requirements Engineering for mAchines that Learn - and Fail).

Here is the core idea, broken down with simple analogies:

1. The Problem: The "Perfect World" Trap

Traditional software is like a recipe: if you follow the steps exactly, you get the same cake every time. Machine Learning (ML) is different. It's more like teaching a dog to fetch. You show it a ball, it learns to fetch. But if you throw a stick, or a frisbee, or a shoe, the dog might get confused.

The authors argue that when we build these "learning" systems, we often assume the world will look exactly like the training data. But the real world is full of surprises (fog, weird clothing, small children). When the system fails, engineers usually just try to "fix the bug." The authors say: Stop treating failure as a bug. Treat it as a clue.

2. The Solution: The REAL Framework

The REAL framework is a three-step loop that turns "failures" into "better instructions." Think of it as a detective story where the failure is the crime scene, and the goal is to figure out what rule was missing.

Step 1: Hunt for the "What-Ifs" (Scenario Exploration)

Instead of just testing the car on a sunny day, the framework uses a "grammar" (a set of rules) to automatically generate thousands of weird, tricky, or rare situations.

Analogy: Imagine a video game level generator that creates millions of different weather conditions, pedestrian sizes, and crossing angles. It specifically looks for the moments where the car almost crashes or misses a person.
The Goal: Find the "valid failures"—situations that are realistic but where the system messed up.

Step 2: The "Why" Investigation (Obstacle Analysis)

Once a failure is found, the framework asks: Why did this happen? It doesn't just say "the sensor failed." It maps the failure back to the original rules.

Analogy: If the car misses a small child, the framework doesn't just say "fix the camera." It realizes the original rule was "See all pedestrians." The failure reveals a hidden assumption: "We assumed all pedestrians are big enough to be seen easily."
The Result: The failure is reclassified as an Obstacle. It's like putting a sign on a map that says, "Here is where our current map is wrong."

Step 3: The Multi-Layer Fix (Mitigation)

Now that we know what is wrong, we fix it. But the authors say you can't just fix one thing; you have to look at four different layers, like fixing a house:

Data Layer (The Ingredients): Maybe the robot never saw a child in the rain during training. Fix: Show it more pictures of children in the rain.
Model Layer (The Brain): Maybe the robot's "brain" is too slow or confused. Fix: Change the math inside the brain to be more careful.
System Layer (The Body): Maybe the brain is slow, so the car needs a backup plan. Fix: Add a rule that says, "If you aren't 100% sure, just slow down gently."
Requirement Layer (The Contract): Maybe the original rule was impossible. Fix: Admit that the car can't see in thick fog and update the rule to say, "Don't drive in fog."

3. The Real-World Test: The Braking Car

To prove this works, the authors tested REAL on an autonomous braking system (a car that stops for pedestrians).

The Setup: They used a simulator (a video game world) to create scenarios with children, adults, fog, and different clothing.
The Discovery: The system kept missing small children, especially in bad weather.
The REAL Process:
- They found the failure (missing the child).
- They identified the obstacle (the system didn't account for "small size" and "low visibility").
- They applied fixes: They added more training data of children, adjusted the car's confidence thresholds, and clarified the rules about when the car is allowed to drive.
The Result: By using this loop, they reduced the number of missed children from 100% (in the worst cases) down to 8%, and they made the rules for the car much clearer.

The Big Takeaway

The paper claims that failure is not the end of the road; it is the map.

Instead of trying to build a perfect system that never fails (which is impossible), the REAL framework uses failures to constantly update the system's "contract" with the real world. It connects the dots between:

What the humans want (Safety).
What the world is actually like (Fog, small kids).
What the computer is actually doing (Missing things).

By treating failure as a diagnostic tool, the framework helps engineers build systems that are honest about their limits and safer in the real world.

What the paper does NOT claim:

It does not claim this solves every problem in AI.
It does not claim this works perfectly for all types of AI (like chatbots or medical diagnosis) yet, though they plan to try.
It does not promise that the system will never fail again; it promises that the system will be better at understanding why it failed and how to adjust its rules.

Technical Summary: From Failure to Alignment: A Requirements Engineering Framework for Machine Learning Systems

Problem Statement

Organizations developing Machine Learning Systems (MLS) face significant challenges in ensuring these systems are trustworthy and aligned with stakeholder needs. Unlike traditional software, MLS behavior is learned from data rather than explicitly programmed, making performance distribution-dependent and failure inherent due to generalization and uncertainty. Current Requirements Engineering (RE) practices struggle to systematically model and reason about MLS requirements because:

Underspecification: Operational domains are open-ended, and complete upfront specification is unrealistic.
Implicit Assumptions: Domain assumptions (e.g., data coverage, sensing limits) are often implicit, leading to misalignment between stakeholder goals, domain assumptions, and system behavior when failures occur.
Lack of Integration: While failure detection is common in ML testing, there is a gap in systematically integrating failure discovery into the RE process to drive requirement refinement. Existing approaches often treat failures as testing artifacts or debugging issues rather than diagnostic evidence for refining requirements and domain assumptions.

The paper addresses three Research Questions (RQs):

RQ1: How can reasoning about requirements be operationalised for MLS?
RQ2: How can failure discovery and mitigation be systematically integrated into the RE process?
RQ3: To what extent does failure-driven, cross-layer adaptation measurably improve requirement satisfaction?

Methodology: The REAL Framework

The authors propose REAL (Requirements Engineering for mAchines that Learn - and Fail), a model-based framework that treats failure as a first-class construct. REAL operates on three core principles: weaving data, model, and system requirements; using failure to drive alternative requirement exploration; and iterative, traceable refinement.

The framework establishes an iterative alignment loop involving three phases:

1. Identifying Failure through Scenario Exploration

Instead of assuming uniform requirement satisfaction, REAL operationalizes the domain as a parametrized scenario space $D(\theta)$ .

Grammar-Guided Generation: It uses Grammatical Evolution (GE) with a Domain Specific Language (DSL) defined by a formal grammar to automatically generate structured scenarios (actors, environments, behaviors).
Admissibility Filtering: The framework distinguishes between valid failures (occurring within admissible domain assumptions) and spurious failures (triggered by out-of-scope scenarios). Only valid failures, where $D(\theta), S \not\models R$ and admissibility constraints hold, are propagated to analysis.

2. Analysing Failure through Obstacle Analysis

Failures are reinterpreted as obstacles in a KAOS-style goal model.

Counterexample Interpretation: A failure trace is treated as a counterexample to the satisfaction relation.
Obstacle Derivation: Failures are clustered to identify recurring patterns, elevating empirical counterexamples into explicit obstacle conditions ( $O$ ) where $O \Rightarrow \neg R$ . This diagnoses misalignments between stakeholder goals, domain assumptions, and system behavior.

3. Mitigating Failure through Multi-Layer Adaptation

REAL distinguishes four layers of adaptation to restore the satisfaction relation ( $D, S \models R$ ):

Data-level: Augmenting training distributions or rebalancing classes to address distributional gaps.
Model-level: Adjusting architectures, hyperparameters, or decision thresholds to reshape the learned decision boundary.
System-level: Introducing confidence-aware control logic, fail-safe behaviors, or redundancy without modifying the learned model itself.
Requirement-level: Refining the requirements ( $R$ ) or domain assumptions ( $D$ ) themselves (e.g., scoping the Operational Design Domain) when failures reveal underspecified needs or unrealistic assumptions.

This process is iterative; new adaptations may expose new violations, requiring further refinement of the triple $(D, S, R)$ .

Key Contributions

The paper makes four primary contributions:

Operational Linkage: It connects stakeholder requirements to executable domain variability via grammar-guided scenario exploration, enabling systematic discovery of requirement-relevant counterexamples.
Obstacle-Based Interpretation: It reinterprets empirical failures as obstacles in a goal model, providing a principled mechanism to diagnose misalignment.
Multi-Layer Mitigation Strategy: It defines and evaluates coordinated adaptation across data, model, system, and requirement layers, explicitly showing how mitigation decisions affect requirement satisfaction.
Autonomous Driving Demonstrator: It provides a publicly available replication package demonstrating the framework on an automated braking system using CARLA, Scenic, and evolutionary scenario generation.

Experimental Results

The framework was evaluated using an automated braking system with a pedestrian detection component (YOLOv5) and a deterministic braking controller.

Experimental Setup:

Environment: CARLA simulator with Scenic for scenario specification.
Generators: GRAPE (Grammatical Evolution) for scenario generation.
Focus: Systematic variation of pedestrian size, distance, direction, clothing, and weather (fog).

Findings:

Failure Identification: The framework successfully identified valid requirement violations, such as missed detections of child pedestrians (size-related), failures under low visual saliency (clothing contrast), and degraded performance in fog (visibility).
Obstacle Analysis: Failures were grouped into three distinct obstacle classes: PedestrianSizeTooSmall, VisualSaliency, and AdverseWeather.
Mitigation Effectiveness:
- Model-level (M1): Switching to a larger model (YOLOv5m) improved accuracy (+17%) but introduced latency-induced failures in short-distance scenarios.
- Data-level (M2): Targeted fine-tuning on child-pedestrian data eliminated size-related failures and improved fog performance significantly (collision rate dropped from 100% to 0% in nominal weather).
- System-level (M4): Implementing confidence-aware braking reduced residual child-collision rates from 100% to 8% without retraining.
- Requirement-level (M5): Explicitly scoping the Operational Design Domain (ODD) and defining quantitative thresholds clarified the limits of safety guarantees.
Overall Impact: Coordinated cross-layer adaptation reduced child-collision rates from 100% (baseline) to 8% and eliminated nominal-weather failures. The study demonstrated that isolated model optimization is insufficient; alignment requires addressing the interplay between data, model, system, and requirements.

Significance and Claims

The paper claims that REAL provides a systematic process for integrating testing outcomes into RE artifacts. Its significance lies in:

Shifting the Perspective on Failure: Moving from viewing failure as a technical defect to viewing it as diagnostic evidence of misalignment between requirements, assumptions, and implementation.
Operationalizing Alignment: Providing a concrete mechanism to link high-level stakeholder goals to low-level system behaviors and domain assumptions through obstacle analysis.
Cross-Layer Coordination: Demonstrating that restoring alignment in MLS requires coordinated adaptation across multiple layers (data, model, system, requirements) rather than isolated fixes.

The authors maintain a modest stance, acknowledging that the evaluation is limited to a simplified safety-critical example and that the process does not guarantee convergence in all cases. They note that while the framework structures the process, human judgment remains essential for interpreting failures and selecting mitigation strategies. Future work is proposed to extend REAL to Large Language Models (LLMs), incorporate stakeholder values more deeply, and investigate the convergence conditions of the iterative alignment loop.