Technical Summary: LLM Risk Assessment Framework for Systems Engineering

Problem Statement

The integration of Large Language Models (LLMs) into Systems Engineering (SE) presents a dichotomy between the rapid, unpredictable evolution of generative AI and the rigorous, stability-oriented requirements of established engineering lifecycles (e.g., the V-Model). While LLMs offer transformative potential for requirements engineering, software development, and decision support, their adoption is currently hindered by a lack of systematic risk assessment methodologies.

Organizations face significant challenges in integrating LLMs due to:

• Inconsistent Integration: Current implementations are often isolated from structured business processes and cyber-physical system lifecycles.
• Unknown Failure Modes: Issues such as hallucinations, probabilistic behavior, prompt brittleness, and limited interpretability complicate verification and validation.
• Human and Organizational Gaps: Risks include overreliance on generated content, loss of domain knowledge, difficulties in distinguishing human vs. machine authorship, and a lack of standardized governance or data-driven development practices.

Existing literature addresses specific risks or applications but fails to provide a unified, domain-agnostic means to assess the risk of LLM use across the diverse spectrum of engineering tasks.

Methodology

The paper introduces the LLM Risk Assessment Framework (LRF), a structured approach designed to classify and evaluate LLM applications within SE environments. The methodology relies on a two-dimensional classification matrix:

1. Autonomy Dimension

Inspired by SAE levels of driving automation, this dimension defines the degree of independence in LLM operation, ranging from human-centric to fully automated:

• Level 0 (Assisted): The LLM acts as a supportive tool within the user's context. The human retains full control and makes all decisions.
• Level 1 (Guided): The LLM suggests actions or alternatives (e.g., design proposals) that require explicit human confirmation and review before execution.
• Level 2 (Supervised): The LLM performs defined tasks (e.g., generating requirement sets) independently, but human users monitor the process and intervene as necessary.
• Level 3 (Fully Automated): The LLM operates without human oversight, directly executing actions and making decisions. Outputs are used without further supervision.

2. Impact Dimension

This dimension assesses the severity of consequences resulting from incorrect, misleading, or misaligned LLM outputs, analogous to the "severity" parameter in safety standards like IEC 61508 or ISO 26262:

• Low Impact: Errors have minimal or reversible consequences for system performance, safety, or compliance (e.g., documentation support).
• Medium Impact: Errors indirectly affect the system or engineering tasks, potentially propagating to subsequent steps and affecting cost, performance, or quality.
• High Impact: Errors directly influence system design, behavior, or sensitive data, leading to severe consequences for safety, reliability, or compliance.

Risk Classification Mechanism

By mapping specific use cases onto the intersection of Autonomy and Impact, the LRF assigns a risk level (Minimal, Low, Medium, High). This classification dictates the necessary governance measures:

• Minimal Risk: Standard quality management with human-in-the-loop plausibility checks.
• Low Risk: Continuous monitoring and explainable AI mechanisms.
• Medium Risk: Explicit validation steps, explainable AI features, and a mandatory human-in-the-loop approach for confirmation.
• High Risk: Comprehensive validation, fallback mechanisms, robust explainability, and structured assurance processes similar to traditional safety case development.

Key Contributions

The paper makes three primary contributions to the field of Systems Engineering and AI assurance:

1. Proposal of the LRF: A structured, domain-agnostic model that classifies LLM applications based on autonomy levels and system impact.
2. Systematic Risk Assessment: A method to link autonomy and impact to corresponding risk levels, enabling the consistent evaluation and comparison of heterogeneous LLM use cases.
3. Support for Safe Integration: A derivation of suitable mitigation measures for each risk level, guiding organizations toward reliable and transparent LLM adoption.

Results and Illustrative Applications

The paper demonstrates the framework's applicability through two representative examples:

• Requirements Checker (Low Risk): An LLM that analyzes requirements for completeness and consistency and suggests improvements.

  • Classification: Autonomy Level 1 (Guided) + Medium Impact (affects downstream decisions).
  • Outcome: Classified as Low Risk. Since the human remains in charge and approves all changes, continuous monitoring is sufficient.

• Legal Case Assessment (High Risk): An LLM system that determines the applicability of laws to specific situations for users with limited legal expertise.

  • Classification: Autonomy Level 3 (Fully Automated) + High Impact (potential for compliance violations and financial loss).
  • Outcome: Classified as High Risk. The framework dictates that such a system requires comprehensive validation, fallback mechanisms, and robust explainability. If used operationally, human oversight (e.g., legal expert verification) must be reintroduced.

Significance and Claims

The paper claims that the LRF provides a foundational basis for the risk-aware adoption of LLMs in complex engineering environments. Its significance lies in:

• Bridging the Gap: Aligning the agility of AI technologies with the rigor of established engineering lifecycles (reliability, traceability, and controlled process integration).
• Standardization: Representing a first step toward standardized AI assurance practices in systems engineering, moving beyond isolated implementations to unified governance.
• Scalability: Enabling organizations to identify appropriate validation strategies and levels of human oversight, thereby supporting both safe experimentation and scalable deployment.

The authors note that the current work defines how to structure risk assessment but does not yet provide methods to measure or quantify the maturity of an LLM application, identifying this as an area for future research. The framework is intended to evolve alongside AI governance concepts to support the sustainable integration of generative AI into critical engineering systems.