💻 computer science

Model-Driven Requirements Configuration with Three-Valued Uncertainty Scoring

This paper presents a neuro-symbolic multi-agent architecture that combines a Large Language Model with a deterministic symbolic validator and a three-valued uncertainty scoring framework to eliminate structural inconsistencies in LLM-generated requirements while formally quantifying decision uncertainty for safer deployment in requirements engineering.

Original authors: Ahmed Ibrahim

Published 2026-07-30

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Ahmed Ibrahim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a custom robot, but instead of drawing blueprints, you just talk to a very chatty, super-smart robot friend who loves to make things up. You say, "I want a robot that can cook and clean," and your friend excitedly starts listing parts. But because it's so eager, it might accidentally suggest a part that doesn't fit, or two parts that explode if you put them together, or even invent a part that doesn't exist at all. This is the tricky world of Requirements Engineering: the process of figuring out exactly what a piece of software or a machine needs to do. For a long time, experts have used strict, mathematical "rulebooks" (called formal models) to make sure every part fits perfectly, but these rulebooks are hard for humans to read. Recently, we've started using Large Language Models (LLMs)—the same kind of AI that writes poems and answers questions—to help us describe what we want in plain English. The problem? These AI friends are great at talking but terrible at following the strict rules of the rulebook, often creating designs that look good but fall apart when you try to build them.

This paper introduces a clever new team-up to solve that problem. The authors created a system where the chatty AI (the LLM) acts as a creative brainstormer, suggesting ideas based on your natural language description, while a strict, unyielding "rule-checker" (a symbolic validator) stands guard. Think of it like a creative chef and a health inspector working together in a kitchen. The chef suggests wild, delicious recipes, but the inspector immediately checks if the ingredients are safe and if the steps follow the law. If the chef suggests mixing poison with cake, the inspector stops it instantly. The paper shows that this team-up successfully fixes almost all the mistakes the AI makes, turning messy, impossible ideas into solid, buildable plans. They also introduced a new way to measure the AI's confidence, distinguishing between things the AI must pick, things it can pick freely, and things it got wrong.

The Story of the Creative Chef and the Strict Inspector

In the world of software, before you write a single line of code, you have to decide exactly what the program should do. This is called Requirements Engineering. Imagine you are designing a "Smart Home." You need to decide: Does it have a backup battery? Does it use Wi-Fi or Bluetooth? Can you control the lights with your voice? If you get these choices wrong, the whole house might not work.

Traditionally, experts used a rigid system called OOMRAM (Object-Oriented Method for Requirements Authoring and Management). Think of OOMRAM as a giant, complex flowchart or a tree of choices. It has strict rules: "If you pick 'Smart Home,' you must pick a power source. If you pick 'Wi-Fi,' you cannot pick 'Bluetooth' at the same time." It's like a video game character creator where you can't equip a sword and a shield if the game says they are mutually exclusive. The problem is that these flowcharts are hard for humans to navigate. You can't just say, "I want a house that feels cozy and safe," and expect the flowchart to understand you. You have to know the exact names of every single option.

Enter the Large Language Model (LLM). This is the AI that can understand your sentence, "I want a cozy house with voice control," and guess which options you probably mean. It's like a super-fast, creative assistant who speaks your language. But here's the catch: the AI is a bit of a daydreamer. It might invent a "Quantum Battery" that doesn't exist, or it might forget that you can't have both Wi-Fi and Bluetooth, leading to a design that is logically broken.

The Neuro-Symbolic Team-Up

The authors of this paper built a system to fix this. They created a team of four "agents" (little computer programs) that work together:

The Navigator: This agent looks at the giant flowchart (the OOMRAM lattice) and decides which part to look at next. It's like a tour guide pointing at the next door in a maze.
The Interpreter (The AI): This is the LLM. It listens to your "project vision" (your description of what you want) and suggests which buttons to press on the flowchart. It tries to guess which options fit your description.
The Validator (The Inspector): This is the strict, rule-following part. It doesn't use AI; it uses hard math. It checks every single suggestion the AI makes against the flowchart's rules. Did the AI pick two things that can't go together? Did it forget a mandatory part? If the answer is "yes," the Validator says, "Nope, try again," and sends the AI back to fix its mistake.
The Scribe: Once everything is approved, this agent writes down the final, perfect list of requirements.

The magic happens because the AI and the Inspector talk to each other in a loop. The AI suggests, the Inspector checks, and if there's an error, the AI tries again. This keeps happening until the design is perfect.

The Three-Valued Scorecard: Truth, Maybe, and False

One of the coolest parts of this paper is how they measure the AI's work. Usually, we just say an answer is "Right" or "Wrong." But the authors realized that's too simple. Sometimes, the AI picks something that is allowed but not strictly required. To handle this, they invented a Three-Valued Scorecard:

Truth (T): The AI picked something that had to be picked. For example, if the vision says "Smart Home," the system must pick a power source. If the AI picks it, that's a Truth.
Indeterminacy (I): The AI picked something that was allowed, but the vision didn't force it. Maybe the vision just said "Smart Home," and the AI chose "Wi-Fi" over "Bluetooth." Both are valid, but the vision didn't say which one. This is Indeterminacy. It's a "free choice."
Falsity (F): The AI picked something that broke the rules. Like picking "Wi-Fi" and "Bluetooth" at the same time when the rules say you can't. This is Falsity.

By using this scorecard, the researchers could see exactly where the AI was guessing and where it was following orders.

What They Found

The team tested their system with 37 different project visions across 11 different types of applications, like record-keeping systems, smart homes, and car entertainment systems. They used a "lightweight" AI model (Llama 3.1 8B) for most tests.

Here is what happened:

The Fix Rate: Without the strict Inspector, the AI made huge mistakes. But with the Inspector, the system fixed almost all of them. In 35 out of 37 cases (94.6%), the final result had zero structural errors.
The Leftovers: In the remaining 2 cases, the AI got stuck in a loop and couldn't fix a specific type of error before they stopped the test. This left only 6 tiny errors out of over 1,500 decisions (0.39%).
The "Maybe" Zone: The researchers found that about 24.7% of all the decisions the AI made were "Indeterminacy." This means the AI was exercising its freedom to choose between valid options, which is actually a good thing! It shows the system knows the difference between "must do" and "can choose."
The Stronger AI: When they tested the system with a much smarter, "frontier" AI model (NVIDIA Nemotron 3 Ultra), it got 100% of the visions perfect, with zero errors.

Why This Matters

The paper argues that simply letting an AI write requirements is dangerous because it might invent things that don't work. But by putting a strict, mathematical "Inspector" in charge of checking the rules, you can let the AI be creative without worrying about it breaking the laws of logic.

The authors also showed that this system is fast. The "Inspector" takes less than 0.4% of the total time, meaning it doesn't slow things down. They also proved that the system scales well: as the list of options gets bigger, the number of times the AI has to talk to the system only goes up in a straight line, not a crazy curve.

In short, this paper proves that you can have your cake and eat it too: you can use the flexible, natural language of AI to describe what you want, while a strict, unbreakable rulebook ensures that what you get is actually buildable and correct. It turns the "daydreaming" AI into a reliable engineer.

Technical Summary: Model-Driven Requirements Configuration with Three-Valued Uncertainty Scoring

1. Problem Statement

Requirements engineering (RE) is inherently costly and error-prone. While formal domain models like the Object-Oriented Method for Requirements Authoring and Management (OOMRAM) provide rigorous mathematical structures for managing variability in software product lines, their traditional application suffers from an "exact-match bottleneck." These formalisms require analysts to navigate complex lattices using exact requirement identifiers or rigid syntax, preventing the use of natural-language project descriptions (visions) for elicitation.

Conversely, Large Language Models (LLMs) offer the flexibility to interpret natural language but lack formal correctness guarantees. When applied to RE, pure LLM approaches frequently generate structurally invalid requirements, logical inconsistencies (e.g., selecting mutually exclusive options), and hallucinated identifiers. Existing Neuro-Symbolic approaches often verify logic ex-post (after generation) or rely on heuristic mitigation, failing to provide the deterministic enforcement required for high-assurance requirements engineering.

2. Methodology

The paper proposes a neuro-symbolic multi-agent architecture that operationalizes the OOMRAM lattice to bridge natural-language visions with formal structural constraints. The system is designed for the feature-selection phase of product-line requirements elicitation, where the goal is to identify which requirements from a pre-engineered lattice are mandated or compatible with a stakeholder's vision.

2.1. System Architecture

The system, implemented using LangGraph, comprises four agents:

Navigator: Performs a depth-first traversal of the OOMRAM lattice, selecting the next discriminant node to process. It utilizes a frontier-based strategy to ensure linear scaling and avoid redundant processing.
Interpreter (LLM): Acts as a non-deterministic heuristic. It translates the project vision and current context into a proposed set of child requirement IDs (output as JSON). It operates at temperature 0 to minimize stochasticity.
Validator (Symbolic): A deterministic Python function that enforces structural constraints immediately after every Interpreter proposal. It checks the selection set $S$ $S$ against the global consistency predicate $\Psi(S)$ $Ψ (S)$ , which includes four axiomatic invariants:
1. Core Invariant: Mandatory selection of Core nodes if their parent is active.
2. Single-Adaptor (SA) Invariant: Exactly one child must be selected for SA nodes (XOR logic).
3. Multiple-Adaptor (MA) Invariant: At least one child must be selected for MA nodes (OR logic).
4. Structural Integrity: No orphaned nodes; every selected non-root node must have a selected parent.
Scribe: Compiles the final validated set into a formatted specification.

2.2. The Correction Loop

If the Validator detects a violation, it rejects the proposal and returns a detailed error message to the Interpreter. The Interpreter then revises its proposal using this feedback. This loop continues until constraints are satisfied or a "call budget" (capped at 150 LLM calls per vision) is reached.

2.3. Three-Valued Uncertainty Scoring

To address the limitation of binary "correct/incorrect" labeling, the authors introduce a three-valued framework $\langle T, I, F \rangle$ applied at the Validator output:

Truth ( $T$ ): The selection is structurally valid and explicitly mandated by the vision (or a gold-standard specification).
Indeterminacy ( $I$ ): The selection is structurally valid but not explicitly mandated. This represents legitimate discretionary choices (e.g., selecting a specific protocol from a set of valid adapters).
Falsity ( $F$ ): The selection was rejected by the Validator due to a structural violation.

This scoring allows the system to quantify the "legitimate freedom" inherent in the requirements lattice versus actual errors.

3. Key Contributions

Neuro-Symbolic Architecture: A system that offloads structural logic to a deterministic symbolic layer, making formal OOMRAM lattices practical for natural-language elicitation while guaranteeing structural conformance.
Three-Valued Evaluation Framework: A novel metric to distinguish between mandated requirements, valid discretionary choices, and structural errors, providing a formal way to measure neural uncertainty.
Empirical Validation: A comprehensive evaluation across 37 natural-language project visions spanning eleven application families (e.g., Smart Home, eRecord Keeping, Automotive IVI).
Formal Proof and Baseline Comparison: A formal proof of validator convergence and a comparative analysis demonstrating the system's superiority over unconstrained LLMs and standard NLP4RE strategies (RAG, Self-Consistency, Multi-agent Debate).

4. Results

The system was evaluated on an Apple M1 Pro machine using the Llama 3.1 8B model, with a replication using the NVIDIA Nemotron 3 Ultra frontier model.

Structural Conformance (RQ1): The system eliminated structural inconsistencies in 35 out of 37 cases (94.6%). The remaining two cases contained only 6 unresolved structural errors (0.39% of total decisions), attributed to hitting the iteration limit before convergence. Under the frontier model (Nemotron 3 Ultra), the system achieved 100% convergence with 0% structural falsity.
Uncertainty Quantification (RQ2): The three-valued analysis revealed that 24.7% of all decisions were Indeterminate ( $I$ ). These were structurally valid choices not explicitly mandated by the vision, concentrated primarily in Multiple-Adaptor (MA) and Optional nodes. This confirms that the system correctly identifies where stakeholder intent is underspecified.
Baseline Comparison (RQ3): Without the symbolic validator, unconstrained LLM strategies resulted in structural falsity rates ranging from 62.2% to 94.6%. The neuro-symbolic system reduced this to 5.4% (Llama 3.1 8B) and 0.0% (Nemotron 3 Ultra). Notably, a RAG baseline suffered a 94.6% falsity rate due to retrieval anchoring bias, where the LLM selected leaf nodes without their required parent paths.
Scalability (RQ4): The number of LLM calls scales near-linearly with lattice size ( $R^2 = 0.69$ ), confirming the efficiency of the frontier-based pruning strategy. However, wall-clock latency is dominated by network/model-serving overhead rather than lattice complexity. The symbolic validator adds negligible overhead (<0.4% of total latency).

5. Significance and Claims

The paper claims that offloading structural integrity to a deterministic symbolic layer successfully guarantees structural conformance, effectively transforming inconsistent neural outputs into verified, structurally sound specifications.

Safety in Deployment: The architecture provides a general design pattern for safe LLM deployment in safety-critical or quality-sensitive domains by separating the neural heuristic from the symbolic constraint checker. This ensures that the LLM component can be upgraded without compromising formal guarantees.
Actionable Feedback: The three-valued classification provides practitioners with immediate feedback. High indeterminacy signals underspecified visions requiring analyst clarification, while persistent falsity (though rare) pinpoints specific model-level misunderstandings or lattice issues.
Formal Rigor: Unlike heuristic approaches, this system offers a deterministic guarantee: for any run that converges within the call budget, the final output is guaranteed to satisfy all lattice constraints.

The authors maintain a modest stance, acknowledging that the system cannot detect semantic hallucinations (structurally valid but irrelevant selections) and relies on pre-engineered OOMRAM lattices. They note that while the symbolic layer guarantees structural correctness, the LLM's ability to follow error messages and converge remains model-dependent, as evidenced by the difference in performance between the 8B and frontier models. Future work is proposed to automate lattice extraction, integrate semantic routing for family selection, and move from crisp three-valued scoring to continuous probability-based uncertainty visualization.