💻 computer science

Grammar-Constrained Refinement of Safety Operational Rules Using Language in the Loop: What Could Go Wrong

This paper introduces a framework that combines counterfactual reasoning with grammar-constrained refinement to update safety operational rules for cyber-physical systems, demonstrating that while this approach resolves inconsistencies in autonomous driving rules, the use of Large Language Models (LLMs) necessitates rigorous grammar enforcement and semantic validation to prevent overfitting and unsafe refinements.

Original authors: Khouloud Gaaloul, Zaid Ghazal, Madhu Latha Pulimi, Sam Emmanuel Kathiravan

Published 2026-04-28

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Khouloud Gaaloul, Zaid Ghazal, Madhu Latha Pulimi, Sam Emmanuel Kathiravan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot how to drive a car. To keep it safe, you give it a "Rulebook." One rule might say: "If you are driving fast, stay at least 50 feet away from the car in front of you."

But as the world changes—maybe the robot gets a new sensor, or it starts driving in different weather—that rulebook might become outdated. Suddenly, the robot follows the rule, but it still almost crashes. The rule is "inconsistent" with reality.

This paper describes a new way to fix those rules automatically using AI, but with a very important "safety leash" attached.

The Problem: The "Lazy Student" Problem

When we use standard AI (like ChatGPT) to fix rules, it acts like a lazy student trying to pass a test. If you show it a mistake, it might "fix" the rule by making it so incredibly strict that the robot becomes useless (e.g., "Never move unless the road is perfectly empty and the sun is shining"). This is called overfitting. It fixes the error, but it breaks the actual job of driving.

The Solution: The "Detective and the Architect"

The researchers created a two-step system to fix rules without breaking them.

Step 1: The Detective (Counterfactual Analysis)
Before fixing anything, the system acts like a detective. It doesn't just look at the crash; it asks, "What is the tiniest possible change that would have prevented this?" It looks for the "boundary line"—the exact moment where a "Safe" situation turns into a "Danger" situation. This ensures the AI isn't just guessing; it has evidence.

Step 2: The Architect (Grammar-Constrained LLM)
Now, the system brings in a Large Language Model (the AI) to rewrite the rule. But here is the twist: the AI isn't allowed to just write whatever it wants. It is forced to follow a strict "Grammar."

Think of this like a Lego set. The AI can build a new castle (a new rule), but it is only allowed to use the specific bricks provided in the box. It can’t use glue, it can’t use clay, and it can’t use wood. It must use the specific "logic bricks" (like greater than, less than, or and/or) defined by the engineers. This ensures the new rule is something the car's computer can actually understand and execute.

The "What Could Go Wrong?" (The Reality Check)

The researchers tested several different AI models (like GPT and Claude) to see how they performed. They discovered three big lessons:

The "Messy Handwriting" Lesson: Some AIs are smart but "messy." They might give the right answer but wrap it in weird formatting that a car's computer would find confusing. It’s like a student giving the right math answer but writing it in invisible ink.
The "Over-Protective Parent" Lesson: Some AIs try to be too safe. They see one mistake and decide to make the rules so tight that the robot becomes a "nervous wreck," refusing to drive in anything but perfect conditions.
The "Complexity vs. Clarity" Lesson: There is a tug-of-war between making a rule perfect and making it understandable. If an AI changes too much, it becomes a "black box" that humans can't easily double-check.

The Bottom Line

The paper proves that we can use powerful AI to keep self-driving systems safe and up-to-date, but we can't just let the AI "off the leash." We need to combine the detective's evidence with the architect's strict building rules to ensure the robot stays safe, smart, and—most importantly—predictable.

Technical Summary: Grammar-Constrained Refinement of Safety Operational Rules

1. Problem Statement

In Cyber-Physical Systems (CPS), such as autonomous driving, safety operational rules are used to translate high-level safety requirements into testable conditions (e.g., "if speed is $X$ , distance must be $> Y$ "). However, as operating environments evolve, these rules can become inconsistent with observed system behavior.

The authors identify two primary challenges in automating the refinement of these rules:

Syntactic Correctness: Refined rules must strictly adhere to a domain-specific grammar to be usable by downstream verification tools.
Semantic Validity & Overfitting: Using Large Language Models (LLMs) to refine rules introduces the risk of "language-in-the-loop" errors, where a model might produce rules that are syntactically perfect but semantically unjustified (e.g., over-constraining a rule to fit a specific set of observations, thereby losing general safety utility).

2. Methodology

The paper proposes a framework that combines counterfactual reasoning with a grammar-constrained LLM refinement loop. The process follows three main stages:

Counterfactual Analysis: To localize where a rule fails, the framework performs an $L_1$ minimal-change search. For every inconsistent input $x$ , it finds a minimally perturbed input $x'$ that flips the system's observed outcome. This creates "counterfactual evidence" that identifies the precise decision boundary where the rule and reality diverge.
Grammar-Constrained LLM Refinement Loop:
- Synthesis: An LLM is prompted with the original rule, the counterfactual evidence (the boundary points), and a domain-specific grammar. It is tasked with proposing a minimal refinement (e.g., adjusting a threshold or changing an operator).
- Constraint Enforcement: The framework uses a "loop" mechanism. If the LLM produces tokens outside the allowed vocabulary or violates the grammar, the system re-prompts the model with a failure summary.
- Validation: Candidate rules undergo three checks: (1) No contradictions (using SMT solvers to ensure pass/fail rules don't overlap), (2) Preservation of consistency (ensuring previously correct rules remain correct), and (3) Target resolution (ensuring the specific inconsistency is fixed).
Semantic Validation: The final candidate is tested against the existing dataset to ensure it restores consistency without introducing new errors.

3. Key Contributions

A Novel Framework: The first approach to combine counterfactual reasoning with grammar-constrained LLMs specifically for refining inconsistent operational rules.
Counterfactual-Guided Localization: Using minimal perturbations to provide LLMs with high-quality, localized evidence for rule adjustment.
Empirical LLM Safety Study: A systematic evaluation of how different LLM architectures (GPT, Claude, Gemini, etc.) handle safety-critical rule synthesis, identifying specific failure modes.

4. Results

The framework was evaluated using an autonomous driving subsystem in the BeamNG simulator.

Effectiveness: Compared to a conventional genetic programming baseline (which had a Decisiveness Gain of 0.86), the proposed approach achieved a Decisiveness Gain of 1.0 (reducing mismatches to zero) across all tested LLM variants.
LLM Performance Variance:
- GPT-5 (Thinking mode) emerged as the strongest performer, balancing high semantic validity, grammar compliance, and change minimality.
- Claude Sonnet 4.5 was highly interpretable but struggled with semantic validity and minimality (often over-constraining).
- DeepSeek showed significant issues with grammar compliance (e.g., adding unnecessary markdown or list structures).
Safety Lessons: The study revealed that LLMs tend to "over-constrain" rules (making them too specific to the training data) and can produce "syntactically correct but semantically unjustified" bounds that violate the intended Operational Design Domain (ODD).

5. Significance

This work is significant because it moves beyond using LLMs for simple code generation and applies them to formal safety engineering. It highlights a critical tension in AI-assisted safety: while LLMs are excellent at finding intuitive fixes for inconsistencies, they lack the inherent "safety awareness" to distinguish between a valid rule refinement and an overfitted, overly conservative constraint. The paper provides a roadmap for future work, emphasizing the need for strongly typed rule generators and simulation-based falsification to ensure that LLM-refined rules are robust and generalizable.