Technical Summary: CyberLLM Framework for Automotive Cybersecurity

1. Problem Statement

The transition to Software-Defined Vehicles (SDVs) has significantly expanded the automotive attack surface, introducing vulnerabilities across source code, runtime logs, and deployment topologies. While Large Language Models (LLMs) offer the potential to reason over these heterogeneous artifacts and propose remediations, their application in safety-critical environments faces two primary challenges:

1. Probabilistic Nature: LLMs may hallucinate findings or miss issues, making them unreliable as the sole basis for security decisions.
2. Safety Constraints: Autonomous agents acting on vehicles must never execute unsafe or unauthorized actions, even when inputs are adversarially manipulated (e.g., via prompt injection or memory poisoning).

Current tooling typically addresses these layers in isolation using rule-based scanners or manual reviews, lacking the ability to reason across layers or act autonomously with oversight. There is a need for a framework that balances autonomous detection with a deterministic, auditable safety envelope.

2. Methodology: The CyberLLM Framework

CyberLLM is a multi-agent, LLM-orchestrated framework designed to autonomously detect vulnerabilities and execute remediations under a formal, runtime safety guard. The system operates on a schema-driven pipeline where every agent derives from a common BaseAgent, ensuring mandatory provenance for all inputs.

A. Layered Detection Architecture

The detection phase merges deterministic analysis with LLM refinement to achieve high recall without sacrificing precision:

• Deterministic Floor: An "always-on" layer comprising:
  • Regex Rules: Multiline, case-insensitive patterns for code.
  • AST Analyzers: Sandboxed execution of tools like Bandit (Python) and Semgrep (other languages).
  • Topology Graph Checks: Static analysis of deployment YAMLs against invariants (e.g., external actors holding write edges to memory).
  • Log Analyzers: Temporal/statistical detectors for anomalies like brute-force attempts or DNS covert channels.
• LLM Refinement Pass: The deterministic hits are injected as context into LLM scanners. The LLM is strictly instructed to refine, explain, or extend existing findings rather than invent new ones. A "completeness critic" pass specifically targets semantic defects (e.g., data races, TOCTOU) that regex cannot detect.
• Merging: Findings are deduplicated; the deterministic layer acts as the ground truth, with the LLM adding value only when it identifies genuinely new loci.

B. Decision and Response Engine

A central Decision Agent aggregates findings, tags them using a human-centric asset taxonomy (seven categories), and selects a tiered response (none/soft/moderate/hard).

• Memory and Feedback: The agent queries a signed cross-session memory store for similar past incidents. Precedents can raise the response tier but never lower it. If a previous attempt was refused, the system automatically escalates the tier.
• Action Generation: An Action Generator constructs a remediation plan. It relies on a deterministic, rule-ID-keyed playbook as a floor. An optional LLM tool-selector may add steps from an MCP-style tool registry, but only read-only, observe, or verify tools are permitted; destructive actions are forbidden.

C. Runtime Security Guard

Before any action executes, a SecurityGuard enforces four contextual security properties (derived from Siu et al. [1]) and an independent action-alignment oracle:

1. Task Alignment: The objective lies within the allowed set.
2. Action Alignment: The action kind is compatible with the objective.
3. Source Authorization: All inputs originate from authenticated sources.
4. Data Isolation: Actions write only to trusted destinations.

• Oracle ($H_a$): An independent LLM judge verifies if the proposed action genuinely serves the declared objective. This oracle can only tighten the decision (approve $\to$ refuse), never loosen it, ensuring that a subverted scanner cannot force an unsafe action.
• Refusal Handling: If an action is refused, the system triggers re-planning with an escalated tier rather than executing a potentially unsafe fix.

D. Symmetric Attack Pipeline

To evaluate defense quality, CyberLLM includes a red-team pipeline that mirrors the defense agents. It generates non-destructive, canary-only exploit scripts (e.g., shell injection, path traversal) based on the same rule base. This allows both attack and defense to be measured on identical scenarios without executing actual exploits.

3. Key Contributions

The paper outlines four primary contributions:

1. Layered Detection Design: A unified stream merging regex, AST-based, and topology-graph analyzers with an LLM refinement pass, where the deterministic floor grounds and constrains the LLM.
2. Decision-and-Response Engine: A mechanism that selects tiered mitigations, ratchets confidence via signed memory and re-planning feedback, and executes plans through a tool registry with per-step verification and rollback.
3. Runtime Security Guard: A formal enforcement layer applying four contextual security properties and an independent action-alignment oracle to prevent unsafe actions, even if upstream components are subverted.
4. Symmetric Attack Pipeline & Benchmark: A system to generate and replay exploits for side-by-side measurement, evaluated on a hand-labeled corpus of automotive ECU modules.

4. Experimental Results

The framework was evaluated on an independent, ground-truthed benchmark consisting of nine original automotive ECU modules (C, C++, Rust) containing 47 layered vulnerabilities, plus two clean control modules.

• Deterministic Layer (Fast Mode): Covered 34.0% (16/47) of labeled vulnerabilities with 100% precision (zero false positives on clean controls).
• LLM-Augmented Layer (Deep Mode): By adding LLM refinement and completeness passes, coverage increased to approximately 70% (33/47) with an F1 score of 0.83.
• Precision Maintenance: The Deep mode maintained 100% precision on the clean controls. The LLM refinement did not introduce false positives; it primarily improved recall on medium-severity logic and DoS defects (e.g., subtle races, off-by-one errors) that regex patterns missed.
• Language Breakdown: Improvements were consistent across C, C++, and Rust, with the LLM pass significantly boosting detection of complex semantic defects.

5. Significance and Claims

The paper claims that CyberLLM demonstrates that LLM agents can perform useful autonomous cyber-defense when wrapped in a deterministic, auditable safety envelope.

• Balancing Autonomy and Safety: The framework resolves the tension between LLM autonomy and safety constraints by ensuring that no action executes unless it satisfies formal, contextual security properties.
• Quantifiable Contribution: The results quantify the specific value of the LLM: it roughly doubles the recall of the deterministic floor without introducing a single false positive on benign code.
• Robustness: The system is designed such that even if a scanner is subverted by adversarial inputs, the independent guard and memory mechanisms prevent the commitment of unsafe actions.
• Future Outlook: The authors note that while the current static-first approach is effective, future work will integrate live tool servers, retrieval-augmented grounding over CVEs, and validation against high-fidelity SDV testbenches (e.g., CARLA).

The paper concludes that the combination of grounded LLM refinement, tamper-evident memory, and formal runtime enforcement provides a viable path toward autonomous, safe automotive cybersecurity.