💻 computer science

When to Trust the Map: Confidence-Aware LLM Routing for Automotive CVE-to-ATM Mapping

This paper proposes a confidence-aware LLM routing framework that reformulates automotive CVE-to-ATM mapping as a selective automation problem, using hierarchical in-context learning and calibrated confidence scores to route mappings into auto-confirmable, review, or hold categories, thereby significantly improving precision over zero-shot baselines while isolating uncertain cases for analyst review.

Original authors: Heeyun Heo, Sangmin Park, Huy Kang Kim, Sanghoon Jeon

Published 2026-07-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Heeyun Heo, Sangmin Park, Huy Kang Kim, Sanghoon Jeon

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the digital world as a massive, bustling city where every car is now a smart device, constantly talking to the cloud, other cars, and even the traffic lights. Just like any city, this digital metropolis has its share of troublemakers trying to break in, steal data, or cause chaos. To keep everyone safe, security experts use two different "languages" to describe these threats. One language, called CVEs, is like a mechanic's technical manual; it lists exactly which bolt is loose or which wire is frayed. The other language, called ATM (Automotive Threat Matrix), is more like a detective's playbook; it describes how a criminal thinks, what their goals are, and the specific tricks they use to commit a crime.

The problem is that these two languages don't speak to each other very well. A mechanic's note about a broken wire doesn't automatically tell a detective, "Ah, this is a 'Remote Code Execution' tactic!" If we try to translate them automatically using Artificial Intelligence (AI), the AI might get confused, mix up the clues, or confidently invent a crime that never happened. In the world of self-driving cars, a wrong guess isn't just a typo; it could mean a car gets hacked or a safety system fails to activate. So, the big question isn't just "Can AI translate these threats?" but "Can we trust the AI enough to let it decide on its own, or do we need a human to double-check the work?"

This is where the paper "When to Trust the Map" steps in. The researchers, working with a team from Korea University and Kookmin University, decided to stop asking the AI to just "guess and check." Instead, they built a smart system that acts like a confidence-aware traffic cop. They realized that while AI is great at generating ideas, it often suffers from "hallucinations"—making things up with total confidence. To fix this, they created a framework that doesn't just output a list of threats; it assigns a confidence score to every single suggestion, essentially asking the AI, "How sure are you about this?"

Here is how their system works, using a playful analogy: Imagine the AI is a junior detective trying to solve a case. Instead of letting the detective run wild, the system puts them through a rigorous training camp. First, the detective looks at a library of past solved cases (called "in-context learning") to find similar patterns. Then, the system doesn't just take the detective's word for it. It sends the case to a second detective (a "judge" AI) who only looks at the raw evidence—the actual text of the security report—without knowing what the first detective guessed. This second detective checks if the clues actually support the theory.

The system then combines these two voices: the first detective's consistency (did they guess the same thing multiple times?) and the second detective's evidence check (does the text actually say this?). These signals are fed into a "meta-model," which is like a wise old chief of police. The chief calculates a final confidence score (a number between 0 and 1) for every single threat suggestion. Based on this score, the system sorts every suggestion into one of three bins:

AUTO (The Green Light): If the confidence score is very high (above 0.74 in their best mode), the system says, "This is solid! We can trust this automatically." In their tests, when the system was this confident, it was right 87.8% of the time. That's more than double the accuracy of just guessing randomly or using older, simpler AI methods.
REVIEW (The Yellow Light): If the score is in the middle, the system says, "This looks interesting, but I'm not 100% sure. Let's show this to a human expert." This ensures that tricky or ambiguous cases get the human attention they need.
HOLD (The Red Light): If the score is very low, the system says, "I have no idea, and the evidence is weak. Put this on hold." In their tests, every single case the system put in the "HOLD" bin was indeed incorrect, proving the system knows when not to guess.

The researchers tested this on a dataset of 269 real automotive security reports (CVEs) that had been carefully labeled by human experts. They found that their "confidence-aware" approach was much better at filtering out bad guesses than a standard AI that just tries to answer the question directly. By separating the "sure things" from the "maybe things," they showed that we can automate a lot of the dangerous work without losing safety.

The paper explicitly argues against the idea that we can just rely on a single AI model to do this job perfectly on its own. They found that without this extra layer of checking and confidence scoring, AI models are too prone to errors and "hallucinations" to be trusted in safety-critical environments like cars. They also ruled out the idea that simply asking the AI to repeat the answer multiple times (consistency) is enough; you also need that second "judge" to check the actual evidence.

In short, this paper suggests that we don't need to choose between human safety and AI speed. Instead, we can build a system that knows its own limits. It acts like a smart filter, letting the AI handle the easy, obvious threats automatically while flagging the confusing ones for human experts. This "selective automation" could help car manufacturers update their safety plans faster and more accurately, ensuring that when a new vulnerability is discovered, the right countermeasures are applied without the risk of the AI getting it wrong. The results, measured on their specific test set, show a clear path forward: a framework where the AI says, "I'm confident enough to drive this part of the car," while the human stays in the driver's seat for the rest.

Technical Summary: Confidence-Aware LLM Routing for Automotive CVE-to-ATM Mapping

Problem Definition
The paper addresses the critical challenge of mapping Common Vulnerabilities and Exposures (CVEs) to the Auto-ISAC Automotive Threat Matrix (ATM). While CVEs describe technical vulnerabilities, the ATM organizes adversarial behavior into tactics and techniques. Direct automated mapping is difficult because these representations are not aligned; incorrect mappings in safety-critical automotive environments can distort threat interpretation and mitigation prioritization. Furthermore, general-purpose frameworks like MITRE ATT&CK lack specific automotive context (e.g., CAN injection, OTA compromise), and Large Language Models (LLMs) suffer from hallucinations and generation inconsistency, making single-output automation unreliable for safety-critical tasks. The core problem is formulated as a selective automation challenge: distinguishing mappings that are reliable enough for automatic adoption from those requiring analyst review.

Methodology
The proposed framework operates as a four-stage pipeline designed to generate, verify, calibrate, and route candidate mappings:

Two-Stage Hierarchical Candidate Generation:
- The system decomposes the task into predicting a tactic first, then predicting techniques conditioned on that tactic.
- It utilizes Hierarchical In-Context Learning (ICL) with OpenAI GPT-5.2 (reasoning effort = high).
- Stage 1 (Tactic): The model selects from 14 valid tactic IDs using a function-calling schema to prevent ontology violations.
- Stage 2 (Technique): For each predicted tactic, the model predicts techniques from the specific subset associated with that tactic. It retrieves $k=8$ semantically similar examples from an "Exemplar Pool" (derived from older CVEs) using k-nearest-neighbor (kNN) retrieval to focus on tactic-specific patterns.
- Consistency: To mitigate LLM instability, each stage is invoked $N=5$ times. A pair-level consistency score is calculated as the product of the frequency of the tactic and the frequency of the technique given that tactic.
Multi-Signal Verification:
- To distinguish between frequently generated errors and correct mappings, the system employs an evidence-grounded LLM-as-a-Judge.
- The verifier extracts six structured elements from the CVE text (aligned with ISO/SAE 21434 TARA) such as affected components and attack interfaces. Unstated elements are marked "unknown" to enforce evidence-only reasoning.
- The verifier scores each candidate on three axes: support, contradiction, and sufficiency. Candidates are anonymized before verification to prevent bias.
Confidence Calibration:
- A meta-model (L2-regularized logistic regression) is trained on a Development Set to integrate generation stability (consistency) and verification signals (support, contradiction, sufficiency) into a single calibrated confidence score ( $\hat{p}$ ).
- The model includes interaction terms (e.g., consistency $\times$ support) to ensure confidence is high only when both generation stability and factual support are present.
- Post-hoc calibration (sigmoid and temperature scaling) is applied to correct over/under-confidence.
Three-Tier Operational Routing:
- Based on $\hat{p}$ $\overset{p}{^}$ , candidates are routed into three tiers:
  - AUTO: High confidence; automatically confirmed for vulnerability registries.
  - REVIEW: Moderate confidence; routed to analysts with supporting evidence.
  - HOLD: Low confidence; deferred until further evidence is available.
- Thresholds ( $\tau_{auto}$ , $\tau_{hold}$ ) are tuned to balance precision and recall, allowing for different operational modes (High-Confidence, Balanced, High-Coverage).

Key Contributions

Formulation: The paper formalizes automotive CVE-to-ATM mapping as a selective automation problem, explicitly separating auto-confirmable cases from those requiring human oversight.
Framework: It introduces a multi-signal routing framework that combines hierarchical ICL, self-consistency checks, and evidence-grounded verification to drive a calibrated three-tier decision process.
Dataset: The authors constructed a benchmark dataset of 269 automotive CVEs with 483 expert-annotated tactic–technique pairs, addressing the lack of public data for this specific mapping task.

Results
Evaluated on a test set of 58 CVEs (98 gold pairs), the system demonstrated significant improvements over baselines:

Candidate Quality: The proposed system achieved a precision of 0.395 (vs. 0.277 for a Flat Zero-shot GPT-5.2 baseline) while maintaining matched recall (0.745), reducing the average number of predictions per CVE from 4.55 to 3.19.
Routing Performance: In High-Confidence mode (threshold $\tau_{auto} \approx 0.74$ ), the AUTO tier achieved a precision of 0.878, more than double the candidate-set base rate (0.395).
Calibration: The calibrated confidence score $\hat{p}$ achieved an AUROC of 0.868 in distinguishing correct from incorrect candidates, significantly outperforming a consistency-only baseline (AUROC 0.705).
Ablation: Removing the evidence-grounded judge signals caused a sharp drop in AUROC, confirming that consistency alone is insufficient for reliable confidence estimation.

Significance and Claims
The paper claims that this framework provides a viable path for selective automation in automotive cybersecurity. By isolating high-confidence mappings, the system supports autonomous Cybersecurity Management System (CSMS) workflows while preserving analyst attention for ambiguous cases. The authors emphasize that the framework offers traceable decision evidence (confidence scores and verification signals) required for ISO/SAE 21434 compliance and audit workflows.

The authors remain modest regarding the scope, noting that:

The evaluation is limited to the current ATM ontology (14 tactics, 75 techniques) and the constructed dataset.
The system relies on a single backbone model (GPT-5.2) for both generation and verification, though safeguards like evidence grounding and candidate anonymization are used to mitigate self-confirmation bias.
The effectiveness of the "REVIEW" tier in practice requires future field studies with security analysts.
The system does not currently handle indirect secondary effects or speculative extensions, focusing strictly on directly verifiable attack stages.

Technical Summary: Confidence-Aware LLM Routing for Automotive CVE-to-ATM Mapping

More like this