Evaluating False Alarm and Missing Attacks in CAN IDS

Technical Summary: Evaluating False Alarm and Missing Attacks in CAN IDS

Problem Statement
Modern vehicles rely heavily on the Controller Area Network (CAN) to interconnect Electronic Control Units (ECUs) for safety-critical functions. While Machine Learning (ML)-based Intrusion Detection Systems (IDS) are increasingly deployed to secure CAN traffic, their robustness against adversarial manipulation remains under-explored. Existing literature often evaluates IDS performance on clean data or considers false alarms (false positives) and missed attacks (false negatives) in isolation. This paper addresses the gap in understanding how protocol-compliant, payload-level perturbations can simultaneously trigger false alarms and cause malicious traffic to evade detection, thereby compromising the reliability of safety-critical automotive IDS.

Methodology
The study presents a systematic adversarial evaluation using the ROAD dataset, which contains over 3.5 hours of real vehicle CAN traffic, including 12 benign captures and 33 attack captures (fuzzing, fabrication, and ECU masquerade).

• Models Evaluated: The authors compare four shallow learning models—Decision Tree (DT), Random Forest (RF), Extra Trees (ET), and XGBoost (XGB)—against a Deep Neural Network (DNN) with four hidden layers.
• Threat Model: The evaluation assumes a "gray-bus" adversary with white-box knowledge of the IDS. The attacker can inject fabricated messages but must adhere to CAN protocol constraints.
• Adversarial Generation: Using the Adversarial Robustness Toolbox (ART), the authors generate adversarial examples via FGSM (Fast Gradient Sign Method), BIM (Basic Iterative Method), and PGD (Projected Gradient Descent). Crucially, perturbations are restricted to the payload bytes (D0–D7) of the CAN frame. The CAN ID and Data Length Code (DLC) remain fixed, and all perturbed values are clipped to the valid byte range [0, 255] to ensure the generated frames remain protocol-compliant and valid on the wire.
• Evaluation Metrics: The study utilizes Attack Success Rate (ASR) to quantify the frequency of induced misclassifications (both false positives on benign traffic and false negatives on malicious traffic) and the Matthews Correlation Coefficient (MCC) to assess overall classification performance under class imbalance.

Key Contributions

1. Unified Adversarial Framework: The paper introduces a unified evaluation that examines false alarms and missed attacks as coupled failure modes, rather than in isolation.
2. Protocol-Compliant Perturbations: Unlike prior studies that may generate invalid network packets, this work enforces strict domain constraints, ensuring adversarial examples are valid CAN frames that could realistically exist on a vehicle bus.
3. Comparative Architecture Analysis: The study characterizes how architectural choices (shallow ensembles vs. deep learning) influence adversarial robustness across diverse attack scenarios (Fuzzing, MECTA, MSA, RLOFFA, RLONA, CSA).
4. Reproducibility: The authors have made their implementation, experimental pipeline, and evaluation code publicly available.

Results

• Benign Performance: Under normal conditions, all models achieve strong baseline performance. The DNN demonstrates slightly lower MCC (approx. 0.85) compared to shallow models (approx. 0.91), primarily due to a higher number of false negatives, suggesting a more conservative decision boundary.
• False Alarm Robustness: All models, including the DNN, exhibit strong robustness against inducing false alarms on benign traffic. The DNN, in particular, maintains low false-positive rates even under adversarial perturbations, outperforming shallow models in this specific metric.
• Missed Attack Vulnerability: The study reveals substantial vulnerabilities regarding missed attacks (false negatives).
  • DNN Vulnerability: The DNN suffers catastrophic failures under gradient-based attacks. In specific scenarios (e.g., MECTA), the DNN's ASR for missed attacks reaches 1.0, indicating a complete failure to detect adversarially manipulated malicious frames.
  • Shallow Model Resilience: While shallow models also degrade under strong attacks (particularly PGD), the Extra Trees (ET) model demonstrates superior robustness compared to other architectures. ET maintains lower ASR values for missed attacks across most scenarios and shows more stable performance under PGD compared to RF and XGB.
  • Attack Type Sensitivity: Gradient-based attacks (FGSM, BIM, PGD) are highly effective against the DNN. Conversely, protocol-aware fabrication attacks have a stronger impact on tree-based models, though ET generally outperforms DT, RF, and XGB in maintaining lower ASR.
  • CSA Attack: For the Correlated Signal Attack (CSA), no false negatives were produced by any model under adversarial evaluation. The authors attribute this to the strict protocol constraints; valid adversarial samples that could induce a missed attack could not be generated without violating CAN frame validity.

Significance and Claims
The paper claims that adversarial manipulation can simultaneously trigger false alarms and evade detection, highlighting a critical gap in current automotive IDS deployment. The results demonstrate that while deep learning models may offer superior performance on clean data, they are highly susceptible to gradient-driven adversarial manipulation, potentially leading to catastrophic safety failures. In contrast, ensemble methods like Extra Trees offer a more favorable robustness trade-off, exhibiting minimal false alarms and comparatively lower missed-attack rates.

The authors conclude that relying solely on clean test data is insufficient for assessing IDS reliability. They emphasize the necessity of adversarial robustness evaluation in safety-critical automotive systems and suggest that future work should focus on defense strategies such as adversarial training and hybrid IDS designs to improve resilience against adaptive adversaries.