Cross-site scripting adversarial attacks based on deep reinforcement learning: Evaluation and extension study

1. Problem Statement

Cross-Site Scripting (XSS) remains a critical web security threat. While Deep Learning (DL) models have achieved high accuracy in detecting XSS attacks, they are vulnerable to adversarial attacks. A reference study by Chen et al. (2022) claimed to use Deep Reinforcement Learning (DRL) to generate adversarial XSS examples that evade state-of-the-art (SOTA) detectors with an Escape Rate (ER) of over 90%.

However, the authors of this paper identified significant threats to validity in Chen et al.'s work, suggesting that the reported high success rates might be artifacts of the experimental setup rather than genuine adversarial capabilities:

1. Lack of Action Validation (TH1): The original method did not verify if the sequence of mutations applied by the agent preserved the semantic integrity of the XSS payload.
2. Lack of Preprocessing Validation (TH2): The detectors' preprocessing pipelines (tokenization and vocabulary) were not robust against adversarial mutations. Mutations often resulted in Out-of-Vocabulary (OOV) tokens, which were replaced by a generic 'None' token. This effectively destroyed the semantic meaning of the attack, causing the detector to fail not because it was "fooled," but because the input was no longer recognizable as an XSS script.
3. Lack of Reproducibility (TH3): The original code, datasets, and models were unavailable, hindering independent verification.

2. Methodology

The authors conducted a replication study followed by an extension study to address these threats.

A. Replication Study

• Dataset: Since the original dataset was unavailable, they used a public dataset (Mereani and Howe, 2018) containing ~15,000 payloads.
• Detectors: They trained MLP, LSTM, and CNN models to detect XSS, achieving high baseline accuracy (>99%).
• Adversarial Agent: They implemented a DRL agent using the PPO (Proximal Policy Optimization) algorithm (instead of the original SAC) to generate adversarial examples using a set of 27 mutation rules (e.g., encoding, case mixing, adding interference strings).
• Goal: To replicate the high Escape Rates reported by Chen et al. and isolate the causes.

B. The XSS Oracle

To address TH1 and TH2, the authors introduced an XSS Oracle:

• Function: It renders a payload within a known web template and compares the resulting DOM with the original. If the DOM changes, the payload is classified as Malicious; otherwise, it is Benign.
• Metrics Introduced:
  • Ruin Rate (RR): The percentage of payloads that lose their malicious properties (i.e., stop being XSS) after mutation or preprocessing.
  • OOV Rate: The percentage of tokens in the input vector that are replaced by 'None' due to being out-of-vocabulary.

C. Extension Study (Oracle-in-the-Loop)

The authors integrated the Oracle directly into the DRL training loop:

• Reward Function Modification: If the agent generates a payload that the detector misses but the Oracle classifies as Benign (i.e., the attack was "ruined"), the agent receives a negative reward (-2).
• Objective: This forces the agent to learn mutations that successfully evade the detector while preserving the semantic integrity of the XSS attack.

3. Key Contributions

1. Replication and Validation: Successfully replicated Chen et al.'s results, confirming that the original DRL agents achieved ~98% escape rates.
2. Identification of Validity Threats: Demonstrated that the original high escape rates were largely due to TH2 (Preprocessing vulnerabilities). The agents learned to exploit OOV tokens rather than generating sophisticated semantic attacks.
3. XSS Oracle: Developed a robust tool to validate the semantic integrity of XSS payloads, introducing the Ruin Rate and OOV Rate as critical evaluation metrics.
4. Improved Adversarial Training: Proposed an "Oracle-in-the-loop" training strategy that mitigates validity threats. The resulting agent achieves high escape rates (~96-98%) while ensuring the attacks remain semantically valid (low Ruin Rate).
5. Open Science: Released all code, datasets, and results publicly to ensure reproducibility.

4. Key Results

• RQ1 (Action Validation): The lack of action validation (TH1) was found to be a minor issue (RR ~6-7%), indicating the mutations themselves were mostly semantically sound.
• RQ2 (Preprocessing Validation): The lack of preprocessing validation (TH2) was the primary cause of the inflated escape rates. The original agents generated payloads where ~45% of tokens became OOV ('None'), causing the detector to fail. The Ruin Rate for preprocessed payloads was >90%, meaning the attacks were effectively broken before detection.
• RQ3 (Extension): When the Oracle was integrated, the agents learned to avoid OOV tokens. The Escape Rate remained high (~97%), but the Ruin Rate dropped to near zero and OOV Rate dropped to <2.3%. This proves the agents can genuinely bypass detectors without relying on preprocessing flaws.

5. Significance and Implications

• Re-evaluating SOTA Security: The paper highlights that many reported "successful" adversarial attacks against DL-based security systems may be false positives caused by poor preprocessing pipelines (specifically OOV handling) rather than true model weaknesses.
• Defense Strategy: Defenders must prioritize preprocessing robustness. Simply detecting high OOV rates or using an Oracle to validate payloads can neutralize these specific types of attacks.
• Evaluation Standards: The paper establishes that future adversarial studies must include semantic validation (e.g., via an Oracle) to distinguish between "breaking the detector" and "breaking the input."
• Transparency: By addressing the reproducibility crisis (TH3), the authors provide a transparent framework for the community to rigorously test and improve XSS detection systems.

In conclusion, while the original study correctly identified that DRL can generate XSS attacks, this paper proves that the magnitude of the threat was overstated due to experimental flaws. The extended approach provides a more rigorous, realistic, and effective method for evaluating the true robustness of XSS detectors.