Evaluating LLM-Generated Obfuscated XSS Payloads for Machine Learning-Based Detection

1. Problem Statement

Cross-site scripting (XSS) remains a critical web security vulnerability. A major challenge for detection systems is obfuscation, where attackers transform malicious payloads to alter their string representation while preserving their browser-executable behavior.

• The Gap: Traditional and Machine Learning (ML) based detectors often fail to generalize to these transformed variants.
• The Limitation of Current Augmentation: Existing data augmentation methods often rely on rule-based transformations that may produce syntactically diverse but behaviorally invalid payloads (i.e., they look like attacks but do not execute). Training ML models on such "noisy" data can degrade detector performance.
• The Research Question: Can Large Language Models (LLMs) generate obfuscated XSS payloads that not only look different but also preserve the observable runtime behavior of the original malicious code?

2. Methodology

The authors propose a structured pipeline that integrates deterministic transformations, LLM generation, and rigorous runtime validation.

A. Pipeline Overview

1. Deterministic Obfuscation Chains:
   • Starting from a dataset of 13,420 malicious XSS payloads, the system applies fixed, two-step transformation recipes (e.g., hex_escape → split_strings, comment_insertion → case_swap).
   • This creates multi-step chains ($P_0 \to P_1 \to P_2$) to simulate complex adversarial strategies.
2. Browser-Based Runtime Validation:
   • Instead of relying on string similarity, the pipeline executes both the original payload ($P_0$) and the transformed payload ($P_2$) in a controlled browser environment.
   • It records observable effects: alert triggers, console outputs, network requests, and execution errors.
   • Filtering: Only chains where the execution trace of $P_2$ matches $P_0$ are retained as behavior-preserving examples.
3. LLM Training and Generation:
   • Baseline: An untuned LLM is tested on the obfuscation task.
   • Fine-Tuning: A model is fine-tuned on the behavior-filtered source-target pairs ($P_0 \to P_2$) derived from the validated chains.
   • The fine-tuned model generates new candidate obfuscated payloads.
4. Downstream Evaluation:
   • Generated payloads are evaluated for runtime behavior match rates.
   • A Random Forest classifier (using TF-IDF features) is trained under three conditions to test detection impact:
     1. Original data only.
     2. Original data + all generated payloads.
     3. Original data + behavior-filtered generated payloads.

3. Key Contributions

1. Structured Pipeline: A novel framework combining deterministic obfuscation chains with LLM generation, specifically designed to filter for behavioral validity.
2. Runtime Validation Mechanism: The introduction of a browser-based execution checker that evaluates payloads based on observable behavior rather than syntactic similarity, addressing the "validity gap" in security data augmentation.
3. Empirical Evaluation: A comprehensive study quantifying the ability of LLMs to preserve execution semantics and measuring the downstream impact of these generated samples on XSS detectors.

4. Key Results

A. Runtime Behavior Match Rates

• Baseline (Untuned): Achieved a match rate of 0.15 (15% of generated payloads preserved behavior).
• Fine-Tuned Model: Achieved a match rate of 0.22 (22%).
  • This represents a 46.7% relative improvement over the baseline.
  • Insight: While fine-tuning helps, the absolute success rate remains low. Most LLM outputs are syntactically plausible but fail to execute correctly (e.g., breaking event handlers or execution contexts).

B. Transformation Stability

• High Validity: Transformations involving string manipulation (e.g., split_strings → comment_insertion) showed high validity rates (~75-77%).
• Low Validity: Transformations involving heavy encoding (e.g., hex_escape) showed very low validity rates (~2-5%).
  • Reason: Encoding-based obfuscations are brittle; if the reconstruction logic at runtime is slightly off, the payload fails.

C. Downstream Detection Performance

• No Performance Gain: Adding LLM-generated payloads (even filtered ones) to the training set did not improve the detection accuracy, precision, recall, or F1 score of the Random Forest classifier.
  • Reason: The generated samples constituted a tiny fraction of the total dataset, so the classifier remained dominated by the original data distribution.
• No Performance Degradation: Crucially, adding the generated samples did not harm performance. This suggests that behavior-filtered samples are "safe" to include in small quantities, whereas unfiltered (invalid) samples might introduce noise.

5. Significance and Conclusion

• The "Plausibility vs. Correctness" Gap: The study highlights a critical limitation in applying LLMs to security: generating code that looks like an attack is not the same as generating code that executes as an attack. Syntactic diversity does not guarantee functional equivalence.
• Importance of Runtime Checks: The paper argues that for adversarial data generation, runtime validation is non-negotiable. Without it, generated data may teach detectors artifacts that do not correspond to real attacks.
• Future Directions: The authors suggest that future systems should employ:
  • Larger fine-tuning datasets.
  • Constrained decoding to enforce execution logic.
  • Hybrid approaches (rules + LLMs).
  • Multi-browser validation to ensure cross-environment compatibility.

Final Takeaway: While LLMs show promise in learning obfuscation patterns (evidenced by the 46.7% improvement with fine-tuning), they currently struggle to reliably generate behaviorally valid XSS payloads. Therefore, any pipeline using LLMs for security data augmentation must include a strict runtime behavior filter to ensure data quality.