Technical Summary: Benchmarking Large Language Models for IoC Recovery under Adversarial Code Obfuscation and Encryption

Problem Statement

Software obfuscation and encryption are persistent challenges for cybersecurity analysts, particularly when adversaries conceal Indicators of Compromise (IoCs)—such as IP addresses, domains, or API keys—within source code. While Large Language Models (LLMs) have demonstrated significant progress in code reasoning and deobfuscation, their resilience against adversarial concealment techniques, specifically cryptographic encryption, remains largely unexplored. Existing benchmarks often focus on structural recovery or readability improvements rather than the systematic detection of security-relevant artifacts under progressive obfuscation. Furthermore, prior works evaluating JavaScript deobfuscation with LLMs typically exclude encryption techniques and do not focus on the automatic extraction of IoCs. This gap leaves critical questions unanswered regarding whether LLMs can reliably identify hidden artifacts in encrypted code and how different obfuscation layers impact this capability.

Methodology

The authors propose a systematic, automated benchmark designed to evaluate the capacity of LLMs to recover IoCs embedded in obfuscated and encrypted JavaScript programs. The methodology consists of two primary components:

1. Dataset Construction

The study utilizes a dataset derived from 336 original, benign JavaScript files from "The Algorithms Project." These files were systematically transformed into 12 progressive concealment levels to emulate realistic threat scenarios. The transformation pipeline includes:

• Syntactic and Structural Obfuscation: Variable renaming, dead code injection, control-flow flattening, and string encoding.
• Cryptographic Concealment: XOR encryption and AES-256 encryption.
• Progressive Complexity: Levels 1–4 involve lightweight transformations; Levels 5–6 introduce encryption (XOR and AES-256) with the decryption keys and routines embedded directly in the source code; Levels 7–12 combine encryption with the previous syntactic and structural obfuscations.

Crucially, the inclusion of decryption keys within the code ensures that the ground truth is recoverable via static analysis, allowing the study to evaluate the LLM's ability to reason about cryptographic constructs rather than its ability to perform brute-force cryptanalysis.

2. AI-Assisted Deobfuscation and Evaluation

An automated framework was developed to query multiple commercial LLMs (ChatGPT, Gemini, Anthropic, Grok, and Cohere) via their APIs. The process involves:

• Standardized Prompting: A controlled prompt template asks the model to determine if an IoC is present (YES/NO/DON'T KNOW) and, if YES, to extract the value.
• Response Normalization: Raw outputs are normalized into a unified schema to facilitate automated evaluation.
• Metrics: The system measures detection rate, extraction accuracy, false positives/negatives, uncertainty rates, and hallucination rates.

Key Contributions

The paper makes three primary contributions to the field of automated threat intelligence and code analysis:

1. Automated Methodology: A reproducible framework for evaluating multiple LLMs on their ability to detect hidden IoCs across varying levels of obfuscation and encryption.
2. Large-Scale Dataset: The construction and release of a dataset comprising 4,368 modified JavaScript instances (336 originals × 12 transformation levels), specifically designed for benchmarking IoC recovery under adversarial conditions.
3. Comprehensive Empirical Evaluation: A detailed comparison of state-of-the-art LLMs, identifying specific strengths in handling lightweight transformations and consistent limitations when confronting encrypted or semantically distorted code. The study also analyzes hallucination patterns, revealing systematic biases in model outputs.

Key Results

The evaluation reveals a sharp dichotomy in LLM performance based on the type of obfuscation:

• Lightweight Obfuscation (Phases 0–4): LLMs exhibit high success rates. Models such as ChatGPT, Claude, Gemini, and Grok achieved near-perfect detection (100%) for plain text and syntactic obfuscations (Base64, identifier renaming, dead code, structural flattening). Cohere showed progressive degradation but remained functional.
• Cryptographic Obfuscation (Phases 5–12): Performance collapses dramatically. Upon the introduction of encryption (XOR or AES-256), detection rates for all models dropped to near-zero, regardless of whether the decryption key was present in the code.
  • The transition from Phase 4 (structural obfuscation) to Phase 5 (XOR encryption) represents a "performance cliff," where models shift from systematic success to near-complete failure.
  • Adding further syntactic or structural layers to encrypted code (Phases 7–12) did not significantly worsen performance beyond the baseline encryption failure, suggesting that encryption itself is the dominant barrier.
• Hallucinations: While aggregate hallucination rates were low for most models (under 0.25% for Anthropic and ChatGPT), Gemini exhibited a higher rate (4.8%). Hallucinated IoCs were not random; they showed a strong bias toward private IP ranges (e.g., 192.168.17.101), suggesting that when models cannot decode material, they revert to learned heuristics for "plausible" security contexts.

Significance and Claims

The paper claims that its findings establish encryption as a critical frontier for LLM-driven code analysis. The results indicate that while LLMs are effective tools for static analysis and initial triage of unencrypted or lightly obfuscated code, they currently lack the capability to perform symbolic reasoning or simulate code execution required to recover IoCs from encrypted payloads.

The authors conclude that current LLM architectures rely primarily on surface-level token recognition and semantic pattern matching rather than deep program semantics or control-flow reasoning. Consequently, the presence of encryption fundamentally disrupts the mechanisms LLMs use for IoC recognition. The paper advocates for a cautious integration of LLMs into cybersecurity pipelines, suggesting they be used as part of hybrid frameworks where traditional static or dynamic analysis complements their capabilities, rather than relying on them as standalone solutions for encrypted threat intelligence.