Evaluating LLMs for Obfuscation Detection and Classification in Android Apps

Technical Summary: Evaluating LLMs for Obfuscation Detection and Classification in Android Apps

Problem Statement
Android application developers increasingly rely on code obfuscation to protect intellectual property and hinder reverse engineering. However, these techniques simultaneously degrade the effectiveness of Static Application Security Testing (SAST) tools, which often depend on structural patterns, control-flow graphs, and syntactic heuristics to identify vulnerabilities. While the prevalence of obfuscation in the wild is well-documented, the problem of automatically detecting whether an APK is obfuscated—and identifying the specific techniques used—remains underexplored. Existing approaches rely heavily on handcrafted features or rule-based heuristics, which are brittle and struggle to generalize across diverse and evolving obfuscation strategies. This paper investigates whether Large Language Models (LLMs), through semantic reasoning, can overcome these limitations to detect and classify obfuscation in Android apps without requiring handcrafted rules, predefined signatures, or dedicated model training.

Methodology
The authors present a large-scale empirical study utilizing an LLM-driven analysis pipeline that operates directly on Smali code extracted from Android APKs. The methodology consists of three phases:

1. Decompilation: Input APKs are decompiled using apktool to convert Dalvik bytecode into human-readable Smali code.
2. Preprocessing: To focus on developer-implemented logic, the pipeline filters out Android framework components and third-party libraries using the AndroLibZoo dataset. To manage computational costs and token limits, the authors employ a statistically significant random sampling strategy (using Cochran's formula) to select a representative subset of developer-defined classes for analysis.
3. LLM Inference: The pipeline performs binary detection (is the app obfuscated?) and multi-class classification (which technique is used?) on the sampled classes. The study evaluates six state-of-the-art models (including gpt-5-mini, gpt-4o-mini, deepseek-r1, gemma3, qwen3, and gpt-oss) across three prompt variants with increasing levels of domain-specific context. A decision threshold ($T$) is applied to the proportion of obfuscated classes to determine the app-level label.

The evaluation is conducted on two datasets:

• $D_{GT}$ (Ground Truth): A controlled benchmark of 110 APKs (10 original apps from F-Droid, 90 variants obfuscated with nine specific Obfuscapk transformations, and 10 variants obfuscated with R8).
• $D_{W}$ (Real-World): A dataset of 1,000 random Android apps collected from Google Play in 2026, where ground truth is established via manual inspection of a statistically significant sample.

Key Contributions

• First Empirical Study: This is the first large-scale investigation into the capability of off-the-shelf LLMs to detect and classify code obfuscation in Android apps purely through semantic reasoning.
• Comprehensive Landscape Review: The paper provides a detailed review of the Android obfuscation landscape, including common techniques (Identifier Renaming, Control Flow Transformation, String Encryption) and widely used tools (ProGuard/R8, Allatori, DexGuard, etc.).
• Extensive Evaluation: The study evaluates multiple LLMs, prompt formulations, and decision thresholds against both a controlled benchmark with known ground truth and a large-scale real-world dataset.
• Comparative Analysis: The authors compare LLM-based reasoning against existing SAST-based and rule-based obfuscation detection tools.
• Open Resources: The benchmark, prompts, implementation artifacts, and experimental framework are publicly released to support reproducibility.

Results

• Binary Detection: LLMs can effectively detect obfuscation. The best performance was achieved by gpt-5-mini using a detailed prompt (promptv3) and a low decision threshold ($T \in [0.1, 0.3]$), achieving a Youden's J coefficient of 1.00 on the controlled benchmark. Open-weight models like gpt-oss 20b also showed strong performance, though some models (e.g., deepseek-r1) exhibited high recall at the cost of excessive false positives.
• Technique Classification: LLMs can identify specific obfuscation techniques with high accuracy. gpt-5-mini achieved a Top-1 accuracy of 0.84 and Top-3 accuracy of 0.94. Identifier-based transformations (ClassRename, MethodRename, FieldRename) were consistently the easiest to detect, while techniques with limited syntactic footprints like Reflection and Call Indirection remained challenging.
• Comparison with SAST: LLMs substantially outperformed existing SAST tools (SEBASTiAN, Trueseeing, APKHunt, and Niroshan et al.'s tool). Traditional tools failed to detect many obfuscated apps (Recall of 0.00 for some) or produced high false positive rates (FPR up to 0.90). In contrast, the LLM approach achieved a Youden's J of 1.00 on the benchmark.
• Real-World Performance: On the real-world dataset ($D_{W}$), the best LLM configuration achieved an F1-score of 0.88 after manual validation, indicating strong generalizability. The study estimated that approximately 72% of apps in the sample were obfuscated at a low threshold, consistent with trends of increasing obfuscation adoption.

Significance and Claims
The paper claims that LLMs represent a viable and effective approach for Android obfuscation analysis, offering a more flexible and generalizable solution than traditional rule-based or heuristic-driven methods. The authors emphasize that LLMs reason about the characteristics of the transformation (e.g., renaming patterns) rather than relying on tool-specific signatures, allowing them to generalize across different obfuscation tools.

The study highlights that while LLMs are not equally effective across all obfuscation families (struggling with complex control-flow or reflection-based techniques), their strong performance on identifier renaming—which is the most prevalent obfuscation strategy in the Android ecosystem (often via R8)—suggests they can provide meaningful benefits for a large fraction of real-world applications. The authors modestly position their work as a research prototype demonstrating the potential of off-the-shelf LLMs, rather than a production-ready tool, noting limitations such as computational overhead, model non-determinism, and the current scope limited to bytecode-level (Smali) analysis. Future work is proposed to investigate obfuscation in malware and to explore LLMs for code deobfuscation and semantic reconstruction.