Leveraging Large Language Models to Obscure Code Stylometry: A Comparative Study of GPT-3.5 and GPT-4

Technical Summary: Leveraging Large Language Models to Obscure Code Stylometry

Problem Statement
Code stylometry involves analyzing unique stylistic signatures in source code to attribute authorship, a technique critical for cybersecurity and tracing the origins of malicious software. Traditional stylometric methods rely on lexical, layout, and syntactic features (often derived from Abstract Syntax Trees) to identify programmers. However, the emergence of Large Language Models (LLMs) like GPT-3.5 and GPT-4 introduces a significant challenge: these models can generate or transform code while preserving functionality but altering stylistic fingerprints. This capability potentially allows malicious actors to evade authorship detection, complicating cybersecurity efforts. This study investigates the extent to which LLMs can successfully obscure code stylometry to evade detection by a machine learning classifier while maintaining code functionality.

Methodology
The research employed a structured experimental design involving four primary phases: dataset preparation, feature extraction, evaluator model training, and prompt engineering.

1. Dataset: The study utilized the Methods2Test dataset, a large corpus of Java code snippets paired with unit tests. For the experiments, a subset of the test set (78,388 pairs) was selected. The data was converted from JSON to CSV, cleaned of missing values, and assembled to include focal class and test class code in single columns.
2. Feature Extraction: A Python-based tool extracted 109 stylometric features categorized into three groups:
   • Lexical: Keyword counts, token counts, literal values, and line length statistics.
   • Layout: Indentation patterns (tabs vs. spaces), empty lines, and whitespace ratios.
   • Syntactic: Features derived from the Abstract Syntax Tree (AST), including maximum node depth and specific Java keyword occurrences.
3. Evaluator Model: A Random Forest classifier was trained to predict the repository ID (authorship) based on the extracted features. The model achieved a high baseline accuracy of 95.18% on the holdout test set, with cross-validation scores averaging 0.95, establishing a robust baseline for detecting stylistic signatures.
4. Prompt Engineering & LLM Interaction: The study tested GPT-3.5 and GPT-4 using four distinct prompt strategies, each implemented via two methods: Single-Shot (direct instruction) and Multi-Shot (step-by-step conversational guidance).
   • Strategy 1 (Just Change Style): General instruction to alter style without specific targets.
   • Strategy 2 (Specific Style Emulation): Instructing the model to mimic a specific programmer or regional convention.
   • Strategy 3 (Style with an Example): Providing a sample code snippet for the model to emulate.
   • Strategy 4 (Cross-Language Transformation): Translating Java to C++ and back to Java to force stylistic shifts.
5. Evaluation: The LLM-generated code was evaluated on two metrics:
   • Evasion Success: Whether the Random Forest classifier failed to correctly identify the original repository ID (a "0" in the results table).
   • Functionality Preservation: Whether the modified code compiled successfully and passed all original unit tests.

Key Results
The experiments yielded mixed results regarding the ability of LLMs to evade detection while maintaining functionality.

• Model Performance: GPT-4 generally outperformed GPT-3.5, achieving an average evasion success rate of 29% compared to 24% for GPT-3.5. GPT-4 showed particular strength in single-shot methods when using specific style emulation (Strategy 2, 40% success).
• Prompt Strategy Effectiveness:
  • Multi-Shot vs. Single-Shot: Multi-shot methods were generally more effective (28% average success) than single-shot methods (14% average success). Detailed, step-by-step guidance significantly improved GPT-3.5's performance, particularly in Strategy 1 (60% success).
  • Strategy Comparison: The Cross-Language Transformation (Strategy 4) was the most effective overall (35% average success). Specific Style Emulation (Strategy 2) followed at 30%. Just Asking to Change Style (Strategy 1) showed mixed results (25%), heavily dependent on the method used. Style with an Example (Strategy 3) was the least effective (15%).
• Functionality Preservation: A critical finding was the difficulty in maintaining code integrity. Out of 80 tested samples, only 58% (46 samples) compiled and passed all unit tests. A significant portion (41%) failed to compile, often due to LLM-induced changes to class names or data types.
• Dual Success: Only 10% of the total tested samples (8 out of 80) achieved the dual goal of successfully evading the stylometry classifier and maintaining full functionality.

Significance and Contributions
The paper posits that its primary contribution lies in providing critical insights into the robustness of current authorship attribution techniques against advanced AI capabilities. By demonstrating that LLMs can partially obscure stylistic signatures, the study highlights a vulnerability in traditional cybersecurity measures that rely on code stylometry.

The research underscores the importance of prompt engineering, revealing that detailed, structured prompts (multi-shot) are significantly more effective than simple instructions. Furthermore, it exposes the trade-off between stylistic obfuscation and code functionality; while LLMs can alter style, doing so without breaking code compilation remains a significant challenge.

Limitations and Future Directions
The authors acknowledge several limitations, including dataset constraints (reliance on open-source Java projects), computational resource constraints, and the manual nature of functionality verification. Future work is proposed to include larger and more diverse datasets, the development of automated pipelines for functionality testing, the exploration of alternative machine learning algorithms (such as LSTMs) for the evaluator model, and the inclusion of other enterprise LLMs (e.g., Gemini, Claude) for comparative analysis.

In conclusion, while GPT-4 and advanced prompting strategies show promise in obscuring code stylometry, the current technology does not yet allow for seamless, undetectable, and fully functional code transformation at scale. This suggests that while authorship attribution is under threat, it remains a viable, albeit challenged, field requiring continued development of more robust detection techniques.