Investigating Detection and Obfuscation of Prompt Injection Attacks Against Software Reverse Engineering AI Agents

Technical Summary: Investigating Detection and Obfuscation of Prompt Injection Attacks Against Software Reverse Engineering AI Agents

Problem Statement
Agentic software reverse engineering (SRE) systems, which utilize Large Language Models (LLMs) to analyze executable binaries, are vulnerable to indirect prompt injection attacks. In these attacks, adversarial strings are embedded within the source code of a benign executable. When the binary is decompiled and analyzed by an AI agent, the injected strings manipulate the LLM into ignoring the actual program logic and reporting the functionality of a different, target program. While prompt injection in general is a known risk, the specific vulnerability of agentic SRE systems to attacks embedded directly in executable binaries, and the robustness of defenses against obfuscated versions of these attacks, had not been previously examined in academic literature.

Methodology
The research employs a multi-faceted approach involving attack generation, defensive detection, and obfuscation testing:

1. Attack Generation (Genetic Algorithm): The authors utilized a modified version of the AutoDAN genetic algorithm framework to automatically discover prompt injection strings. The algorithm optimizes for attack strings that maximize the probability of the LLM generating a specific target response (decompiled code of a malicious target) while ignoring the actual decompiled code of the benign input. The attack strings consist of pseudo-system tokens (mimicking conversation turns), attack instructions, and the decompiled output of a target program, all embedded within a benign C program.
2. Defensive Detection: Two primary detection methods were investigated:
   • Regular Expression (Regex) Matching: A rule-based approach searching for specific pseudo-system token patterns (e.g., <tag>) within the strings extracted from the binary.
   • Neural Network (NN) Classification: A transfer learning approach using a frozen Qwen3-1.7B LLM as an embedding backbone. The embeddings are fed into a trainable classification head to predict the probability of a tool response (from either a "list strings" or "decompile function" tool) being malicious.
3. Obfuscation and Robustness Testing: The study tested the robustness of these defenses against several obfuscation techniques:
   • Token Variation: Modifying pseudo-system tokens (e.g., changing angle brackets < > to square brackets [ ] or altering bracket styles).
   • String Bypass: Inserting non-printing characters (e.g., \x01) to prevent the "list strings" tool from recognizing the injection string, while keeping it visible to the "decompile function" tool.
   • Learnable Parameters: Modifying the genetic algorithm to treat pseudo-system tokens and target programs as learnable parameters, allowing the attack to evolve against the specific detection model.

Key Results

• Regex Limitations: While regex matching achieved 100% precision, it failed significantly when attackers altered token formats (e.g., using square brackets). The success rate of attacks against the LLM dropped only slightly (from 65% to 50% for a larger model), indicating that regex is insufficient against format variations.
• Neural Network Effectiveness: The classification NN demonstrated superior robustness. It correctly classified all 20 adversarial examples as malicious even when the token formats were altered (square brackets), achieving an F1 score of 0.909 and perfect recall (1.000) on the test set.
• Obfuscation Success:
  • List Strings Bypass: When attacks were obfuscated to hide from the "list strings" tool (using non-printing characters), the NN failed to detect them because the malicious strings were absent from the input provided to the classifier.
  • Learnable Tokens: When the genetic algorithm was allowed to learn and vary pseudo-system tokens, it successfully generated attacks classified as benign by the NN for all 20 programs when using "list strings" output.
  • Decompile Function Vulnerability: When the detection was applied to "decompile function" output, the NN remained effective against format variations but was eventually bypassed when the genetic algorithm optimized for both the classification outcome and the injection success simultaneously. In this scenario, successful benign-classified attacks were found in 13 of 20 examples.
  • Target Learning: When the target program itself became a learnable parameter, the genetic algorithm successfully generated attacks for all programs, further reducing the robustness of the detection.

Key Contributions

• First Examination of Binary-Embedded Attacks: This work addresses the gap in literature regarding prompt injection attacks specifically embedded within executable binaries targeting agentic SRE systems.
• Detection Framework: The paper proposes and evaluates a classification-based detection method using transfer learning, demonstrating its superiority over simple regex matching against token variations.
• Obfuscation Analysis: The research systematically explores how obfuscation techniques (token variation, string hiding, and parameter learning) degrade the effectiveness of detection models, highlighting the fragility of current defenses.
• Dataset Creation: The authors generated a dataset of 100 benign and 40 adversarial examples, including specific configurations for "list strings" and "decompile function" tool responses, to train and evaluate the detection models.

Significance and Claims
The paper claims that while classification neural networks offer a more robust defense than regex matching for detecting prompt injection in SRE agents, they are not a complete solution. The efficacy of the detection is significantly compromised when attackers obfuscate the injection strings to bypass specific tool outputs (like "list strings") or when the attack parameters (tokens and targets) are optimized against the detector.

The authors conclude that the research advances the understanding of risks in agentic software analysis systems necessary for their deployment in production-level cyber workflows. They emphasize that defending these systems requires moving beyond static pattern matching and acknowledges that current detection methods are vulnerable to adaptive obfuscation, necessitating further research into more robust detection methods and analyses on more complex programs. The work serves as a cautionary demonstration of the self-defense challenges inherent in LLM-driven agentic systems.