SoK: AI-Augmented Binary Reversing

This paper presents the first comprehensive systematization of knowledge on AI-augmented binary reversing by analyzing 144 research papers to establish a unified taxonomy that connects traditional techniques with modern AI approaches, thereby clarifying the field's evolution, identifying persistent challenges, and guiding future research.

The Gist

Imagine you have a delicious, complex cake, but all you have left is the empty, crumpled wrapper and a few crumbs. You know the cake was made with flour, eggs, and sugar, but you don't know the recipe, the brand of the ingredients, or who baked it. Binary reversing is the art of trying to figure out the original recipe just by looking at that wrapper and those crumbs.

This is a huge challenge for computer security experts. When software is built (compiled), the "secret sauce" (the original source code, variable names, and logic) is thrown away, leaving only a machine-readable mess. Trying to understand this mess is like trying to read a book written in a language you don't speak, where the words have been scrambled and the page numbers are missing.

For a long time, humans had to do this manually, which is slow and exhausting. Recently, Artificial Intelligence (AI) has stepped in to help, acting like a super-powered detective. However, because so many different researchers are using AI in different ways, the field has become a bit chaotic and confusing.

This paper is a Systematization of Knowledge (SoK). Think of it as a massive, organized map that brings order to this chaos. The authors looked at 144 research papers published since 2015 to create a unified guide.

Here is the simple breakdown of what they found:

1. The Two-Step Dance (The Pipeline)

The paper explains that AI doesn't just "read" the binary file directly. It's a two-step process:

• Step 1: The Human/Tool Detective (Conventional Pipeline): First, traditional tools (like disassemblers) take the messy binary file and turn it into "clues" or artifacts. These clues could be a list of instructions, a map of how the code jumps around (graphs), or a list of numbers.
• Step 2: The AI Detective (AI-Augmented Pipeline): The AI then looks at these clues. It doesn't see the raw binary; it sees the clues the tools prepared. It learns patterns from these clues to guess what the original code was doing.

The Analogy: Imagine a crime scene. The police (traditional tools) collect fingerprints, DNA, and photos (artifacts). The AI (the detective) then analyzes those photos and DNA samples to solve the case. The AI isn't at the crime scene itself; it's analyzing the evidence the police gathered.

2. The 22 Different "Cases"

The authors organized all the AI research into 22 different types of "cases" (domains) that AI is trying to solve. They range from simple to very complex:

• The Basics: Finding where one function ends and another begins (like finding where one paragraph ends and the next starts).
• The Detective Work: Identifying if a piece of code is malware (a virus) or if it was written by a specific hacker group.
• The Translation: Trying to guess the original names of variables (like guessing that a variable named x123 was actually called user_password).
• The Reconstruction: Trying to rewrite the code back into a human-readable language (decompilation).

3. The Big Problems (Validity Risks)

Even though AI is getting better, the paper warns that there are some serious traps, like a detective who is too confident but actually wrong.

• The "Echo Chamber" Problem: Many AI models are trained on the same few types of software (like common Linux tools). If you show them a brand-new type of software they've never seen, they might fail. It's like a student who memorized the answers to last year's test but can't solve a new problem.
• The "Garbage In, Garbage Out" Problem: If the traditional tools make a mistake when creating the "clues" (artifacts), the AI will learn from that mistake. If the map is wrong, the AI will get lost.
• The "Magic Trick" Problem: Sometimes AI seems to work great, but it's actually just memorizing patterns in the data rather than truly understanding the code. It's like a parrot repeating words it doesn't understand.

4. The Future: From Assistant to Partner

The paper suggests that AI shouldn't just be a tool that gives a single answer. The future is Agentic AI.

• Current State: You ask the AI, "Is this code a virus?" and it says "Yes."
• Future State: You tell the AI, "I need to understand how this malware steals data." The AI then acts like a partner: it plans a strategy, uses different tools to gather evidence, checks its own work, and says, "I think it steals data because of this specific line, but I'm not 100% sure, so here is the evidence I found."

Summary

This paper is a roadmap for the next decade of AI in cybersecurity. It tells us:

1. We have a lot of data: We've mapped out 22 different ways AI is being used to reverse-engineer software.
2. We need better maps: The field is messy, and we need a common language to talk about it.
3. We need to be careful: AI is powerful, but it can be fooled by bad data, limited training, or tricky code.
4. The goal isn't to replace humans: The best future is where AI acts as a tireless assistant that gathers evidence and proposes theories, while human experts make the final judgment calls.

In short, the paper says: "AI is a fantastic new tool for unlocking the secrets of software, but we need to stop treating it like magic and start understanding exactly how it works, where it fails, and how to use it safely."

Technical Summary

Technical Summary: SoK: AI-Augmented Binary Reversing

1. Problem Statement

Binary reversing is a fundamental capability for software understanding, vulnerability discovery, malware investigation, and firmware auditing. However, it remains inherently challenging because compilation irreversibly discards high-level semantic information (e.g., variable names, types, and design intent), and compiler optimizations transform program structures, causing semantically equivalent source code to yield vastly different binaries. Furthermore, many reversing tasks face computability limits, making perfect automated semantic recovery infeasible.

While traditional techniques (static/dynamic analysis, symbolic reasoning) have partially addressed these challenges, they often rely heavily on human experts to reconcile incomplete or ambiguous evidence. Recently, Machine Learning (ML), Large Language Models (LLMs), and agentic AI systems have been adopted to augment these workflows. However, this rapid adoption has resulted in a fragmented research landscape. Studies are scattered across different domains, artifact representations, learning approaches, and evaluation practices. This lack of synthesis makes it difficult to assess which techniques are robust, where they provide meaningful assistance, and what barriers remain for practical deployment.

2. Methodology and Scope

This paper presents a Systematization of Knowledge (SoK) based on a comprehensive review of 144 research papers published between 2015 and 2025 across 11 top-tier venues in security, software engineering, and artificial intelligence.

Scope Constraints:

• Inclusion: Research leveraging binary analysis artifacts as inputs to ML/LLM models for binary reversing tasks.
• Exclusion: Approaches relying on source code during inference, Android bytecode (Java/Dalvik), and interpreted-language bytecode (e.g., Python). The study focuses strictly on native executables produced by compilers.

Taxonomic Framework:
The authors introduce a unified taxonomy connecting two distinct pipelines via an artifact interface:

1. Conventional Pipeline: Generates analysis artifacts through four modes: Triage, Static Analysis, Dynamic Analysis, and Security Testing.
2. AI-Augmented Pipeline: Consumes these artifacts, transforms them into learning-ready representations, and performs downstream inference.

The taxonomy organizes the field into 22 binary reversing domains categorized by 8 inference types:

• Classification: Assigning targets to predefined categories.
• Clustering: Grouping similar instances into unlabeled clusters.
• Detection: Determining the presence of a property.
• Discovery: Identifying unknown candidates for validation.
• Reconstruction: Inferring latent structural relations.
• Recovery: Restoring source-level semantics lost during compilation.
• Restoration: Reversing intentional concealment (e.g., obfuscation).
• Summarization: Generating concise semantic descriptions.

3. Key Contributions

The paper makes three primary contributions:

1. Comprehensive Systematization: The first holistic view of AI-augmented binary reversing, organizing 144 papers into 22 domains and mapping the relationships between reversing tasks, artifacts, representations, and learning paradigms.
2. Unified Taxonomy and Visualization: A framework bridging conventional and AI-augmented pipelines through analysis artifacts, accompanied by an interactive visualization platform to support researchers and practitioners.
3. Critical Insights and Future Directions: An evaluation of the field's maturity, identification of persistent technical challenges and validity risks, and the proposal of promising research directions.

4. Key Findings and Results

4.1. Landscape and Maturity (RQ1)

• Domain Imbalance: Progress is uneven. Domains with stable labels and tractable evaluation criteria (e.g., code boundary detection, binary similarity) are mature. Conversely, generative semantic recovery (e.g., pseudocode generation, variable name recovery) is less mature due to underconstrained outputs and vague evaluation criteria.
• Artifact Imbalance: Learning is heavily skewed toward static artifacts (bytes, assembly, CFGs) because they are easy to collect. Behavior-grounded artifacts (dynamic traces, memory snapshots) are underutilized despite their significance for validating runtime semantics and analyzing protected code.

4.2. Validity Risks in AI-Driven Pipelines (RQ2)

The paper identifies ten critical insights regarding validity risks:

• Corpus Validity: Many studies rely on "corpus monoculture" (limited software diversity, specific compilers/architectures), risking poor generalization to real-world, proprietary, or obfuscated binaries.
• Data Leakage: Train-test splits at the binary or function level often fail to prevent code overlap (e.g., shared libraries), leading to over-optimistic performance estimates based on memorization rather than generalization.
• Ground Truth Fidelity: Labels are often proxies (e.g., debug symbols, decompiler outputs) rather than semantic truths. These proxies contain noise and inconsistencies, placing an upper bound on model quality.
• Tool Dependence: AI models inherit the errors and limitations of upstream reversing tools (disassemblers, decompilers). Tool-specific artifacts can propagate errors through the learning pipeline.
• Representation Validity: Feature engineering is a "semantic commitment," not neutral preprocessing. Choices in canonicalization and tokenization determine what information is preserved or discarded, yet there is no principled guidance on optimal representations for specific tasks.
• Model Validity: Reported gains may stem from extensive hyperparameter tuning rather than algorithmic novelty. Robustness is often under-characterized, with models failing to generalize across distribution shifts (e.g., new compiler versions, obfuscation).
• Evaluation Validity: Standard metrics (accuracy, BLEU, F1) provide only indirect evidence of semantic recovery. They often fail to capture behavioral correctness or actual analyst utility.

4.3. Challenges and Future Directions (RQ3)

The paper outlines several critical directions for future research:

• Common Evaluation Framework: The need for reproducible testbeds that explicitly capture build recipes, tool provenance, and artifact-generation procedures to distinguish algorithmic gains from dataset/tooling artifacts.
• Semantic Validation: Moving beyond artifact-level agreement to evidence-backed semantic claims that combine static, dynamic, and symbolic evidence.
• Adversarial Robustness: Developing models robust to distribution shifts, including obfuscation, packing, and virtualized code, which remain largely underexplored.
• Advanced Multimodal Reasoning: Integrating multiple evidence sources (static, dynamic, symbolic) while preserving traceability and quantifying uncertainty.
• Goal-Driven Agentic Reasoning: Transitioning from prompt-driven assistance to goal-directed agents that plan analysis, select tools, and synthesize hypotheses based on analyst objectives.
• Autonomy Spectrum: Viewing the evolution from AI-augmented to AI-powered reversing as a spectrum where human roles shift from routine analysis to high-stakes judgment and objective specification.

5. Significance

This paper asserts that AI-augmented binary reversing complements, rather than replaces, conventional reversing. By establishing a common vocabulary and structured framework, the authors clarify the current state of the field. The study highlights that while significant progress has been made in specific domains, the field lacks a unified understanding of its methodological foundations and evaluation practices. The insights provided aim to guide the development of the next generation of reliable, scalable, and generalizable AI-augmented binary reversing systems, ensuring that future advancements address the fundamental challenges of semantic recovery in the presence of incomplete information and adversarial conditions.