Technical Summary: AutoDecompiler

Problem Statement

Binary decompilation is critical for security tasks such as vulnerability discovery and malware analysis, yet it faces a fundamental challenge: existing Large Language Model (LLM) approaches largely operate under a single-turn generation paradigm. In this paradigm, an LLM generates decompiled code once and terminates. While the output may be syntactically readable and even compile successfully, it often deviates from the semantic behavior of the original binary. Current methods fail to leverage diagnostic signals—such as compiler errors, runtime exceptions, and failed input/output (I/O) tests—to iteratively correct these semantic errors. Furthermore, existing reinforcement learning (RL) applications in code generation typically optimize for a single continuous token sequence, rather than a multi-turn refinement process where the model must diagnose and revise its own outputs based on environmental feedback.

Methodology

The paper proposes AutoDecompiler, a decompilation-specialized LLM trained with reinforcement learning to perform feedback-driven multi-turn binary decompilation. The system treats decompilation not as a one-shot task, but as an iterative refinement loop where the model revises code based on validation feedback.

Training Pipeline

The construction of AutoDecompiler follows a two-stage pipeline:

1. Domain-Specialized Supervised Fine-Tuning (SFT): The base model (DeepSeek-Coder or Qwen3-Coder) is fine-tuned on a balanced subset of 300K binary functions from the CID dataset. This stage adapts the model to map assembly code or decompiler-generated pseudo-code into C-like source code, establishing a static input-output mapping.
2. Reinforcement Learning (RL) for Multi-Turn Refinement: The SFT-initialized model is further optimized using the GRPO (Group Relative Policy Optimization) algorithm. In this stage, the model generates a code candidate, which is validated through C-code validity checks, recompilation, re-execution, and I/O testing. The resulting feedback is used to compute rewards and construct diagnostic prompts for subsequent refinement turns.

Key Technical Innovations

To address the challenges of multi-turn refinement, the authors introduce four novel mechanisms:

1. Weighted Multi-dimensional Decompilation Reward:
   Instead of relying on a single objective, the reward function ($R_t$) is a weighted sum of four components:

   • C-like Validity ($R_C$): Ensures the output is a syntactically valid C function, filtering out hallucinations.
   • Compilation and Execution Correctness ($R_{exe}$): A staged reward that penalizes recompilation failures most heavily, followed by runtime errors and I/O mismatches, encouraging the model to produce executable and behaviorally consistent code.
   • Syntactic Reward ($R_{syn}$): Measures AST-level similarity between the generated code and the reference source to preserve structural plausibility.
   • Semantic Reward ($R_{sem}$): Based on data-flow matching, this ensures variable dependencies and value propagation relations are preserved, addressing semantic fidelity beyond syntax.

2. Stage-aware Diagnostic Feedback Construction:
   The system transforms heterogeneous validation signals (compiler errors, runtime crashes, I/O failures) into natural language diagnostic messages. These messages are categorized by failure stage (e.g., "Compilation Error," "Runtime Error," "I/O Mismatch") and embedded into the prompt for the next turn, providing actionable guidance rather than scalar rewards alone.

3. Progress-aware Trajectory Rewarding:
   To prevent the model from learning shortcuts or drifting from the original semantics, the trajectory reward ($R_{traj}$) combines the final outcome reward with a progress term. This term measures the reward improvement ($\Delta r_t$) between consecutive turns, encouraging steady improvement and penalizing regressions (e.g., turning compilable code into non-compilable code).

4. Turn-aware Advantage Reweighting:
   Standard RL assigns a uniform advantage to all tokens in a sequence. AutoDecompiler introduces a strategy to redistribute the trajectory advantage based on the contribution of each specific refinement turn. Turns that yield significant reward improvements receive stronger optimization signals, while ineffective or harmful revisions are down-weighted.

Results

The AutoDecompiler family was trained on 310K binary functions and evaluated on the HumanEval and ExeBench benchmarks across two input settings: end-to-end (assembly input) and pseudo-code-based. Models were instantiated at 1.3B, 6.7B, and 30B parameter scales.

• Performance vs. Baselines: AutoDecompiler consistently outperformed its single-turn counterparts and other state-of-the-art (SOTA) decompilation LLMs (e.g., LLM4Decompile, CIM, FAE) under the same model size and input settings.
  • On HumanEval, the 6.7B AutoDecompiler-E2E achieved a Re-executability (Re-exe) rate of 63.11%, outperforming the previous best (CIM-6.7B at 62.05%) and significantly surpassing general-purpose models like GPT-4o and DeepSeek-V3.
  • In the pseudo-code setting, AutoDecompiler-Pscode 30B achieved competitive results (75.30% Re-exe) against DecLLM using a 671B parameter model, despite having far fewer parameters.
• Efficiency: The model achieved these results using a training dataset (310K samples) substantially smaller than prior works (e.g., LLM4Decompile used 7.2M samples), demonstrating the efficiency of the feedback-driven refinement approach.
• Ablation Studies: Removing any key component (e.g., execution rewards, progress-aware trajectory rewarding, or turn-aware advantage reweighting) resulted in performance degradation, confirming that each mechanism contributes to the system's effectiveness.
• Case Studies: Qualitative analysis showed that AutoDecompiler successfully repaired specific semantic errors (e.g., missing boundary conditions) and interface mismatches (e.g., incorrect function signatures) by leveraging execution and compilation feedback.

Significance and Claims

The paper claims that learning to exploit program feedback with reinforcement learning is an effective direction for improving the functional correctness of LLM-based binary decompilation.

Key contributions include:

• Paradigm Shift: Formulating decompilation as a feedback-driven multi-turn refinement problem rather than a single-turn generation task.
• Specialized Reward Design: Introducing a multi-dimensional reward system that jointly optimizes for validity, recompilability, execution consistency, and semantic fidelity, moving beyond superficial code plausibility.
• Refinement Mechanisms: Developing stage-aware feedback and turn-aware advantage reweighting to ensure progressive improvement and suppress regressions during iterative refinement.
• Open Source: The authors have open-sourced the AutoDecompiler family to facilitate community research.

The authors acknowledge limitations, noting that the number of refinement turns is constrained by context length, the current scope is limited to single-function decompilation (lacking inter-procedural context), and generalization to heavily obfuscated binaries remains an open challenge. However, the results demonstrate that a smaller, specialized model trained with iterative feedback can achieve superior functional correctness compared to larger, single-turn models.