Technical Summary: CODEFUSE-DEBENCH

Problem Statement

Binary decompilation is a cornerstone of software security and reverse engineering, aiming to recover executable machine code into high-level source code. However, existing evaluation methodologies suffer from significant limitations:

1. Over-reliance on Syntactic Similarity: Current metrics often focus on superficial consistency (e.g., lines of code, character-level matching) or single-axis readability scores.
2. Misalignment with Practical Utility: These metrics fail to measure the actual engineering value of decompiled code. A decompiler might produce visually clean code that is semantically incorrect, or code that is difficult to read but behaviorally faithful.
3. Lack of Comprehensive Frameworks: There is no standardized, automated framework that simultaneously assesses readability, recompilability, and functionality across diverse compilation settings and architectures.

This gap creates confusion for users attempting to utilize decompiled code for downstream applications (e.g., patching, migration) and obscures the true capabilities and failure modes of modern decompilers.

Methodology: DEBENCH Framework

The authors propose DEBENCH, a reusability-driven evaluation paradigm that assesses decompilers along three orthogonal dimensions: Readability, Recompilability, and Functionality.

1. Benchmark Construction

• Atomic Test Suite: DEBENCH comprises 240 atomic test functions grouped into 8 dimensional source files covering diverse reverse engineering challenges (Control Flow, Data Types, Memory Operations, Function Calls, Object-Oriented C++, Compile-Time Specialization, System Interaction, and Special Challenges).
• Diversity: These sources are compiled into 640 distinct binaries by varying five axes:
  • Compilers: GCC, Clang
  • Optimization Levels: -O0, -O1, -O2, -O3, -Os
  • Debug Symbols: With -g, Stripped
  • Architectures: x86, x64, ARM32, ARM64
• Design Principles: The suite emphasizes Atomicity (isolating specific language features) and Progressive Complexity (categorizing cases into difficulty levels L1–L5 based on control/data-flow complexity, optimization resistance, and semantic loss risk).

2. Evaluation Pipeline

The framework employs a fully automated, three-stage pipeline:

• Stage 1: Readability (URAF)

  • Uses an LLM-as-Judge approach with the Unified Readability Assessment Framework (URAF).
  • Evaluates code against the ground-truth source across 18 sub-dimensions organized into 5 levels: Lexical Clarity, Structural Intelligibility, Type-System Fidelity, Semantic Transparency, and Contextual Coherence.
  • Employs three independent LLM judges (GLM-4.7, Qwen3.5-Plus, MiniMax-M2.5) to mitigate bias.

• Stage 2: Recompilability

  • Implements an LLM-driven Iterative Repair pipeline.
  • An automated agent parses compiler diagnostics (GCC/Clang) and delegates fixes to an LLM.
  • Operates under a fixed 50-iteration budget.
  • Outcomes are categorized into three tiers: Full Success (FS), Link Fail (LF), and Compile Fail (CF).

• Stage 3: Functionality

  • Uses Frida-based differential dynamic tracing to compare the original binary against the recompiled binary.
  • Observes runtime behavior at three granularities: Program Level (stdout consistency), Function Level (register/return value matching), and Instruction Level (sequence similarity).
  • Classifies results into: Exact Stdout, Partial, Fail, and Unsupported.

Key Contributions

1. Reusability-Driven Evaluation Paradigm: Shifts the focus from syntactic similarity to practical utility by jointly assessing readability, recompilability, and functionality.
2. End-to-End Automated Infrastructure (DEBENCH): The first framework to integrate LLM-as-judge scoring, iterative compile-and-repair, and dynamic behavior verification on a large-scale, multi-dimensional benchmark.
3. Empirical Insights: Provides a systematic characterization of five mainstream decompilers (IDA, Ghidra, RetDec, Angr, BinaryAI) and three repair LLMs, revealing failure modes and capability limits invisible to single-axis evaluations.

Key Findings

The study yields four primary findings that challenge conventional assumptions:

1. The Reusability Cliff: There is a steep, reproducible gap between recompilability and behavioral equivalence.

   • The best decompiler–LLM pair (IDA + Qwen3.5-Plus) achieves 73.8% recompilability but only 22.3% program-level behavioral overlap (Exact + Partial).
   • Exact Stdout match is only 1.2%.
   • This represents a ~50 percentage point cliff between code that can be compiled and code that behaves correctly.

2. Decoupling of Readability and Functionality: Compilation choices that maximize readability do not necessarily maximize functionality.

   • Optimization: Aggressive -O3 yields the lowest readability scores but the highest functionality (16.0% Exact+Partial vs. 8.6% for -O0).
   • Compiler: Clang produces lower readability scores than GCC but results in 2.6× higher functionality (26.5% vs. 10.2% Exact+Partial).

3. Asymmetric Gaps (Decompiler vs. LLM): Engineering investment should target decompiler engines, not larger repair models.

   • Switching decompilers (e.g., RetDec to IDA) under a fixed LLM causes a 20× swing in functionality (1.1% to 22.3%).
   • Switching LLMs under a fixed decompiler causes only a 1.6× swing (13.9% to 22.3%).

4. Failure Decomposition: Failures fall into three categories with distinct reparability:

   • Syntactic Noise: Repairable by LLMs (e.g., missing headers, syntax errors).
   • Type-System Collapse: Accounts for ~19% of errors (e.g., conflicting types, pointer mismatches). Partially repairable but often leads to behavioral drift.
   • Irreversible Upstream Losses: Unfixable by LLMs. Includes ARM64-specific relocation idioms (ADRP+ADD) and C++ ABI features (vtables, virtual dispatch) that decompilers fail to reconstruct.

Significance and Implications

The paper reframes decompiler quality as a multi-axis continuum rather than a single score. It argues that "moderate readability" often overstates downstream utility by roughly 3×.

The authors identify three priority directions for decompiler-side engineering to bridge the gap from syntactic translation to semantic reconstruction:

1. Composite-Type Reconstruction: Recovering complex types that survive symbol stripping.
2. IR-Level Symbolic Relocation: Tracking relocations (especially on ARM64) without collapsing them into opaque arithmetic.
3. C++ ABI-Aware Front-End Passes: Specifically for reconstructing vtables and virtual dispatch to handle Object-Oriented C++ code.

The study concludes that while LLMs are effective for surface-level repair, the fundamental bottlenecks lie in the decompiler engines' ability to recover semantic structure and type information, necessitating engine-level innovation over prompt-level tuning.