🤖 AI

Evaluating Fine-Tuning and Metrics for Neural Decompilation of Dart AOT Binaries

This paper presents a systematic empirical study on neural decompilation of Dart AOT binaries, revealing that fine-tuning often fails to improve functional correctness (pass@k) despite superficial metric gains, while introducing a new HumanEval-Dart benchmark and demonstrating that assembly sequence length is the primary predictor of task difficulty.

Original authors: Raafat Abualazm, Ayman AboElhassan, Amr G. Wassal

Published 2026-07-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Raafat Abualazm, Ayman AboElhassan, Amr G. Wassal

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a secret recipe written in a highly compressed, coded language (machine code) that a computer understands perfectly, but a human chef cannot read. Your goal is to translate this code back into a readable, delicious recipe (source code) that anyone can follow. This process is called decompilation.

Recently, scientists have started using "AI chefs" (Large Language Models) to do this translation. But there's a big problem: How do we know if the AI chef actually recreated the dish correctly, or if it just made something that looks like a recipe but tastes terrible?

This paper is a rigorous taste-test of these AI chefs, specifically trying to translate Dart code (used in apps like Flutter) from its machine form back to human-readable form. Here is what they found, explained simply:

1. The "Smell Test" vs. The "Taste Test"

The researchers discovered a dangerous trap in how we usually judge these AI chefs.

The Smell Test (Surface Metrics): They used tools like CodeBLEU and compile@k. These check if the AI's output looks like a recipe (correct grammar, correct structure) and if it can be "cooked" (compiled) without the kitchen exploding.
The Taste Test (Functional Correctness): They used a metric called pass@k. This actually runs the recipe to see if it produces the correct dish (does the math work? does the logic hold?).

The Shocking Finding: The AI chefs often passed the "Smell Test" perfectly but failed the "Taste Test."

Analogy: Imagine an AI writes a recipe that looks perfect, uses the right ingredients, and follows the steps. But if you actually cook it, the cake collapses. The "Smell Test" said "Great job!" but the "Taste Test" said "This is inedible."
In fact, for some AI models, the "Smell Test" scores went up after training, while the actual ability to cook the dish correctly went down significantly.

2. The "Training Trap" (Fine-Tuning)

The researchers tried to make the AI chefs better by giving them extra practice (fine-tuning) on specific examples of Dart code. They expected this to help.

The Result: It mostly made things worse.
Analogy: Imagine a brilliant, general-purpose chef (a large AI model) who can figure out how to cook almost anything by understanding the principles of cooking. You then force them to memorize 1,000 specific recipes by rote.
- What happened: The chef stopped using their brain to understand why a dish works and started just memorizing patterns. They became great at simple, short recipes but forgot how to handle complex, long ones.
- The Irony: The strongest, smartest chef (the 8-billion-parameter model) actually got worse after this specific training. Their ability to solve hard problems dropped by nearly 6 percentage points, while the "Smell Test" scores barely changed. The training essentially "overwrote" their natural reasoning with rigid patterns.

3. The "Language Clash" (Cross-Lingual Interference)

They also tried training the AI on a mix of Dart and another language, Swift, to see if it could learn both at once.

The Result: For the smaller, less powerful AI chefs (4-billion parameters), mixing in Swift was a disaster. It confused them, and their Dart translation skills tanked.
The Silver Lining: For the bigger, smarter chefs (8-billion parameters), the confusion disappeared. They were big enough to handle both languages without mixing them up.
Analogy: If you teach a small child to speak two very different languages at once, they might get confused and speak neither well. But a teenager (the larger model) can handle both languages easily without mixing them up.

4. The "Length Limit"

The researchers analyzed why the AI failed. They found one major culprit: Length.

The Cliff: If the machine code was short (under 50 lines), the AI could often translate it. But once the code got longer (over 200 lines), the AI hit a "cliff" and stopped working almost entirely.
Analogy: It's like asking someone to translate a short sentence versus a whole novel. The AI could handle the sentence, but the moment the story got long and complex, it got lost. Traditional measures of "complexity" (like how many loops or variables were in the code) didn't matter as much as the sheer length of the machine instructions.

Summary of the Takeaways

Don't trust the "Smell Test": Just because AI-generated code looks good or compiles doesn't mean it works. You must test if it actually does the job (pass@k).
More training isn't always better: For smart AI models, forcing them to memorize specific tasks can actually make them dumber at solving new problems.
Size matters for multitasking: If you want an AI to learn multiple languages at once, it needs to be big enough to handle the confusion.
Short is sweet: Current AI can only reliably translate short snippets of code. Long, complex machine code is still too difficult for them.

The paper concludes that we need to stop relying on superficial metrics and start testing AI decompilers by actually running the code to see if it works, just like you wouldn't judge a restaurant solely by how pretty the menu looks.

Technical Summary: Evaluating Fine-Tuning and Metrics for Neural Decompilation of Dart AOT Binaries

1. Problem Statement

Neural decompilation, the task of translating machine code back into high-level source code, has traditionally focused on C and C++ targets. However, modern programming languages like Dart (used in Flutter) present unique challenges due to their Ahead-of-Time (AOT) compilation pipelines, which apply aggressive optimizations such as type specialization, function inlining, and dead code elimination. These optimizations strip away high-level semantic information, making the resulting x86-64 assembly significantly harder to reverse than unoptimized C.

Furthermore, the evaluation methodology for neural decompilation remains underdeveloped. Prior studies often rely on surface-level metrics like CodeBLEU or compilation success rates, which may not correlate with functional correctness. There is a critical gap in understanding whether fine-tuning Large Language Models (LLMs) on assembly-source pairs actually improves functional correctness, how cross-lingual training affects performance, and whether standard evaluation metrics reliably predict the utility of decompiled code.

2. Methodology

2.1 Dataset Construction

The authors constructed a new benchmark, HumanEval-Dart, comprising 154 Dart functions adapted from the HumanEval benchmark, each equipped with comprehensive unit test harnesses.

Natural Data: 246 Dart functions sourced from RosettaCode.
Synthetic Data: 948 synthetic Dart and 754 synthetic Swift function pairs generated by multiple LLMs (GPT-4, Claude, DeepSeek, Qwen).
Token-Matching: To isolate the effect of language choice from data volume, the authors created two training datasets with token counts matched within 2%:
1. Dart+Synthetic: 1,194 assembly–Dart pairs.
2. Dart+Swift: 1,000 assembly–source pairs (246 Dart + 754 Swift).
Optimization Note: Dart training data used AOT-optimized binaries, while Swift data used unoptimized (-O0) binaries, introducing a known confounding factor regarding optimization levels.

2.2 Experimental Design

The study employed a $3 \times 2$ factorial design, fine-tuning six model variants across three base architectures:

Qwen3-4B (4B parameters)
DeepSeek-R1-0528-Qwen3-8B (8B parameters, reasoning-distilled)
Qwen3-8B (8B parameters, standard pretraining)

Each base model was fine-tuned using LoRA with DoRA (Weight-Decomposed Low-Rank Adaptation) on both the Dart+Synthetic and Dart+Swift datasets.

2.3 Evaluation Metrics

The study evaluated models using three complementary metrics:

CodeBLEU: Semantic similarity (n-gram, AST, and data-flow matching). A custom Dart implementation was developed.
compile@k: Syntactic validity (proportion of generated outputs that compile).
pass@k: Functional correctness (probability that at least one of $k$ outputs passes all unit tests), calculated using the unbiased estimator.

2.4 Statistical Analysis

The authors applied rigorous paired statistical tests to compare fine-tuned variants against their respective base models:

McNemar's exact test for binary pass@1 outcomes (solved vs. not solved).
Bootstrap confidence intervals for mean differences.
Wilcoxon signed-rank tests and t-tests for continuous metrics.

3. Key Results

3.1 Fine-Tuning Effectiveness (RQ1)

Contrary to the hypothesis that fine-tuning improves performance, no fine-tuned configuration produced a statistically significant improvement in pass@k.

Qwen3-8B (Strongest Base): Fine-tuning caused a highly significant regression. The Dart+Synthetic variant (v5) dropped from 6.36% to 0.71% pass@1 ( $p < 0.001$ ), losing 22 tasks and gaining none. The Dart+Swift variant (v6) showed a similar catastrophic drop.
Qwen3-4B (Weakest Base): The Dart+Synthetic variant (v1) showed a directionally positive but non-significant trend (+0.71 pp, $p=0.21$ ).
DeepSeek-R1-8B: Showed no significant change, with a slight non-significant regression.

3.2 Cross-Lingual Interference (RQ2 & RQ3)

The study investigated whether training on Swift code interferes with Dart decompilation.

Scale-Dependent Interference: At 4B, adding Swift training data caused a significant degradation in Dart pass@k ( $\Delta = -2.66$ pp, $p < 0.001$ ).
Scaling Hypothesis: At 8B, this interference became statistically indistinguishable from zero. However, the authors note this is complicated by the fact that both 8B variants suffered massive regressions due to fine-tuning itself, making the "equivalence" between the two degraded models less meaningful.

3.3 Metric Divergence (RQ4)

A critical finding is the divergence between surface metrics and functional correctness.

Case Study (v3): The DeepSeek-R1-8B variant trained on Dart+Synthetic showed significant improvements in CodeBLEU (+0.051, $p=0.001$ ) and compile@1 (+16 pp). However, its pass@1 regressed (from 3.90% to 3.12%), though not significantly.
Case Study (v5): The Qwen3-8B variant showed a massive collapse in pass@1 (from 6.36% to 0.71%) while CodeBLEU and compile@k remained relatively stable or showed only modest declines.
Conclusion: Models can learn to produce syntactically valid and superficially similar code (high CodeBLEU) while failing to capture the underlying logic required for functional execution.

3.4 Error and Complexity Analysis

Assembly Length: The strongest predictor of task difficulty was assembly sequence length ( $p=0.001$ ). A "capability cliff" exists at approximately 200 instructions; solvability drops from 48% for tasks <50 lines to 6% for tasks >200 lines.
Traditional Metrics: Standard code complexity metrics (cyclomatic complexity, nesting depth) showed negligible predictive power.
Trade-off: Fine-tuning at 4B parameters appeared to trade "range" for "reliability." The fine-tuned model (v1) became more reliable on short, simple assembly sequences but lost the ability to solve complex tasks that the base model occasionally solved via stochastic general reasoning.

4. Contributions

HumanEval-Dart Benchmark: A new benchmark of 154 Dart functions with unit tests for evaluating functional correctness in decompilation.
Dart CodeBLEU Implementation: A language-specific implementation of CodeBLEU with AST and data-flow analysis for Dart.
Empirical Evidence on Fine-Tuning: A systematic study demonstrating that task-specific fine-tuning on small-to-medium models (4B–8B) does not guarantee functional improvement and can cause significant regressions, particularly in stronger base models.
Metric Divergence Demonstration: Evidence that CodeBLEU and compile@k can improve while pass@k regresses, challenging the reliance on surface metrics in code generation tasks.
Cross-Lingual Analysis: Controlled evidence that cross-lingual interference is significant at smaller scales but attenuates with model capacity, though confounded by optimization mismatches.

5. Significance and Claims

The paper positions itself as a systematic analysis of evaluation methodology and fine-tuning effectiveness rather than a deployment-ready decompilation system. The authors emphasize that absolute pass@k numbers remain low across all models, including commercial systems (Qwen3-Max: 18.38% pass@1).

The primary significance lies in the cautionary findings:

Rejection of Surface Metrics: The study argues that CodeBLEU and compile@k are insufficient proxies for functional correctness in decompilation. A model can appear to improve on surface metrics while becoming functionally useless.
Capacity-Dependent Fine-Tuning: The authors propose a hypothesis (not a confirmed law) that fine-tuning may overwrite valuable general reasoning capabilities in stronger models, leading to catastrophic regression, while having neutral or slightly positive effects on weaker models.
Evaluation Protocol: The paper advocates for pass@k with unit tests as the primary evaluation metric for neural decompilation and code generation tasks where functional correctness is paramount.

The authors explicitly state that their findings on the "capacity-dependent trend" are based on three architectures and one hyperparameter configuration, framing them as hypothesis-generating observations that motivate further scale sweeps rather than definitive scaling laws. They also acknowledge the limitation of optimization mismatches in their cross-lingual experiments, noting that future work with optimization-matched datasets is required to isolate linguistic interference from distributional shifts.