💻 computer science

Revisiting Code Debloating with Ground Truth-based Evaluation

This paper introduces a ground-truth-based evaluation paradigm to address the lack of standardized benchmarks in application-level code debloating, revealing that existing tools suffer from significant functional incorrectness due to either excessive code removal by dynamic analysis or high false retention rates from coarse-grained static analysis.

Original authors: Muhammad Bilal, Moiz Ali, Mohit Kumar, Fareed Zaffar, Fahad Shaon, Ashish Gehani, Sazzadur Rahaman

Published 2026-04-21

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Muhammad Bilal, Moiz Ali, Mohit Kumar, Fareed Zaffar, Fahad Shaon, Ashish Gehani, Sazzadur Rahaman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Cleaning a Messy Garage

Imagine you have a massive garage (a computer program) that has been accumulating junk for years. It's full of old tools, broken bikes, and boxes you haven't opened in a decade. You want to debloat it—remove the junk to make the garage smaller, safer, and easier to navigate.

In the software world, this is called Code Debloating. The goal is to strip away unused code to make programs run faster, use less memory, and have fewer security holes (like leaving a backdoor open for thieves).

The Problem: The "Test Case" Trap

For a long time, researchers tried to automate this cleaning process. They built robots (debloating tools) to scan the garage and throw things away.

But how did they know if the robot did a good job? They used a flawed measuring stick:

The Old Method: They gave the robot a specific list of tasks (Test Cases). "Robot, please open the door and grab the hammer." If the robot did that without breaking, they assumed the garage was clean and safe.
The Flaw: Just because the robot could open the door and grab the hammer doesn't mean it didn't accidentally throw away the fire extinguisher, the first-aid kit, or the lock on the back door. The test cases were too narrow. They didn't catch the subtle, dangerous mistakes.

The authors of this paper say: "We've been lying to ourselves. We think we're cleaning the garage, but we might be burning it down."

The Solution: The "Ground Truth"

To fix this, the authors decided to stop guessing. They created a Ground Truth.

Think of this as a Master Blueprint or a Perfectly Cleaned Garage that was cleaned by a team of expert human inspectors. They manually went through 11 real-world programs (like mkdir, tar, nginx—the "garages" of the internet) and carefully removed only the junk, keeping everything essential.

Now, instead of asking, "Did the robot pass the test?", they can ask, "Does the robot's result match our Master Blueprint?"

The Experiment: Testing the Robots

The authors took 8 of the most famous "cleaning robots" (debloating tools) and let them clean the same 11 programs. Then, they compared the robots' work against their human-made Master Blueprint.

Here is what they found, broken down by the two main types of robots:

1. The "Aggressive" Robots (Dynamic Analysis)

Who they are: Tools like Blade, Chisel, and Razor.
How they work: They watch the program run with a specific set of tasks (like watching someone open the garage door). If a tool isn't used during that specific time, they throw it away.
The Result: They were too aggressive.
- The Analogy: Imagine a robot that sees you grab a hammer, so it assumes you don't need a screwdriver. It throws the screwdriver away. But then, you need to fix a shelf later, and you can't.
- The Finding: These tools removed up to 94% of the code that should have stayed. They threw away critical safety features, error messages, and thread locks (like removing the bolts holding the garage door together).
- The Danger: The programs still "worked" for the test tasks, but they were unstable. They could crash, get hacked, or delete files they shouldn't.

2. The "Cautious" Robots (Static Analysis)

Who they are: Tools like Lmcas, Trimmer, and Occam.
How they work: They look at the code structure without running it. They are very conservative. They think, "I'm not 100% sure this tool is useless, so I'll keep it just in case."
The Result: They were too safe.
- The Analogy: Imagine a robot that sees a box of old tools and thinks, "Well, maybe I'll need a wrench in 10 years." So, it keeps the entire box, even though you only need a hammer.
- The Finding: They kept almost 100% of the junk. They didn't break anything, but they didn't actually make the program smaller or faster. They just left the garage full of clutter.

3. The "Hybrid" Robots

Who they are: Tools like CovA.
The Result: They tried to mix both approaches but ended up inheriting the worst of both worlds. They still missed critical safety code because they relied too much on the "Aggressive" method's observations.

The 7 Hidden Disasters

The paper didn't just measure size; they looked at what was broken. They found 7 specific ways these robots fail, many of which were previously unknown:

Merging Mutually Exclusive Paths: The robot takes two different paths (like "If it's raining, take an umbrella" and "If it's sunny, wear sunglasses") and smashes them into one path. Now, you might wear sunglasses and an umbrella at the same time, breaking the logic.
Leaving "Residual" Danger: The robot removes the main function but leaves behind a half-broken door. If a hacker tries to use that door, the program crashes or leaks data.
Unsafe States: The robot removes the "safety check" that stops a program from running in the middle of a dangerous operation.
Thread Synchronization Failure: In multi-threaded programs (like a garage with 10 people working), the robot removes the "stop signs" (locks) that prevent people from crashing into each other. This causes chaos (deadlocks).
Removing Error Handling: The robot deletes the "Fire Alarm." If the program fails, it just silently crashes instead of telling you what went wrong.
Variable State Issues: The robot removes the initialization of a variable (like forgetting to set the thermostat to 70 degrees). The program then uses random, garbage numbers, leading to unpredictable behavior.
Broken Syntax: The robot cuts out a piece of code but forgets to close the bracket, making the whole program impossible to compile (like trying to build a house with a missing roof beam).

The Takeaway

The paper concludes that we cannot rely on test cases alone to judge if software is safe to shrink.

Dynamic tools (watching the code run) are too reckless and break safety.
Static tools (looking at the code structure) are too cautious and don't clean enough.

The Lesson: To truly clean up software, we need a better way to measure success. We need a "Ground Truth"—a human-verified standard—to ensure that when we remove code, we aren't accidentally removing the safety features that keep the system from falling apart.

In short: Don't just ask, "Did it pass the test?" Ask, "Did you throw away the fire extinguisher?"

1. Problem Statement

Software debloating aims to remove unused code to reduce performance overhead, attack surfaces, and maintenance costs. While application-level debloating is central to this field, its evaluation relies heavily on imperfect proxy metrics:

Correctness: Assessed via test-case coverage.
Efficiency/Security: Inferred from reductions in code size or Return-Oriented Programming (ROP) gadget counts.

The authors argue that these proxies are flawed. Test suites often fail to capture subtle semantic errors, leading to a "false sense of accomplishment" where tools report high code reduction with no reported correctness loss, yet the resulting binaries are unstable or insecure. Furthermore, there is a lack of standardized benchmarks to assess the true performance of debloating tools, particularly regarding False Removals (removing necessary code) and False Retentions (keeping unnecessary code).

2. Methodology: Ground Truth-Based Evaluation

To address the lack of reliable benchmarks, the authors introduced a Ground Truth-based Evaluation Paradigm.

Dataset Creation: They manually curated a dataset of 11 diverse real-world programs (ranging from 5K to 76K Lines of Code), including 9 GNU core utilities, 1 SIR benchmark, and nginx.
Manual Debloating Process:
- Phase 1 (Main Function Simplification): Annotators simplified the main() function by eliminating code paths associated with non-required flags, tracking global states, and performing dead code elimination.
- Phase 2 (Fine-grained Simplification): A breadth-first traversal of the remaining code to ensure every line contributed to the required functionality or critical error handling. This involved analyzing conditional statements, utilizing LLVM coverage, and consulting documentation.
- Validation: Two authors independently created ground truth versions for each program. Inter-annotator agreement was measured using Cohen's Kappa, yielding an average score of ~0.93, confirming the objectivity of the manual process.
Evaluation Metrics: The authors defined quantitative metrics comparing automated debloated outputs ( $P_d$ $P_{d}$ ) against the manual ground truth ( $P_{gt}$ $P_{g t}$ ):
- False Removal (FRm): Code incorrectly removed (critical for functionality/security).
- False Retention (FRt): Bloat incorrectly retained (critical for efficiency).
- F1-Score: A balance between FRm and FRt.
Tools Evaluated: 8 state-of-the-art tools were tested across three paradigms:
- Source-to-Source (S2S): Blade, Chisel, Cov, Cov𝐴 (Dynamic/Hybrid analysis).
- IR-to-IR (IR2IR): Lmcas, Trimmer, Occam (Static analysis).
- Binary-to-Binary (B2B): Razor (Dynamic analysis).

3. Key Results

Quantitative Findings

Dynamic Analysis Tools (S2S/B2B): Tools like Blade, Chisel, and Razor are highly aggressive. They achieved high code reduction but suffered from massive False Removals (up to 94%). For example, Blade removed 90% of essential lines in mkdir and 94% in sed.
Static Analysis Tools (IR2IR): Tools like Lmcas, Trimmer, and Occam are highly conservative. They achieved near-zero False Removals but suffered from massive False Retentions (70–100%), retaining large portions of unnecessary code due to coarse-grained dependency over-approximation.
Hybrid Approaches (Cov𝐴): While better than pure dynamic tools, they still exhibited significant False Removals (up to 45%) and False Retentions (up to 85%), as their static heuristics were biased by the gaps in dynamic coverage.

Qualitative Findings (7 Critical Issues)

The ground truth evaluation uncovered 7 distinct classes of issues, only 3 of which were previously known via test-case evaluations:

Failures in Logical Integrity: Tools merged mutually exclusive code blocks or forced independent blocks into nested structures, breaking control flow.
Residual Path Vulnerabilities: Partially removed code paths remained, which could be triggered by unsupported inputs, leading to crashes or security exploits (e.g., deleting files without output).
Unsafe Intermediate State Execution: Removing "guard logic" (e.g., checks for . and .. in directory traversal) caused the debloating process itself to crash or destroy system files (e.g., deleting /bin).
Thread Synchronization Failures: Dynamic tools removed critical mutexes and condition variables because race conditions are rarely triggered during standard test execution, leading to deadlocks or data corruption.
Removal of Error Handling: Critical error logging and handling code (especially for environment-dependent failures like disk full or kernel errors) was systematically removed because test suites cannot easily simulate these conditions.
Variable State Management: Removal of initialization code led to undefined behavior, such as propagating uninitialized pointers or variables, causing memory corruption.
Complex Control Structure Failures: Tools relying on regex/pattern matching (e.g., Cov) failed to maintain syntactic correctness in complex control flows, resulting in non-compilable code.

4. Key Contributions

Ground Truth Dataset: Creation of a manually curated, high-quality benchmark of 11 programs with verified ground truth debloated versions.
New Evaluation Paradigm: Introduction of a methodology that moves beyond test-case coverage to direct comparison with empirical ground truth.
Comprehensive Tool Analysis: A systematic evaluation of 8 tools revealing that dynamic analysis tools are dangerously aggressive (high FRm), while static analysis tools are overly conservative (high FRt).
Discovery of New Vulnerabilities: Identification of 7 critical issue classes, 4 of which were previously unknown to the community, highlighting that test cases are insufficient for validating debloating safety.

5. Significance and Implications

Limitations of Test Cases: The paper conclusively demonstrates that test-case-driven evaluation is fundamentally flawed for debloating. It cannot capture edge cases like race conditions, environment-dependent errors, or guard logic necessary for safety.
Security Risks: The "False Removals" identified are not just functional bugs; they introduce exploitable vulnerabilities (e.g., removing authentication checks, synchronization primitives, or error handlers).
Future Directions: The authors suggest that future debloating research must move toward ground-truth-based validation. They also note that while their dataset is source-code based, the principles apply to IR and binary debloating, though mapping instructions back to source remains a challenge.

In summary, the paper argues that the current state of software debloating is unsafe for production use due to the inability of existing evaluation methods to detect critical semantic errors, and it provides the necessary tools (dataset and metrics) to rigorously assess and improve future debloating technologies.