💻 computer science

Residual Risk Analysis in Benign Code: How Far Are We? A Multi-Model Semantic and Structural Similarity Approach

This paper introduces Residual Risk Scoring (RRS), a framework combining semantic and structural similarity analysis to reveal that approximately 61% of patched functions in the PrimeVul benchmark retain significant residual security risks, thereby challenging the assumption that patched code is inherently benign.

Original authors: Mohammad Farhad, Shuvalaxmi Dass

Published 2026-04-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mohammad Farhad, Shuvalaxmi Dass

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: "The Band-Aid Problem"

Imagine you have a leaky roof. You put a Band-Aid on it to stop the water for now. Technically, the leak is "fixed," and the house is no longer "vulnerable" to the immediate drip.

But here's the catch: The rest of the roof is still old, rotting, and full of weak spots. If you just look at the roof, it looks almost exactly the same as it did before you put the Band-Aid on.

This paper asks a scary question: Just because we patched the hole, does that mean the house is actually safe? Or did we just cover up a problem while leaving the rest of the structure dangerously close to how it was?

The authors call this "Residual Risk." It's the hidden danger that remains even after a "fix" is applied.

The Problem: We Assume "Fixed" Means "Safe"

In the world of software (like the apps on your phone or the code running the internet), developers constantly find bugs and patch them. Usually, when a patch is released, security tools assume the code is now "benign" (safe). They stop looking at it.

The authors argue this is a mistake. Developers often make tiny, surgical fixes to stop a specific crash without rewriting the whole program.

The Result: The "safe" code looks 99% identical to the "dangerous" code it replaced.
The Risk: Because they are so similar, other hidden dangers (like a loose screw or a weak wire nearby) might still be there, waiting to cause a new problem later.

The Solution: A "Three-Part Detective" System

To find these hidden dangers, the authors built a new tool called Residual Risk Scoring (RRS). Think of it as a three-squad detective team that investigates a "fixed" piece of code to see if it's really safe.

1. The Semantic Detective (The "Meaning" Reader)

How it works: This uses advanced AI (Code LMs) to read the code and understand what it does, not just what the words are.
The Analogy: Imagine two people writing a story. One story has a typo that makes a character die. The editor fixes the typo. The Semantic Detective reads both stories and says, "These stories are 99% identical. The plot, the characters, and the mood haven't changed."
The Clue: If the "safe" story is too similar to the "dangerous" one, the detective raises a flag.

2. The Structural Detective (The "Blueprint" Inspector)

How it works: This looks at the code's skeleton (called an Abstract Syntax Tree or AST). It checks how the code is built, like looking at the beams and joints of a house.
The Analogy: The Semantic Detective might say, "The house looks the same." But the Structural Detective looks at the blueprints and says, "Wait, they only changed one brick in the wall, but the rest of the wall is still made of rotting wood."
The Clue: This detective ignores the whole house and zooms in on the specific spot that was fixed. If the fix was too small and the surrounding structure is unchanged, it's a red flag.

3. The Consistency Judge (The "Group Think" Validator)

How it works: The authors used five different AI models to do the reading.
The Analogy: Imagine asking five different experts to review the same house. If all five experts agree, "This house is 99% the same as the dangerous one," you can be very sure that the similarity is real and not just a mistake by one expert.
The Clue: If all the AIs agree the code hasn't changed much, the risk score goes up.

Putting It All Together: The "Risk Score"

The system combines these three clues into a single Residual Risk Score (RRS).

High Score: The code looks almost identical to the dangerous version, the structure barely changed, and all the AIs agree. Verdict: "This 'fix' might be a Band-Aid on a bullet wound. We need to inspect this closely."
Low Score: The code looks totally different, or the structure changed significantly. Verdict: "This looks like a real fix. We can probably move on."

What Did They Find?

The authors tested this on thousands of real-world software patches (from the "PrimeVul" dataset). Here is what they discovered:

The "Band-Aid" is Real: About 61% of the patches that looked "safe" to traditional tools actually still had hidden problems.
The Problems are Real: When they ran these "high-risk" patches through professional security scanners, the scanners found real issues like:
- Null Pointer Dereferences: Trying to open a door that doesn't exist (crashes the program).
- Memory Leaks: Leaving the faucet running, slowly draining the system's resources.
- Buffer Overflows: Trying to pour a gallon of water into a cup, causing it to spill everywhere (security breach).
The Old Way Missed It: Traditional tools assumed that because the code was "patched," it was safe. They missed these 61% of cases because they didn't look at how similar the new code was to the old, broken code.

The Takeaway

This paper teaches us that in software, "fixed" doesn't always mean "safe."

Just because a developer put a Band-Aid on a leak doesn't mean the whole house is secure. By using a smart system that compares the "before" and "after" versions of the code (both its meaning and its structure), we can find the patches that are just cosmetic and prioritize the ones that actually need a deeper look.

It's like hiring a home inspector who doesn't just check if the leak is stopped, but checks if the whole foundation is still shaky because the repair was too quick.

1. Problem Statement

The paper addresses a critical gap in software security: the assumption that once a vulnerability is patched, the code is inherently benign. In reality, developers often apply minimal, localized fixes to address specific flaws without refactoring the broader logic. Consequently, "benign" (patched) code often retains significant semantic and structural similarities to its vulnerable predecessor.

This "representational continuity" creates Residual Risk: the possibility that latent vulnerabilities (e.g., null pointer dereferences, unsafe memory allocation) persist despite the patch. Current vulnerability benchmarks and static analysis tools often fail to detect these risks because they treat patched code as safe or rely on binary vulnerability detection that misses subtle, post-patch logical flaws. The authors argue that high similarity between vulnerable and benign code is not just a byproduct of efficient patching but a signal of potential residual risk requiring further inspection.

2. Methodology

The authors propose a unified framework called Residual Risk Scoring (RRS) to quantify the likelihood of residual risk in patched code. The approach integrates three distinct signals:

A. Semantic Similarity (Embedding-Based)

Mechanism: The authors use multiple pre-trained Code Language Models (Code LMs) to generate vector embeddings for both the vulnerable function ( $f_v$ ) and the benign function ( $f_b$ ).
Models Used: A diverse set including encoder-based models (CodeBERT, UniXCoder, GraphCodeBERT) and encoder-decoder models (CodeT5-base, CodeT5-770M).
Metric: Cosine Similarity is used to measure the distance between embeddings. High similarity indicates that the functional behavior encoded by the model has not significantly changed.
Rationale: Cosine similarity was chosen for its robustness and invariance to vector magnitude, providing stable scores (mean ~0.99) across different models.

B. Structural Similarity (AST-Based)

Mechanism: The authors analyze the Abstract Syntax Trees (ASTs) of the code pairs using Tree-sitter.
Metric: Instead of global Tree Edit Distance (TED), which can be inflated by large unchanged regions, they propose Localized TED Similarity (LTS). This focuses on the specific subtrees modified by the patch.
Rationale: Global metrics often obscure fine-grained security-critical changes. Localized analysis captures the minimal structural transformations typical of patches (e.g., changing a function call to a field access) while preserving the overall tree structure.

C. Cross-Model Agreement

Mechanism: To ensure the semantic similarity is not an artifact of a specific model's bias, the framework calculates the variance of similarity scores across the multiple Code LMs used.
Metric: A Cross-Model Agreement Score ( $C_{agree}$ ) is derived from the inverse of the variance. Low variance (high agreement) strengthens the evidence that the code is semantically preserved.

D. The RRS Formula

The final Residual Risk Score (RRS) is a weighted sum of these three components:
$RRS(f_v, f_b) = \alpha \cdot \bar{S}_{sem} + \beta \cdot R_{struct} + \gamma \cdot C_{agree}$
Where:

$\bar{S}_{sem}$ : Mean semantic similarity across models.
$R_{struct}$ : Localized structural similarity.
$C_{agree}$ : Cross-model agreement score.
$\alpha, \beta, \gamma$ : Weighting coefficients (e.g., 0.5, 0.3, 0.2) that balance the contributions.

3. Key Contributions

Formalization of Residual Risk: This is the first work to formalize residual risk as a function of representation-level proximity (semantic and structural) between vulnerable and benign code.
Localized AST Analysis: The paper demonstrates that Localized TED is superior to global tree metrics for patch analysis, as it avoids diluting security-relevant changes with unchanged code regions.
Unified Framework (RRS): The introduction of a multi-signal scoring system that combines embeddings, structural analysis, and model consensus to prioritize post-patch inspection.
Empirical Validation: The framework is validated against state-of-the-art static analysis tools (Cppcheck, Clang-Tidy, Facebook Infer) and manual inspection, proving its utility in real-world scenarios.

4. Experimental Results

The study was conducted on the PrimeVul dataset, analyzing 3,789 vulnerable-benign function pairs (7,578 functions) in C/C++.

High Preservation of Properties (RQ1):
- Semantic: Benign functions showed extremely high cosine similarity to their vulnerable counterparts across all models (Mean $\approx$ 0.98–0.99).
- Structural: Localized structural similarity was also high (Median $\approx$ 0.93), confirming that patches are typically minimal and localized.
- Implication: Patches often preserve the "shape" and "meaning" of the original code, leaving residual patterns intact.
Robust Risk Estimation (RQ2):
- The authors categorized code pairs into four quadrants based on similarity levels. Quadrant I (High Semantic + High Structural similarity) contained ~29.5% of pairs and was identified as the highest risk zone.
- Sensitivity analysis showed that the RRS ranking remained stable across different weighting configurations ( $\alpha, \beta$ ), proving the framework's robustness.
Static Analysis Validation (RQ3):
- When high-RRS functions were analyzed using Cppcheck, Clang-Tidy, and Facebook Infer:
  - 61% of high-RRS functions were flagged for security-relevant issues (e.g., memory safety violations, null pointer dereferences, uninitialized variables).
  - 43% were flagged by two tools, and 27% by all three.
- The study identified 13 distinct categories of residual issues, confirming that high similarity correlates with actual, detectable security flaws.

5. Significance and Impact

Shift in Security Paradigm: The paper challenges the binary view of "vulnerable vs. benign." It posits that "benign" code requires continuous scrutiny if it remains too similar to its vulnerable predecessor.
Prioritization Tool: RRS serves as a practical signal for security teams to prioritize which patches need deep manual review or additional testing, rather than treating all patches as equally safe.
Scalability: By leveraging pre-trained models and automated AST analysis, the approach offers a scalable method for assessing risk in large open-source ecosystems (e.g., Linux, OpenSSL) where manual review of every patch is impossible.
Future Directions: The authors suggest extending this approach to dynamic analysis and diverse programming languages to further refine risk estimation and reduce false positives.

In conclusion, the paper establishes that code-level similarity is a viable and necessary predictor of residual risk, providing a data-driven framework to enhance the reliability of software supply chains.