💻 computer science

CauSec: Unboxing the Causal Drivers of Static Vulnerability Analysis Performance

This paper introduces CAUSEC, a causal analysis framework that formalizes and validates the underlying assumptions of Static Application Security Testing (SAST) tools to determine whether their design trade-offs genuinely yield the intended performance gains, demonstrating its utility through a systematic study of 57 crypto-API misuse assumptions across four popular tools.

Original authors: Md Akram Khan (William & Mary), Daniel Rodriguez-Cardenas (William & Mary), Alejandro Velasco Dimate (William & Mary), Denys Poshyvanyk (William & Mary), Adwait Nadkarni (William & Mary)

Published 2026-08-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Md Akram Khan (William & Mary), Daniel Rodriguez-Cardenas (William & Mary), Alejandro Velasco Dimate (William & Mary), Denys Poshyvanyk (William & Mary), Adwait Nadkarni (William & Mary)

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the digital world, software tools act as the gatekeepers of security, scanning code to find hidden flaws before they can be exploited. These tools, known as static application security testers, are used everywhere from small startups to massive corporations. They work by making educated guesses about how code behaves, often choosing to ignore certain parts of a program to run faster or to avoid flagging harmless code as dangerous. The industry has long operated on a set of unproven beliefs: that skipping specific types of code will make the tool more accurate, or that ignoring certain libraries will reduce false alarms. These beliefs guide how the tools are built, but until now, no one had systematically tested whether these assumptions were actually true or if they were just convenient stories the designers told themselves.

A team of researchers at William & Mary decided to put these beliefs to the test. They built a new framework called CAUSEC, which treats the assumptions behind security tools not as facts, but as hypotheses that can be proven or disproven. Instead of just looking at whether a tool finds bugs, they asked why it finds them and what happens when you change the rules. By applying a method called causal inference, which helps separate true cause-and-effect relationships from simple coincidences, they were able to isolate specific design choices and measure their real impact. Their work reveals that the rules security tools follow are often more fragile and specific than anyone realized, and that a strategy that works perfectly for one tool can fail completely for another.

The researchers began by looking at the history of tools designed to catch mistakes in how software uses cryptography, the math that keeps data safe. They sifted through twenty years of research papers and found fifty-seven distinct assumptions that tool designers had made. These assumptions ranged from the idea that focusing on specific code rules improves accuracy, to the belief that ignoring third-party code libraries makes the tool faster and more precise. The team realized that many of these claims were based on correlations—things that happened to occur together—rather than proven causes. For instance, a tool might skip third-party libraries and happen to have fewer false alarms, but that doesn't mean the skipping was the actual reason for the improvement. There could be other hidden factors at play.

To untangle this, the team focused on one very common assumption: that reporting security alerts from third-party libraries decreases the tool's precision, meaning it creates more false alarms. They gathered a massive dataset of over 57,000 alerts generated by four popular security tools: Semgrep, CodeQL, CogniCrypt, and CryptoGuard. They then manually checked every single alert to see if it was a real problem or a false alarm, creating a ground truth that the tools could be measured against. Using their new framework, they simulated what would happen if they forced each tool to report alerts from third-party libraries, while carefully accounting for other variables like the size of the app or how popular it was.

The results were surprising and showed that the assumption was not a universal truth. For two of the tools, the assumption held up: when they started reporting alerts from third-party libraries, the rate of false alarms did indeed go up. However, for the other two tools, the exact opposite happened. When those tools included alerts from third-party libraries, their accuracy actually improved. This discovery proved that the design of the tool itself acts as a modifier; the same rule can have completely different effects depending on the tool's internal mechanics. The researchers found that the impact of including third-party code varied wildly depending on the specific type of library and the specific tool being used. For one tool, utility libraries caused a significant drop in accuracy, while for another, those same libraries improved accuracy.

The study also highlighted that many of the assumptions made by tool designers were based on unverified trade-offs. Designers often sacrifice the ability to find every possible bug in exchange for speed or fewer false alarms, believing that this trade-off is necessary. The researchers found that while these trade-offs are real, the specific outcomes are often unpredictable. They discovered that the way a tool is built—its specific rules, how it filters data, and how it handles context—determines whether a design choice helps or hurts. This means that a security team cannot simply copy the design choices of a successful tool and expect the same results. What works for one tool is not guaranteed to work for another.

Ultimately, the paper argues that the security community needs to move away from accepting assumptions as facts. The researchers demonstrated that by using causal analysis, it is possible to test these assumptions rigorously and understand exactly how they affect performance. They found that while some assumptions are valid, many are not, and their validity depends entirely on the specific context of the tool being used. The study concludes that tool designers should not inherit assumptions from previous work without testing them in their own specific environment. Instead, they should treat every design choice as a hypothesis to be tested, ensuring that the tools they build are based on evidence rather than intuition. This approach offers a clearer path to building security tools that are not just fast, but truly effective at protecting software.

Technical Summary: CauSec – Unboxing the Causal Drivers of Static Vulnerability Analysis Performance

1. Problem Statement

Static Application Security Testing (SAST) tools are widely deployed in industry and academia to detect software vulnerabilities. However, these tools often make design choices that sacrifice detection capabilities (e.g., recall) to achieve performance goals such as higher precision, reduced runtime, or improved scalability. These choices are frequently motivated by implicit or explicit assumptions regarding the target code or the analysis technique (e.g., "analyzing third-party libraries increases false positives").

The core problem addressed by this paper is the lack of rigorous validation for these underlying assumptions. While tools may claim that specific design choices (like excluding third-party libraries) improve performance, it remains unclear whether these assumptions are valid or if the resulting "gains" are merely artifacts of confounding variables. Without causal validation, tools may inadvertently introduce significant detection gaps, leading to failures in critical use cases like compliance certification. The paper asks: Do the sacrifices in detection capabilities actually help tools achieve the expected performance gains, and are the underlying assumptions valid?

2. Methodology: The CAUSEC Framework

To address this, the authors propose CAUSEC, a causal analysis framework that transforms informal SAST assumptions into testable causal hypotheses. The framework moves beyond simple correlation to establish causal relationships using Pearl's Structural Causal Models (SCMs) and do-calculus.

2.1. Security Assumption Abstraction

CAUSEC formalizes informal assumptions into a Security Assumption tuple: $A_i = (P_i, C_i, O_i)$ .

Precondition ( $P_i$ ): The scope or context (e.g., dataset constraints, platform).
Condition ( $C_i$ ): The specific claim, heuristic, or constraint (e.g., "reporting alerts from third-party libraries").
Outcome ( $O_i$ ): The expected consequence (e.g., "decreased precision").

This abstraction allows assumptions to be operationalized as causal treatments (e.g., data-level treatments like including/excluding alerts, or method-level treatments like changing analysis depth).

2.2. Causal Analysis Workflow

Data Collection & Treatment Construction: Based on the condition and outcome, CAUSEC constructs treatments. For example, to test if reporting third-party alerts affects precision, the treatment is applied post-hoc to the alert dataset (data-level treatment).
Causal Modeling (DAG): The framework models the problem as a Directed Acyclic Graph (DAG) identifying the treatment ( $T$ ), outcome ( $Y$ ), and confounders ( $Z$ ). Confounders (e.g., app popularity, APK size) are variables that influence both the treatment assignment and the outcome, potentially creating spurious associations.
Estimand Identification: CAUSEC identifies the causal estimand (e.g., Average Treatment Effect - ATE) using the backdoor criterion to block non-causal paths, ensuring the effect is identifiable from observational data.
Effect Estimation: The framework estimates the causal effect using statistical methods such as Propensity Score Matching (PSM) or regression adjustment, controlling for measured confounders.
Robustness Validation: To ensure reliability, CAUSEC employs refutation tests:
- Random Common Cause: Adding an independent random variable to check if the estimate changes.
- Placebo Treatment: Replacing the treatment with a random one to see if the effect vanishes.
- Data Subset Refuter: Re-estimating on random subsets to check stability.
- Dummy Outcome Refuter: Replacing the outcome with a synthetic one to verify the estimator produces near-zero effects.

2.3. Systematic Literature Review (SLR)

To understand the landscape of assumptions, the authors conducted an SLR of 20 papers from the last 20 years focusing on crypto-API misuse detectors. This process extracted and qualitatively analyzed 57 unique security assumptions regarding rules, pruning, scope, and performance trade-offs.

3. Key Contributions

Characterization of Security Assumptions: The identification and qualitative analysis of 57 assumptions in crypto detectors, revealing patterns such as strong claims about rule coverage driving accuracy, the trade-off between precision and scalability, and assumptions about developer preferences.
The CAUSEC Framework: A novel framework that contextualizes causal inference for SAST evaluation. It provides a systematic method to formalize, operationalize, and evaluate assumptions while explicitly accounting for confounding factors.
Empirical Evaluation: A large-scale evaluation of a common assumption ("reporting alerts from third-party libraries decreases precision") using four popular tools (Semgrep, CodeQL, CogniCrypt, CryptoGuard) and a manually labeled ground truth dataset of 57,038 alerts from 558 Android applications.

4. Key Results and Findings

The evaluation of the "third-party library" assumption yielded several critical insights:

Tool Design as an Effect Modifier (F8): The causal effect of reporting third-party alerts is not uniform.
- CogniCrypt: Reporting third-party alerts significantly decreases precision (ATE: -0.283).
- CodeQL: Reporting third-party alerts decreases precision (ATE: -0.057).
- CryptoGuard & Semgrep: Reporting third-party alerts actually increases precision (ATE: +0.043 and +0.109, respectively).
- Conclusion: The same assumption holds for some tools but is invalid for others, proving that tool design (rules, filtering, implementation) acts as an effect modifier.
Heterogeneity Across Library Categories (F9): The impact of third-party alerts varies by library type (e.g., Utilities, Android, Small libraries). For instance, Utilities significantly reduced precision for CogniCrypt but increased it for CryptoGuard.
Robustness (F10): The adjusted causal estimates were largely stable across refutation tests, supporting the reliability of the findings. However, the estimate for CryptoGuard was sensitive to placebo refuters, suggesting caution in interpreting its positive effect.
Discrepancy with Raw Observations (F7): For CogniCrypt, the adjusted causal effect (-0.283) was substantially larger than the raw unadjusted observation (-0.216), demonstrating that unadjusted correlation can underestimate the true impact of design choices.

5. Significance and Takeaways

The paper argues that SAST design often relies on unverified assumptions that may not hold across different tools or contexts. The significance of this work lies in shifting the paradigm from ad-hoc validation to principled causal analysis.

The authors distill three main takeaways for the security community:

Operationalization Limits: While diverse assumptions exist, their feasibility for causal evaluation depends on whether they can be operationalized (e.g., data-level vs. method-level treatments) and measured in controlled settings.
Causal Inference as a Validation Tool: Causal inference offers a robust alternative to ad-hoc validation, capable of revealing effects that differ significantly from unadjusted observations by controlling for confounders.
No Inheritance of Validity: A critical lesson is that assumptions cannot be inherited. Because tool design acts as an effect modifier, a design choice or assumption valid for one SAST cannot be assumed valid for another. Designers must test assumptions within the specific context of their own tool rather than relying on prior work.

The paper concludes that CAUSEC enables researchers to systematically test the validity of SAST assumptions, leading to more reliable and transparent security tools.