Automatic Selection of Protections to Mitigate Risks Against Software Applications

Problem Statement
Software applications face significant risks from Man-At-The-End (MATE) attacks, where adversaries possess full access to the software and the systems running it. Attackers utilize tools such as debuggers, disassemblers, and decompilers to reverse engineer algorithms, tamper with logic (e.g., bypassing license checks), or execute unauthorized copies. While defenders can deploy Software Protections (SPs) within the application to mitigate these risks, current practices suffer from a lack of formal risk analysis and reliance on "security-through-obscurity." The selection of SPs is typically manual, performed by experts based on intuition ("feeling"), and assessed only post-deployment. This approach is inefficient, costly, and difficult to scale, particularly given the shortage of security experts and the pressure on vendors to protect multiple application versions and platforms without degrading performance or usability. There is a critical need for a formalized, automated method to select and configure SPs that delays attackers while adhering to strict overhead constraints.

Methodology
The paper proposes a novel, automated approach for selecting software protections, framed as a game-theoretic optimization problem. The methodology is built upon a Knowledge Base (KB) that formalizes the decision-making elements: code artifacts (Application Artifacts or AAs), assets, security requirements (confidentiality and integrity), attack paths, and concrete software protections (CSPs).

1. Formalization and Risk Assessment: The process begins with annotating source code to identify assets and their security requirements. The system constructs a formal representation of the application and identifies potential attack paths as ordered sequences of atomic attack steps. These paths are derived from expert taxonomies and represent the sequence of tasks an attacker must perform to breach a security requirement.
2. The Software Protection Index (SPI): A central contribution is the introduction of the SPI, which extends existing notions of "potency" and "resilience." The SPI quantifies the effectiveness of a protection solution by evaluating its impact on specific attack paths. It incorporates software metrics (e.g., Halstead size, cyclomatic complexity) and expert assessments to model how protections increase the effort required for attack steps (potency) and how attack efforts erode protection effectiveness (resilience).
3. Game-Theoretic Optimization: The selection of optimal protections is modeled as a non-interactive game between a defender and an attacker.
   • The Defender: Proposes a candidate solution (an ordered sequence of CSPs applied to specific artifacts) to maximize the SPI.
   • The Attacker: Invests effort in specific attack paths to minimize the SPI (residual SPI).
   • The Algorithm: The system employs a heuristic based on a mini-max depth-first exploration strategy. To manage the exponential complexity of the search space, the algorithm utilizes dynamic programming optimizations, including alpha-beta pruning, aspiration windows, transposition tables, and futility pruning.
   • Code Correlation Sets (CCSs): To further improve efficiency, the application is partitioned into independent CCSs, allowing the optimization to be solved for smaller, manageable subsets of the code rather than the entire application at once.
4. Overhead Constraints: The selection process is constrained by overhead thresholds (computation time, memory, network bandwidth). The system uses Machine Learning (ML) models to predict the metrics and overheads of candidate solutions without requiring the actual compilation of every possible variant, ensuring the selected solution remains usable.

Key Contributions

• A Formal Model for SP Selection: The paper presents a formal model that integrates code artifacts, assets, security requirements, and attack paths to automate the selection of optimal SPs under overhead constraints.
• The Software Protection Index (SPI): An original metric that evaluates protection effectiveness against specific attack paths, combining software metrics with expert assessments to quantify both potency and resilience.
• Automated Optimization Algorithm: A game-theoretic approach using mini-max depth-first search with dynamic programming optimizations to find near-optimal protection solutions in a feasible timeframe.
• Proof-of-Concept (PoC) Implementation: A working tool (ESP) implemented as Eclipse plugins that supports C source code, integrating tools like Tigress and the ASPIRE Compiler Tool Chain (ACTC).

Results and Validation
The approach was validated through a quantitative and qualitative evaluation involving industry experts from the ASPIRE project consortium.

• Quantitative Analysis: The PoC was tested on three Android application use cases (a license manager, an OTP generator, and a DRM-enabled video player). The optimization time was measured against the number of protection objectives (POs), attack paths, and search depth. While the theoretical complexity is exponential, the use of CCSs and heuristics allowed the system to find optimal solutions in minutes for realistic scenarios (e.g., search depths of 3–6 and up to 512 POs).
• Qualitative Evaluation: Experts (both internal and external to the project) evaluated the PoC's outputs. They confirmed that the tool could generate customized, high-quality protection solutions comparable to manual expert selection. The experts noted that the selected SPs preserved the application's original semantics and introduced acceptable computational overhead. The tool was deemed particularly useful for supporting experts in their daily tasks, though it was noted that without expert fine-tuning of parameters, the security level might not match that of a fully manual process.
• Real-World Testing: The solutions generated by the PoC were subjected to attacks by professional penetration testers, who reported significant delays in attacking the protected applications, validating the practical effectiveness of the automated selection.

Significance
The paper claims that automated software protection is a practical and effective solution for risk mitigation in software. By formalizing the risk analysis and automating the selection process, the approach addresses the skill gap in the industry and the time-to-market pressures faced by software vendors. It moves beyond "security-through-obscurity" by providing a repeatable, reproducible, and formally grounded method to delay attackers while ensuring application usability. The work demonstrates that a game-theoretic approach, augmented with machine learning for metric prediction and dynamic programming for search optimization, can effectively navigate the complex trade-offs between security and performance in the context of MATE attacks.