💻 computer science

SA-DRL: Security-Aware Deep Reinforcement Learning for Ransomware Detection with Asymmetric Reward Design

This paper proposes a Security-Aware Deep Reinforcement Learning (SA-DRL) framework that utilizes asymmetric reward shaping and a security-optimal model selection criterion to significantly reduce ransomware false-negative rates compared to conventional symmetric detection methods.

Original authors: Jannatul Ferdous, Rafiqul Islam, Md Zahidul Islam

Published 2026-07-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jannatul Ferdous, Rafiqul Islam, Md Zahidul Islam

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the head of security for a massive digital city. Your job is to spot a specific type of criminal: Ransomware. These are digital bandits that lock up your files and demand money to unlock them.

The problem is, these bandits are fast. They can encrypt your data in seconds. If you miss them (a False Negative), the city gets locked down, and the damage is permanent and expensive. If you accidentally flag a harmless citizen as a bandit (a False Positive), you just put them in a holding cell for a quick check. It's annoying, but it's fixable.

For a long time, computer security systems treated these two mistakes as equally bad. It was like saying, "It's just as bad to let a bank robber walk free as it is to arrest a baker by mistake." The paper argues this is a terrible way to run a security system.

Here is how the authors, Jannatul Ferdous and her team, fixed this using a new approach called SA-DRL (Security-Aware Deep Reinforcement Learning).

1. The Old Way: The "Equal Penalty" Game

Think of traditional security software like a student taking a test. If they get a question wrong, they lose one point. It doesn't matter which question they got wrong.

The Flaw: In the real world, missing a ransomware attack is like failing the final exam and losing your degree. Catching a harmless file is just a small deduction on a homework assignment. The old systems didn't know the difference, so they often let the "big bads" slip through to keep the "small mistakes" low.

2. The New Way: The "Security-First" Coach

The authors created a new training method using Deep Reinforcement Learning (DRL). Imagine a video game where an AI agent (the security guard) learns by playing millions of rounds.

The Reward System: In this game, the "Coach" (the reward function) gives points for catching bad guys and takes points away for mistakes.
- Old Coach (Symmetric): "You lose 1 point for catching a baker. You lose 1 point for letting a bandit go."
- New Coach (Asymmetric - SA-DRL): "You lose 1 point for catching a baker. But if you let a bandit go, you lose 4 points!"

By making the penalty for missing a ransomware attack four times heavier than the penalty for a false alarm, the AI learns a very important lesson: "It is better to be safe than sorry." It starts being slightly more paranoid, catching almost every single bandit, even if it means checking a few more innocent bakers.

3. The Training Ground: The "Shuffled Deck"

To make sure the AI didn't just memorize the order of the files, the researchers used a clever trick. Imagine dealing cards from a deck.

The Trick: Every time the AI starts a new training session, they shuffle the deck of files completely.
The Result: The AI can't just learn "File #1 is good, File #2 is bad." It has to learn the behavior of the files. Because the "bad" files (ransomware) keep getting hit with that heavy 4-point penalty whenever they are missed, the AI learns to pay extra attention to the tricky ones. This acts like an automatic "highlighter" for the most dangerous files without needing to manually mark them.

4. The Race: Who Wins?

The researchers tested four different types of AI "players" (algorithms) to see who could learn this security-first lesson best:

DQN
DDQN (Double DQN)
PPO
A2C

They also tested different "discount factors." Think of this as how much the AI cares about the future versus the now.

High Discount: "I'm thinking about the whole game."
Low Discount: "I need to make the right move right now."

The Winner: The DDQN player, using the Low Discount (focusing on immediate action) and the Asymmetric Reward (the heavy penalty for missing bandits), was the champion.

5. The Results: A Massive Improvement

When they put the winning AI to the test against the old methods:

The Old Best: Missed about 2.47% of ransomware attacks.
The New SA-DRL: Missed only 0.80% of attacks.

This is a 67.6% reduction in missed ransomware. The AI became much better at spotting the real threats. It did this while keeping the number of false alarms (catching innocent bakers) very low and actually training faster than some other complex models.

6. The "Security-Optimal" Rule

Finally, the authors introduced a new rule for picking the best model, called SOMS.

Old Rule: "Pick the model with the highest overall score."
New SOMS Rule: "First, pick the model that misses the fewest bandits. Then, look at the score. Then, look at how fast it is."

This ensures that the chosen model is built for safety first, not just for looking good on a spreadsheet.

Summary

The paper shows that by changing how we "pay" our AI for its mistakes—making the penalty for missing a ransomware attack much heavier than the penalty for a false alarm—we can build security systems that are far better at protecting our data. The DDQN model with this new "Security-Aware" mindset is the most effective tool they found, catching significantly more ransomware than previous methods while staying fast and efficient.

Technical Summary: SA-DRL for Ransomware Detection

Problem Statement
Ransomware poses a critical threat to modern infrastructure, with recovery costs reaching $1.53 million globally in 2025 and attacks occurring every two seconds. While machine learning (ML) and deep learning (DL) detectors have achieved high accuracy (often >98%), they typically rely on symmetric loss functions (e.g., binary cross-entropy) that treat false negatives (FN) and false positives (FP) as equally costly. In operational reality, this assumption is flawed: an FN allows irreversible data encryption and system compromise, whereas an FP results in a reversible quarantine or analyst review. Furthermore, existing Deep Reinforcement Learning (DRL) applications in ransomware detection have largely utilized static features (e.g., PE headers) or symmetric reward structures, failing to embed the operational cost asymmetry directly into the learning objective.

Methodology
The authors propose SA-DRL (Security-Aware Deep Reinforcement Learning), a framework that models ransomware detection as a finite-horizon Markov Decision Process (MDP) using behavioral runtime telemetry.

Dataset and Environment:
- A balanced dataset of 2,000 Windows executable samples (1,000 ransomware across 30 families, 1,000 benign) was constructed.
- Samples were executed in the ANY.RUN sandbox (Windows 11) to extract 103 behavioral features (API calls, registry modifications, file operations, network activity, CryptoAPI usage).
- The environment employs episode-level random permutation, shuffling sample order at the start of each episode to eliminate fixed-order bias and induce an implicit adaptive sample-weighting effect.
Reward Design:
- Baseline ( $R_1$ ): A symmetric reward (+1 for correct, -1 for incorrect).
- Security-Aware ( $R_2$ ): An asymmetric reward function designed to reflect operational costs. Correct predictions receive +1. False Positives receive -0.5, while False Negatives receive a heavier penalty of -2.0 (a 4:1 cost ratio). This structure explicitly guides the agent to prioritize minimizing missed detections.
Agents and Training:
- Four DRL agents were evaluated: DQN, DDQN, PPO, and A2C.
- Experiments covered four discount factors ( $\gamma \in \{0.1, 0.5, 0.9, 0.99\}$ ), two reward functions, five-fold stratified cross-validation, and three random seeds, resulting in 480 controlled training runs.
- Baseline supervised models (MLP, CNN1D, Logistic Regression) were trained with equivalent class-weighting for comparison.
Model Selection (SOMS):
- The study introduces the Security-Optimal Model Selection (SOMS) criterion. Unlike standard selection based on aggregate accuracy, SOMS uses a lexicographic optimization:
  1. Minimize False Negative Rate (FNR).
  2. Maximize F1-Score.
  3. Minimize Training Time.

Key Contributions

Asymmetric Reward Integration: The first framework to embed FN–FP cost asymmetry directly into the DRL reward signal for behavioral ransomware detection, moving beyond post-hoc threshold adjustments.
Comprehensive Evaluation: A controlled factorial comparison of four DRL algorithms across multiple reward structures and hyperparameters (480 runs), providing robust statistical validation.
SOMS Criterion: A formal, security-first model selection protocol that prioritizes minimizing missed attacks over aggregate accuracy.
Implicit Adaptive Weighting: Utilization of episode-level random permutation to naturally emphasize difficult samples through asymmetric reward-driven temporal-difference updates, without explicit per-sample weight computation.
Statistical Validation: Use of Friedman and Wilcoxon signed-rank tests to confirm that performance improvements are statistically significant and not due to stochastic variance.

Experimental Results

Best Configuration: The DDQN agent trained with the asymmetric reward ( $R_2$ ) and a low discount factor ( $\gamma = 0.1$ ) was selected as optimal by SOMS.
Performance Metrics: This configuration achieved an FNR of 0.0080 (0.80%), an F1-score of 0.9915, and an AUC of 0.998.
Comparative Improvement: Compared to the best supervised baseline (MLP with FNR = 0.0247), the SA-DRL configuration reduced missed detections by 67.6%.
Reward Impact: Across all configurations, the asymmetric reward ( $R_2$ ) reduced the mean FNR by 43% relative to the symmetric baseline ( $R_1$ ), with a negligible increase in False Positive Rate (0.20 percentage points).
Algorithmic Sensitivity: Value-based methods (DDQN, DQN) demonstrated superior stability and lower FNR compared to policy-gradient methods (PPO, A2C), particularly under higher discount factors. Lower discount factors ( $\gamma=0.1, 0.5$ ) were found to be more effective for this security-critical classification task.

Significance and Claims
The paper claims that SA-DRL represents a paradigm shift in ransomware detection by aligning the learning objective directly with operational security priorities. The authors assert that:

Reward function design is a critical mechanism for security-sensitive decision-making, not merely a tuning parameter.
The proposed framework is the first to combine runtime behavioral features, asymmetric reward design, and a formal security-first selection criterion (SOMS) with rigorous statistical validation.
The results demonstrate that cost-aware reinforcement learning can effectively minimize the most damaging error type (missed ransomware) while maintaining high overall detection performance, offering a more robust solution for high-risk environments than traditional symmetric loss approaches.