Reliable Remediation Impact Prediction for Black-Box Security Ratings

Technical Summary: Reliable Remediation Impact Prediction for Black-Box Security Ratings

Problem Statement
Commercial security rating platforms (e.g., BitSight, SecurityScorecard) aggregate externally observable cyber evidence into numerical scores to help organizations prioritize remediation. While these platforms provide current scores, organizations require "what-if" analysis to understand how specific remediation actions (e.g., deploying SPF or DMARC) would impact their scores. However, revealing exact score responses for hypothetical configurations risks leaking the proprietary scoring logic of the platform, which is designed to remain a black box.

Furthermore, predicting these impacts using a machine learning surrogate model introduces a reliability challenge. Prediction quality is not uniform across all organization configurations; it depends heavily on the amount and structure of observable evidence (specifically, the set of applicable security checkpoints). Configurations with fewer applicable checkpoints often yield less reliable predictions. The core problem is to predict remediation score impacts without exposing the scoring engine while simultaneously identifying and filtering out cases where the surrogate's predictions are unstable.

Methodology
The authors propose a four-phase surrogate-based approach designed to respect the opacity of the scoring engine while ensuring predictive reliability.

1. Surrogate Construction with Applicability Awareness:
   The authors construct a surrogate model (a LightGBM regressor) trained on historical organization configurations to approximate the platform's scoring engine. Crucially, the feature representation ($x_i$) is enriched with two components:

   • Outcome features ($o_i$): Observed results of applicable checkpoints.
   • Applicability features ($s_i$): Explicit indicators of which checkpoints apply to a specific configuration and structural summaries of the checkpoint set (e.g., count of applicable checkpoints, distribution of applicability).
     This design allows the model to distinguish between missing data due to a lack of evidence versus missing data due to a checkpoint being structurally inapplicable.

2. Sensitivity Analysis (Diagnostic):
   To quantify prediction fragility, the authors perform a controlled restriction experiment. They systematically mask portions of the applicable checkpoint information in the feature representation and measure the resulting score drift ($\Delta$). Configurations exhibiting a score change exceeding a threshold $\tau$ under restricted information are labeled as "unstable." This phase establishes a diagnostic link between the size/structure of the checkpoint set and prediction stability.

3. Reliability Assessment (Selective Prediction):
   A reliability layer (a LightGBM classifier) is trained to predict the probability of instability ($p_{unstable}$) based solely on the applicability features ($s_i$). Using Conformal Risk Control (CRC), the system defines an acceptance rule. For a target risk level $\alpha$, the system selects a threshold $\lambda^*$ such that the instability risk among accepted predictions is guaranteed to be below $\alpha$ with high confidence. This allows the system to selectively accept only those remediation impact predictions that are deemed dependable.

4. Remediation Score Impact Prediction:
   For a user-simulated candidate remediation action, the system constructs a post-action configuration, updates the applicability features to reflect the new technical state, and predicts the new score. The predicted impact is the difference between the new and current scores. The reliability layer then filters these predictions: if the post-action configuration's instability probability exceeds $\lambda^*$, the prediction is flagged as unreliable and should be interpreted with caution.

Key Contributions

• Applicability-Aware Surrogate Modeling: The paper demonstrates that explicitly modeling checkpoint applicability and measurement structure significantly improves score prediction accuracy compared to models using only outcome features or only applicability features.
• Reliability Layer with Formal Guarantees: The authors introduce a mechanism to identify configurations prone to unstable predictions using checkpoint applicability features. By integrating Conformal Risk Control, they provide finite-sample risk guarantees for accepted predictions, moving beyond average accuracy metrics.
• Remediation Impact Validation: The approach is evaluated on real-world data to predict the impact of specific remediation actions (SPF and DMARC deployment). The study validates that the reliability layer successfully identifies cases where predicted gains diverge significantly from trusted reference scores.

Experimental Results
The approach was evaluated on a dataset of 5,188 distinct organization configurations from a commercial security rating platform's Mail analysis domain.

• Prediction Accuracy: The full applicability-aware surrogate achieved a test Mean Absolute Error (MAE) of 0.78, significantly outperforming an outcome-only baseline (MAE 1.46) and an applicability-only baseline (MAE 4.64).
• Sensitivity: Sensitivity analysis revealed a sharp transition in prediction stability. When the number of retained applicable checkpoints dropped from 22 to 18, the fraction of unstable predictions (score drift > 2) jumped from 2.27% to 69.38%.
• Reliability: The reliability model achieved a ROC AUC of 0.9255 (at $\tau=2$) in predicting instability. Using CRC, the system could accept a substantial portion of predictions (e.g., up to 89.4% coverage at $\alpha=0.30$) while maintaining the instability risk below the target threshold.
• Remediation: For DMARC deployment, the surrogate predicted positive score gains. The reliability layer effectively separated dependable predictions (mean gain 11.26, MAE 6.94) from unstable ones (mean gain -1.58, MAE 9.35), showing that large prediction errors correlate strongly with high predicted instability.

Significance and Claims
The paper claims that surrogate-based score feedback in opaque security rating environments cannot rely on average predictive accuracy alone. The authors argue that checkpoint applicability is a critical signal that serves a dual purpose: it improves the accuracy of the score prediction model and provides the necessary feature basis for a reliability layer to detect unstable cases.

By combining applicability-aware modeling with Conformal Risk Control, the proposed approach enables organizations to receive actionable remediation impact feedback without compromising the proprietary nature of the scoring engine. The system does not automate remediation recommendations but acts as a decision support tool that quantifies the confidence of its own predictions, warning users when a predicted score impact should be interpreted cautiously due to insufficient or structurally limited evidence.