💻 computer science

CALIBURN: A Regime-Sensitivity Study of Operationally Calibrated Streaming Intrusion Detection

This paper introduces CALIBURN, an operationally calibrated streaming intrusion detection pipeline that enables pre-deployment alerting configuration via cost and budget constraints, demonstrating superior performance in rare-attack regimes while providing critical insights into the sensitivity of streaming methods across varying attack prevalence levels.

Original authors: Michel A. Youssef

Published 2026-05-26

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Michel A. Youssef

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the security guard for a massive, busy airport. Your job is to spot intruders (hackers) among millions of normal travelers (network traffic).

The problem with current security systems is that they are like guards who shout "INTRUDER!" at the slightest hint of suspicious behavior, or worse, they wait until the shift is over to decide who they should have shouted at. This leads to two bad outcomes: either the guard is so tired from false alarms they ignore a real threat, or they miss the threat entirely because they didn't know what to look for until it was too late.

CALIBURN is a new system designed to fix this. It's not just a detector; it's a "smart decision-maker" that learns to balance the cost of missing a bad guy against the cost of bothering innocent people.

Here is how CALIBURN works, broken down into five simple steps using everyday analogies:

1. The "Run-Length" Watcher (The Detective)

Most detectors just look at a single person and say, "You look weird." CALIBURN is different. It watches the flow of people and asks, "Has the pattern of the crowd changed?"

It uses a method called Bayesian Online Change-Point Detection. Think of it like a detective tracking how long a group of people has been walking together in a straight line.

If the group keeps walking straight, the detective thinks, "Everything is normal."
If the group suddenly stops, turns around, or starts running, the detective thinks, "Something just changed!"
The Trick: To keep the detective from getting overwhelmed by history, CALIBURN only remembers the last few hundred steps (truncation). This keeps the system fast and memory-efficient, like a detective who only cares about what happened right now, not what happened last year.

2. The "Translator" (The Calibration Layer)

The detective's "change" signal isn't a direct "This is a hacker" score. It's more like "The pattern shifted."

The Problem: A pattern shift could be a hacker, but it could also be a scheduled bus arrival or a fire drill.
The Solution: CALIBURN uses a Translator (Isotonic Calibration). It takes the detective's raw "shift" signal and translates it into a clear probability: "Given this shift, there is a 90% chance this is an attack."
The Result: This step makes the system much more accurate. In the paper's tests, this translation reduced the "guessing error" by about 30%.

3. The "Price Tag" Decision (Cost-Sensitive Threshold)

Now the system has a probability. But when do you actually sound the alarm?

Old Way: Pick a random number (e.g., "If probability > 50%, shout!").
CALIBURN Way: Ask the boss: "How much does it cost us to miss a hacker vs. how much does it cost to wake up the team for a false alarm?"
- If missing a hacker is 10 times worse than a false alarm, the system sets the alarm to go off even at a low 9% probability.
- If false alarms are very expensive (causing "alert fatigue"), it waits for a 90% certainty.
This ensures the alarm is set based on business reality, not just math.

4. The "Budget" Manager (Conformal Risk Control)

Even with a perfect price tag, you might accidentally set the alarm too sensitive and blow your budget.

The Concept: Imagine you have a "False Alarm Budget" of 100 alerts per day. You don't want to spend it all in the first hour.
The Solution: CALIBURN uses a Budget Manager (Conformal Risk Control). It looks at the data and mathematically guarantees: "If we set the alarm at this specific level, we will not exceed your budget of 100 false alarms."
It acts like a strict accountant ensuring the system doesn't overspend on false alarms, even if the data is tricky.

5. The "Burn Rate" Alarm (The SRE Layer)

Finally, the system decides how to deliver the news.

The Problem: One weird event might be a glitch. A hundred weird events in a row is a crisis.
The Solution: CALIBURN uses Burn-Rate Alerting (borrowed from Site Reliability Engineering).
- Short Window: If 5 weird things happen in 5 minutes, it checks the Long Window.
- Long Window: If those 5 things are part of a pattern that has been happening for an hour, then it escalates.
- Result: It ignores one-off glitches (noise) but screams immediately if a sustained attack is happening. It can send a "Ticket" for slow issues, a "Slow Page" for medium issues, or a "Fast Page" for emergencies.

What the Paper Actually Found (The "Regime Sensitivity")

The authors didn't just say "CALIBURN is the best." They tested it in three different "weather conditions" to see where it works and where it fails:

Rare Attack Weather (LITNET-2020):
- Scenario: 95% of traffic is normal; 5% is attacks.
- Result: CALIBURN wins big. It was 2.2 times better than the next best streaming system and 4 times better than the best "batch" (offline) system. It caught the rare intruders without crying wolf.
Moderate Attack Weather (CICIDS2017):
- Scenario: About 22% of traffic is attacks.
- Result: CALIBURN is still the best streaming system, but a traditional "offline" system (which looks at all the data at once) beats it. This makes sense: when attacks are common, the "pattern watcher" gets confused because the "normal" pattern is now polluted with attacks.
High Attack Weather (UNSW-NB15):
- Scenario: 64% of traffic is attacks.
- Result: Everything breaks. CALIBURN and all other streaming systems failed.
- Why? The system assumes the stream is mostly normal. When the stream is mostly attacks, the system thinks the attacks are the new "normal" and stops flagging them. The paper proves this isn't a bug in the code or a trick in the data; it's a fundamental limit of streaming detection when the "bad guys" are the majority.

The Bottom Line

CALIBURN is a tool for Security Operations Centers that need to handle rare attacks in a sea of normal traffic.

It is great when: Attacks are rare, you need to set a budget for false alarms, and you can't wait for a human to label data before you start protecting the network.
It is not for: Situations where attacks are the majority of the traffic, or where you have unlimited time to retrain models offline.

The paper concludes that CALIBURN is a practical, explainable system that lets operators define their own "costs" and "budgets" rather than guessing at magic numbers. It works best when the world is mostly calm, and the intruders are the exception.

Technical Summary: CALIBURN: A Regime-Sensitivity Study of Operationally Calibrated Streaming Intrusion Detection

1. Problem Statement

Network Intrusion Detection Systems (NIDS) in production face a chronic gap between statistical anomaly scores and operational alerting requirements. Current academic methods typically treat threshold selection as a post-hoc tuning problem, optimizing for metrics like F1 or AUC-ROC on validation labels that are often unavailable, delayed, or scarce in real Security Operations Centers (SOCs).

Operators require alerting behaviors defined before deployment using operational inputs: the relative cost of false negatives versus false positives, and a finite alerting budget (e.g., derived from Service Level Objectives). Furthermore, streaming detectors must process flows continuously with bounded memory, yet most existing streaming anomaly detection methods leave the translation of scores into actionable alerts as a separate, heuristic problem. This paper addresses the need for a streaming pipeline that is operationally calibrated, deriving thresholds directly from operator-specified costs and budgets rather than label-dependent search.

2. Methodology: The CALIBURN Pipeline

CALIBURN is a five-component streaming alerting pipeline designed to separate statistical scoring, operational decision-making, and alert escalation. It is framed not as a universal solution, but as a regime-sensitivity study evaluating performance across varying attack prevalences.

Core Components

Truncated Bayesian Online Change-Point Detection (BOCPD):
- Function: Processes network flows sequentially to produce a probabilistic anomaly score $s_t = P(r_t = 0 | x_{1:t})$ , representing the posterior probability that the current flow initiates a new regime (run-length reset).
- Constraint: Uses a truncated run-length approximation (max length $L$ ) to ensure bounded update cost and constant memory, making it suitable for continuous streaming.
- Model: Employs a Gaussian observation model with online sufficient statistics updates.
Isotonic Calibration Layer:
- Function: Maps the raw BOCPD score $s_t$ (a regime-shift probability) to an empirical conditional attack probability $\hat{p}_t = \hat{P}(y_t = 1 | s_t)$ .
- Method: Fits a monotone calibration map using isotonic regression (Pool-Adjacent-Violators algorithm) on a validation split.
- Rationale: The raw BOCPD score is not a direct attack probability; calibration bridges the gap between "regime shift" and "attack" using available labels.
Cost-Sensitive Decision Threshold:
- Function: Derives a decision threshold $\tau^*$ based on operator-specified misclassification costs.
- Formula: $\tau^* = \frac{C_{FP}}{C_{FP} + C_{FN}}$ , where $C_{FP}$ and $C_{FN}$ are the costs of false positives and false negatives, respectively.
- Distinction: This threshold is derived from cost ratios, not validation set optimization.
Conformal Risk Control (CRC) Wrapper:
- Function: Converts an operator-specified alert budget $\alpha$ (e.g., maximum false positive rate) into a valid threshold $\hat{\tau}_\alpha$ under the assumption of exchangeability between validation and test negatives.
- Guarantee: Provides a marginal expectation bound on the false positive rate: $E[FPR_{test}(\hat{\tau}_\alpha)] \leq \alpha$ .
- Mechanism: Selects the infimum threshold $\tau$ such that the empirical FPR on validation negatives, plus a slack term $1/(n_0+1)$ , remains below $\alpha$ .
Multi-Window Burn-Rate Alerting:
- Function: An escalation layer adapted from Site Reliability Engineering (SRE) practices.
- Logic: Converts threshold crossings into alerts only when the "burn rate" (consumption of the alert budget) exceeds a threshold over paired short and long time windows. This filters transient noise while escalating sustained anomalies.

3. Key Contributions

The paper frames its contributions as mapping the operational scope and limitations of the pipeline rather than claiming uniform dominance:

Regime-Sensitivity Mapping: The study evaluates the pipeline across three distinct attack-prevalence regimes:
- Rare (LITNET-2020, ~5.2%): CALIBURN excels.
- Moderate (CICIDS2017, ~22.06%): CALIBURN remains the strongest streaming method but is outperformed by batch density methods.
- High (UNSW-NB15, ~64%): All streaming methods, including CALIBURN, degrade to near-random performance due to base-rate inversion.
Identification of CRC-Collapse Mechanisms: The authors identify and quantify two mechanisms where Conformal Risk Control can fail at small alert budgets $\alpha$ :
- Theoretical Overshoot: The bound $2B/(n_0+1)$ can cause the procedure to overshoot the target $\alpha$ .
- Empirical-Density Failure: When the distribution of validation negative scores is heavily concentrated below candidate thresholds, the procedure may select a threshold that produces zero alerts (degeneracy) to satisfy the bound vacuously.
Explicit Calibration Analysis: The paper demonstrates that isotonic calibration significantly reduces Brier scores (30–63% reduction across datasets) compared to raw BOCPD posteriors, validating the necessity of mapping regime-shift scores to conditional attack probabilities.
Ablation of Feature Artifacts (UNSW-NB15): The authors test the hypothesis that the collapse of streaming methods on UNSW-NB15 is caused by synthetic TTL feature artifacts. By removing these features, performance did not degrade but slightly improved. This confirms the collapse is intrinsic to streaming detection under high prevalence (base-rate inversion), not an artifact of removable feature leakage.
Operational Guidelines: The paper delineates the specific conditions under which CALIBURN is appropriate:
- Attack prevalence < ~25%.
- Alert budget $\alpha$ is comfortably above the theoretical overshoot bound $2B/(n_0+1)$ .
- The calibration distribution is approximately exchangeable with the deployment stream.

4. Experimental Results

Datasets and Baselines

Datasets: LITNET-2020 (5.2% attack), CICIDS2017 (22.06% attack, corrected version), UNSW-NB15 (64% attack).
Baselines: Five streaming methods (Half-Space Trees, KitNET, LODA, Robust Random Cut Forest, iForest_ASD) and three batch references (LOF, ECOD, COPOD).

Performance by Regime

Rare-Attack Regime (LITNET-2020):
- CALIBURN achieved an AUC-PR of 0.943.
- Outperformed the best streaming baseline (LODA) by 2.21× and the best batch reference (ECOD) by 4.12×.
- Achieved high precision (0.976) with moderate recall (0.451), reflecting the cost-sensitive trade-off.
- Isotonic calibration reduced the Brier score by 30%.
Moderate-Prevalence Regime (CICIDS2017):
- CALIBURN achieved an AUC-PR of 0.545, remaining the strongest streaming method.
- However, it was outperformed by the batch method LOF (AUC-PR 0.863).
- The authors attribute this to the contamination of the streaming reference distribution by the higher volume of attacks, which batch methods avoid by using the full training set.
High-Prevalence Regime (UNSW-NB15):
- CALIBURN achieved an AUC-PR of 0.653, which is indistinguishable from the prevalence floor (0.676).
- All streaming methods collapsed to near-random ranking (AUC-ROC ~0.5).
- Batch methods (LOF, ECOD) performed better but in a setting where the "normal" baseline is no longer benign.

Ablation Findings

In the rare-attack regime, the full pipeline (V1) significantly outperforms the "strawman" variant (V4: raw scores + cost threshold), which suffered a high False Positive Rate (9.6%).
In high-prevalence regimes, the full pipeline (V1) often degenerates to zero alerts due to CRC constraints, while the raw-score variant (V4) produces more usable (though lower precision) results.

5. Significance and Claims

The paper modestly claims that CALIBURN is not a universal replacement for all intrusion detection methods. Its significance lies in:

Operational Alignment: It provides a framework where alerting thresholds are derived from operator-specified costs and budgets rather than post-hoc label tuning, making the system explainable and defensible before deployment.
Regime Awareness: It explicitly identifies the "rare-attack" regime as the sweet spot for streaming, cost-sensitive, calibrated detection. It warns practitioners that streaming methods are structurally limited when attack prevalence inverts the base rate (i.e., when attacks become the majority).
Practical Guidelines: It offers concrete deployment checks, such as verifying the alert budget $\alpha$ against the CRC overshoot bound and monitoring for the "empirical-density failure" mode where CRC produces zero alerts.

The authors conclude that CALIBURN is best suited for Security Operations Centers where attacks are infrequent, analyst capacity is finite, and the alerting policy must be defined by operational constraints rather than dataset-specific tuning. Future work is suggested for online recalibration to handle concept drift and adaptive cost-ratio estimation.