💻 computer science

Cyber Dynamics I: Finite Macrostates for Behavioral Anomaly Detection in Network Telemetry

This paper proposes a finite-dimensional macrostate framework for network telemetry that moves beyond scalar entropy metrics to model behavioral state-space transitions, thereby improving the discrimination between benign workload drift and adversarial reorganization while offering more interpretable anomaly detection.

Original authors: Abdul Rahman, Eranga Bandara, Sachin Shetty

Published 2026-07-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Abdul Rahman, Eranga Bandara, Sachin Shetty

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to detect a thief in a busy office building.

The Old Way (Entropy-Only)
For a long time, security guards have used a simple method: they count how many people are moving around in a hallway. If the hallway is usually quiet and suddenly 50 people rush through, the alarm goes off. Or, if the hallway is usually chaotic and suddenly becomes perfectly still, the alarm sounds.

In the world of computer networks, this is called entropy. It measures "disorder." If the data traffic looks unusually messy or unusually organized compared to the norm, the system flags it.

The Problem
The problem is that this method is too simple.

Imagine a scheduled fire drill. Suddenly, 500 people rush out of the building. The guard's "disorder" meter spikes, but it's not a thief; it's just a normal event.
Conversely, imagine a thief who is very quiet. They don't rush; they just walk slowly from one office to another, stealing files. The "disorder" meter stays normal because the crowd isn't moving wildly. The thief gets away because they didn't create a "mess."

The old method looks at the noise (the chaos) but misses the story (who is doing what, and how they are connected).

The New Way (Cyber Dynamics & Macrostates)
This paper proposes a new way to look at the building. Instead of just counting people, the security system creates a "Macrostate."

Think of a Macrostate as a snapshot of the building's "vibe" or "personality" over a short period (like a 60-second window). This snapshot doesn't just count people; it captures a whole profile:

Activity: How much is happening? (Is the building full or empty?)
Disorder: Is the movement chaotic or orderly? (The old entropy part).
Structure: Who is talking to whom? (Is everyone chatting with everyone, or is one person whispering to a specific few?)
Volatility: Is the movement jerky and sudden, or smooth?
Persistence: Are the same people walking the same path over and over?
Deviation: How does this vibe compare to what "normal" looks like?

The Analogy: The Orchestra

The Old Method listens only to the volume. If the orchestra gets too loud or too quiet, it sounds an alarm.
The New Method listens to the music. It notices if the violins suddenly start playing the drums, or if the conductor is whispering instructions to the bass section while the rest of the band plays a lullaby. Even if the volume (entropy) stays the same, the structure of the music has changed. That change in structure is the anomaly.

How It Works in Practice
The researchers tested this on real network data (like traffic logs from the UNSW-NB15 and CIC-IDS2017 datasets). They compared their new "Macrostate" system against the old "Entropy-only" systems.

The Results

Better Detection: The new system was much better at spotting the "quiet thieves" (attacks that look organized rather than chaotic) and ignoring the "fire drills" (normal busy times that look chaotic).
The "60-Second" Sweet Spot: They found that looking at the data in 60-second chunks worked best. Looking at 30 seconds was okay, but looking at 5 minutes was too blurry to see the details.
It's About the Story, Not Just the Noise: The experiments showed that adding "Structure" (who is talking to whom) and "Activity" (how busy it is) to the "Disorder" score made the system much smarter. The old entropy-only method was like trying to understand a movie by only looking at the brightness of the screen; the new method watches the plot.

What It Doesn't Do (Yet)
The paper is honest about its limits.

It focuses on network traffic (the hallways of the building), not on what happens inside the computers (the offices) or who has the keys (identity).
It didn't fully master the art of predicting the next step. While the system is great at describing the current "vibe," predicting exactly how the "vibe" will change in the next second is still a work in progress.

The Bottom Line
This paper says: "Stop just counting the chaos. Start understanding the story." By turning raw network data into a rich, multi-dimensional "personality profile" of the system, we can spot bad actors who are trying to hide in plain sight, even when they aren't making a lot of noise.

Technical Summary: Cyber Dynamics I: Finite Macrostates for Behavioral Anomaly Detection in Network Telemetry

1. Problem Statement

Modern cyber defense operates in dynamic, heterogeneous environments where distinguishing between benign operational variability (e.g., scheduled backups, cloud autoscaling) and adversarial reorganization (e.g., APT lateral movement, coordinated staging) is a central challenge. Existing anomaly detection methods, particularly those based on entropy, often treat network telemetry as isolated events or narrow feature vectors. These approaches typically compute entropy as a scalar statistic over marginal distributions (e.g., source IPs, ports).

The paper argues that this "scalar entropy" approach has two critical limitations:

Narrow Representation: It reduces system behavior to disorder metrics, failing to capture structural organization, persistence, or coordination patterns.
Static View: It treats anomalies as unusual distributions at a single point in time, ignoring the temporal trajectory and regime transitions that often characterize sophisticated attacks.

Consequently, benign workload drift can mimic attack-induced instability, and organized attacks can preserve marginal distributions while fundamentally altering system structure, leading to false positives or missed detections.

2. Methodology: The Finite Macrostate Framework

The paper proposes a finite-dimensional macrostate framework instantiated over the Canonical Security Telemetry Substrate (CSTS). This approach shifts the analysis from isolated event records to coarse-grained behavioral states.

A. Formal Construction
The framework operates through three levels:

Microstates: The atomic unit is a typed network event $e = (\tau, src, dst, svc, proto, dir, attrs)$ , representing a temporally localized interaction between persistent entities.
Windowed Micro-Ensembles: Events are aggregated into time windows $W_t$ to form a multiset of events $E_t$ . This captures local behavioral context rather than isolated points.
Macrostate Map ( $\Phi$ ): A coarse-graining function maps each ensemble $E_t$ $E_{t}$ to a finite-dimensional vector $X_t \in \mathbb{R}^d$ $X_{t} \in R^{d}$ . This vector is composed of seven interpretable observable families:
- Activity ( $A_t$ ): Load and occupancy (e.g., flow counts, unique entities).
- Disorder ( $H_t$ ): Distributional dispersion (Shannon, Rényi, or Tsallis entropy over roles, services, protocols).
- Structural Order ( $S_t$ ): Interaction topology (graph density, degree concentration, motif counts).
- Temporal Volatility ( $V_t$ ): Short-horizon instability and burstiness.
- Persistence ( $P_t$ ): Recurrence of edges, paths, or motifs across windows.
- Coupling ( $C_t$ ): Dependencies between behavioral components (omitted in the first empirical pass for simplicity).
- Deviation ( $D_t$ ): Distance from benign baselines (e.g., Jensen-Shannon divergence).

B. Dynamical Analysis
The framework treats cyber behavior as a trajectory $\{X_t\}$ through macrostate space. Anomaly detection is defined via two mechanisms:

State Anomaly: The current macrostate $X_t$ lies outside the benign regime set $\mathcal{B}$ .
Transition Anomaly: The step from $X_t$ to $X_{t+1}$ deviates from the nominal transition operator $T$ , which models expected benign evolution. This allows detection of abnormal motion even if individual states appear benign.

C. Theoretical Justification
The paper provides formal propositions demonstrating that:

Macrostate maps are well-defined and permutation-invariant.
Entropy-only macrostates are structurally incomplete; distinct micro-ensembles can have identical entropy but different structural organization.
Multiscale analysis is necessary, as anomaly visibility depends on temporal resolution (window size).

3. Experimental Design

The framework was evaluated on benchmark network telemetry datasets: UNSW-NB15 and CIC-IDS2017.

Preprocessing: Events were normalized to a unified schema, roles were mapped, and data was chronologically ordered to preserve temporal structure.
Windowing: Experiments utilized multiple scales ( $\Delta \in \{30s, 60s, 300s, 900s\}$ ) with overlapping strides.
Baselines: Comparisons were made against:
- Entropy-only detectors (thresholding, logistic regression on entropy vectors).
- Standard anomaly detectors (Isolation Forest, One-Class SVM, Autoencoders) on aggregate features.
Ablation: The study systematically tested the contribution of each macro-variable family (Activity, Disorder, Structure, Volatility, Persistence, Deviation).

4. Key Results

A. Superiority of Macrostates over Entropy-Only
On both datasets, the full macrostate representation consistently outperformed entropy-only baselines.

UNSW-NB15: At the 60s scale, the full macrostate achieved an AUPRC of 0.9730, significantly outperforming the best entropy-only baseline (One-Class SVM at 0.9631).
CIC-IDS2017: The full macrostate (AUPRC ~0.7917) outperformed the best entropy-vector baseline (AUPRC ~0.7672) at 60s.

B. Value of Structural and Temporal Variables
Ablation studies revealed that gains were not merely due to increased dimensionality but to specific behavioral components:

Adding Activity to Disorder provided a clear gain, contextualizing entropy.
Adding Structural Order yielded the most significant improvement, confirming that attacks often manifest as organized reconfiguration rather than just increased randomness.
Persistence and Deviation provided a final, smaller but positive lift.

C. Scale Sensitivity
The "best" macrostate composition varied by temporal scale. For example, on CIC-IDS2017, the intermediate $A+H+V$ (Activity, Disorder, Volatility) configuration was strongest at 30s, while the full macrostate was best at 60s. This supports the theoretical claim that anomaly visibility is scale-dependent.

D. Transition Scoring Limitations
Contrary to the theoretical promise, transition-aware scoring (state + transition fusion) did not outperform state-only scoring in the initial experiments. The authors attribute this to the simplicity of the nominal transition operators used (e.g., naive persistence, linear regression), suggesting that the full dynamical benefit requires more sophisticated trajectory modeling in future work.

5. Significance and Claims

The paper positions itself as the first empirical step in a broader "cyber dynamics" program. Its primary contributions and claims are:

Shift from Scalar to State: It demonstrates that cyber anomaly detection is better framed as a problem of behavioral state-space construction rather than scalar irregularity detection.
Entropy as a Coordinate, Not a Solution: It proves that while entropy is a useful component, it is insufficient as a standalone descriptor for complex cyber behaviors that involve structural reorganization.
Interpretability: The macrostate framework provides a compact, interpretable representation (activity, structure, volatility, etc.) that allows analysts to reason about why a state is anomalous, rather than just that it is anomalous.
Foundational Substrate: By instantiating the framework over CSTS, the paper establishes a canonical foundation that is portable across network, endpoint, and heterogeneous telemetry, setting the stage for future work on stability, recovery, and cross-modal analysis.

The authors explicitly note limitations: the transition scoring results are preliminary, multiscale fusion is not yet superior to the best single-scale model, and the current evaluation is network-centric. These are framed not as failures, but as the defined boundary of "Paper I" and the roadmap for "Cyber Dynamics II and III."