💻 computer science

Near-Deterministic Reconstruction of Enterprise Firewall Decisions for Policy Audit: Proxy Structure, Context Shift, and Residual Uncertainty

This study demonstrates that enterprise firewall decisions can be near-deterministically reconstructed from traffic logs using supervised learning, revealing that high model accuracy often reflects the replication of specific policy proxies and context-dependent rules rather than independent threat detection, thereby enabling retrospective policy audits while highlighting significant limitations in cross-environment generalization and uncertainty calibration.

Original authors: Özkan Canay, Cem Özkurt, Fatmir Garri

Published 2026-08-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Özkan Canay, Cem Özkurt, Fatmir Garri

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out why a bouncer at a super-strict club lets some people in and turns others away. You have a giant notebook filled with millions of entries, each recording exactly what the bouncer did: "Allowed," "Dropped," or "Denied." In the world of computer security, this notebook is called a "firewall log," and the bouncer is a firewall—a digital gatekeeper that protects a company's network from bad actors. Usually, security experts use these logs to hunt for hackers, looking for patterns that scream "danger!" But there's a tricky problem: sometimes the bouncer's decision isn't about whether a person is actually a criminal; it's just about whether they fit the club's specific dress code or membership list. If you train a computer to guess the bouncer's decisions, it might just learn the dress code perfectly, not because it's a genius detective, but because the clues in the notebook (like the person's shoe size or the time they arrived) give away the answer too easily. This paper asks a fascinating question: If we hide the obvious clues, can a computer still guess the bouncer's moves with near-perfect accuracy? And if it can, does that mean the computer is smart, or just that the clues were too easy to find?

This study dives into a massive notebook containing over one million records from a real corporate firewall, specifically a Palo Alto device. The researchers wanted to see if they could rebuild the "decision surface"—the invisible line the firewall draws to decide who gets in—using only the surrounding details, while hiding the direct reasons for the decision. They treated the firewall's choices (Allow, Drop, Deny) like a puzzle. They removed the most obvious pieces of the puzzle, like the specific application name or the rule number that triggered the action, to see if the remaining clues (like the size of the data packet or the port number) were enough to solve it.

The results are a bit like finding out the club's bouncer is actually following a very rigid, predictable script. The researchers discovered that even after removing the most obvious "shortcuts" (like the application name), powerful computer programs called "tree ensembles" (think of them as super-smart decision trees that ask a series of yes-or-no questions) could still guess the firewall's decision almost perfectly. In fact, two of these programs, XGBoost and LightGBM, got a perfect score, making zero mistakes on the test data. It turns out that the firewall's decision is so tightly linked to the other details in the log that you don't even need to know the rule to know the outcome. For example, they found that just knowing the "Source Port" (a specific number identifying the connection) and the "Bytes" (the size of the data) was enough to determine the decision in millions of cases.

However, the paper is very careful not to call this a "win" for hacking detection. The authors argue that this near-perfect score doesn't mean the computer has learned to spot real hackers. Instead, it means the computer has learned the firewall's internal "grammar." It's like a student who memorizes the answer key to a test rather than understanding the math. When the researchers tried to make the puzzle harder by removing even more clues—like the size of the data or the country of origin—the computer's perfect score dropped, but it was still surprisingly good. This suggests that the firewall's decisions are "redundantly encoded," meaning the same information is repeated in many different fields of the log.

The study also looked at what happens when the computer encounters a situation it hasn't seen before, like a new type of app or a different destination. Here, the magic spell breaks. The computer's confidence wavers, and it starts making mistakes, especially with the rare "Deny" decisions. The researchers used special tools to measure this uncertainty, showing that while the computer is a master at guessing the firewall's moves in familiar territory, it gets lost when the context shifts. They found that if you hide the "application" details, the computer can still guess 99.9% of the time, but if you hide the "volume" and "duration" of the traffic, the accuracy drops slightly, proving that those details were acting as hidden shortcuts.

Ultimately, this paper isn't about building a better hacker detector; it's about auditing the firewall itself. It shows that for this specific company and this specific time window (about 46 minutes of traffic), the firewall's decisions are almost entirely predictable from the surrounding data. The researchers found seven tiny combinations of just four fields that could predict the outcome perfectly. But they also warn that this is a "near-deterministic" reconstruction, meaning it works because the data is consistent, not because the computer understands security. If the firewall's rules change or the network environment shifts, this perfect prediction might vanish. The paper concludes that while we can rebuild the firewall's decision-making process with high accuracy, we must be careful not to mistake this reconstruction for a genuine ability to detect new, unknown threats. It's a powerful tool for checking if the firewall is behaving consistently, but it's not a crystal ball for the future.

Technical Summary: Near-Deterministic Reconstruction of Enterprise Firewall Decisions

Problem Statement
This study addresses a methodological interpretation risk in network security machine learning: the conflation of high-performance supervised learning on firewall logs with independent threat detection. Enterprise firewall decisions (Allow, Drop, Deny) are deterministic outcomes generated by configured policies, application identification, and session termination behaviors, rather than human-validated labels of maliciousness. When models achieve near-perfect accuracy on these logs, it is unclear whether they have learned generalizable security reasoning or merely the "operational grammar" and proxy structures of a specific device's decision surface. The paper investigates how far a deployed firewall's logged decision surface can be reconstructed from structured log fields after excluding direct label-source fields, and what residual uncertainty remains when proxy variables are reduced or contexts shift.

Methodology
The research utilizes a real-world dataset of 1,048,576 traffic-log records from a Palo Alto Networks next-generation firewall, captured over a 46-minute window. The target variable is a three-class operational decision (Allow, Drop, Deny), constructed from traffic subtype and session-end reason fields.

Data Preprocessing: Direct label-source fields (raw action, traffic subtype, session-end reason) are strictly excluded from all predictor views. The dataset is split into an 80/20 stratified hold-out (random and time-ordered variants) and a capped sample for cross-validation.
Feature Views: The study employs several feature configurations to test proxy dependence:
- Core Baseline: 24 fields including application context, zones, interfaces, ports, countries, and post-decision telemetry (bytes, duration).
- No-Application-Context: Removes application taxonomy and category fields to test reliance on overt application proxies.
- Strict Proxy-Minimal: Retains only zone/interface and transport/port fields (with/without country), removing application, volume, and policy-proximal fields.
- With-Policy: Adds rule and action-source fields as an empirical upper-bound diagnostic.
Model-Free Audit: Before training classifiers, the authors perform a deterministic audit using exact grouping to identify minimum determining sets (field subsets that perfectly predict the label) and calculate conditional entropy. This includes checking for conflict rows (contexts with multiple labels) and singleton contexts.
Machine Learning Models: The study evaluates scalable tree ensembles (XGBoost, LightGBM, Extra Trees, CatBoost) and single decision trees. Models are trained with balanced class weights to address the severe class imbalance (Allow: ~77%, Drop: ~22%, Deny: ~1.3%).
Uncertainty Diagnostics: The study employs conformal prediction (marginal and Mondrian variants), adaptive prediction sets (APS), and selective classification (confidence-based abstention) to quantify residual uncertainty, calibration (ECE, Brier score), and error pockets under context shifts.

Key Contributions

Model-Free Reconstructability Audit: The study identifies seven minimum four-field determining sets in the full export that perfectly determine the decision label. These sets consistently include Source Port and session-end Bytes, combined with two other fields (e.g., Outbound Interface, Application Category). The audit confirms zero conflict rows in the observed export, establishing empirical determinism.
Boundary-Stress Diagnostics: Through feature-family ablations and context-held-out tests (holding out entire application/category, destination-service, or rule groups), the paper maps the stability of the decision boundary. It demonstrates that while reconstruction remains near-perfect when removing application context, it degrades significantly under strict proxy-minimal views and context shifts.
Conformal and Selective Diagnostics: The research exposes that marginal conformal prediction can hide undercoverage of the rare "Deny" class. It shows that probability-threshold queues and confidence-based selective classification are highly sensitive to model representation (ordinal vs. native categorical) and calibration, particularly under context shift.
Schema-Independent Instantiation: The protocol is applied to an independent public dataset (UCI Internet Firewall), reproducing high reconstructability and the "accuracy vs. minority-class performance" gap, confirming that the observed patterns are not unique to the specific enterprise export's schema.

Results

Reconstruction Performance: In the core baseline (excluding direct label sources), XGBoost and LightGBM achieved perfect hold-out performance (1.000 Macro-F1, 0 errors). Even in the "no-application-context" ablation, performance remained near-perfect (0.999981 Macro-F1).
Determinism: The model-free audit found that 64.7% to 71.4% of records occupy singleton key contexts. The seven determining four-field sets found in the full data also held exactly in the training-only discovery and maintained agreement on seen contexts in held-out tests.
Proxy Reduction: Removing volume and duration fields reduced Macro-F1 to 0.9962. A stricter proxy-minimal view (zones, interfaces, ports only) caused a significant drop: XGBoost stabilized around 0.951 Macro-F1, while LightGBM became unstable (0.42–0.90 range) depending on the random seed.
Context Shift: When entire application/category groups were held out, Macro-F1 for ordinal LightGBM dropped to ~0.673. However, native-categorical CatBoost recovered this to 0.9923 on the same data, indicating that representation (handling of unseen categories) is a critical factor in context-shift performance.
Uncertainty: In random splits, uncertainty metrics (entropy, ECE) were near zero. Under context shift, uncertainty rose, and conformal sets expanded or became empty. Crucially, marginal conformal sets showed significant undercoverage for the rare "Deny" class (e.g., 58% coverage at $\alpha=0.10$ ), which was restored to >90% using Mondrian (class-conditional) conformal prediction.
Runtime: The full million-row reconstruction was computationally feasible on modest cloud hardware, with XGBoost training in ~25 seconds and predicting at ~6 ms per 1,000 rows.

Significance and Claims
The paper claims that near-perfect firewall decision scores should be interpreted as evidence of a stable, deterministic decision surface within a specific export, rather than proof of general attack detection or independent security reasoning. The high reconstructability arises because operational logs contain redundant traces of the policy that generated the decision (e.g., session-end bytes and ports reflecting the outcome).

The study positions itself as a tool for retrospective, claim-bounded decision reconstruction and simulated uncertainty diagnostics. It argues that such models are valuable for policy audits, migration rehearsals, and identifying configuration inconsistencies or data-quality issues, but they cannot replace deterministic rule analysis or serve as cross-environment intrusion detectors. The findings highlight that "perfect" scores in security ML often reflect data provenance and proxy structures, necessitating rigorous ablation, context-shift testing, and uncertainty quantification before drawing operational conclusions.

Technical Summary: Near-Deterministic Reconstruction of Enterprise Firewall Decisions

More like this