📊 statistics

Information Leakage Detection through Approximate Bayes-optimal Prediction

This paper proposes a theoretical framework that leverages automated machine learning to approximate Bayes-optimal predictors for accurately estimating mutual information, thereby overcoming the limitations of conventional methods and enabling superior detection of information leakage in both synthetic and real-world datasets.

Original authors: Pritha Gupta, Marcel Wever, Eyke Hüllermeier

Published 2026-06-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Pritha Gupta, Marcel Wever, Eyke Hüllermeier

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a security guard at a high-tech bank. Your job is to figure out if a vault is leaking secrets. Sometimes, the vault doesn't just "open" to reveal its contents; it might leak information through subtle clues, like the sound of the lock clicking, the heat coming off the metal, or the time it takes to open. In the digital world, this is called Information Leakage (IL). Hackers can listen to these "side channels" (like network delays or power usage) to guess secret passwords or keys.

The problem is: How do you know for sure if a leak is happening?

The Old Way: Guessing the Weather

Traditionally, scientists tried to measure this leakage using a complex mathematical concept called Mutual Information (MI). Think of MI as a "leak meter." If the meter reads zero, the system is safe. If it reads high, secrets are leaking.

However, measuring this meter is like trying to guess the exact temperature of a stormy ocean from a single drop of water. It's incredibly hard when the data is messy, huge, or unbalanced (like having 99% of your data be "safe" and only 1% "leaking"). Old methods often got confused, gave false alarms, or took too long to compute.

The New Idea: The "Perfect Predictor" Test

The authors of this paper propose a clever new way to check for leaks. Instead of trying to measure the leak directly, they ask a different question: "Can a smart computer learn to predict the secret just by looking at the public clues?"

Here is the analogy:

The Secret: A hidden number (like a password).
The Clue: A public signal (like how long a server took to reply).
The Test: You hire a super-smart AI (the "Bayes-optimal predictor") to guess the secret number based only on the public clue.

The Logic:

If there is NO leak: The public clue tells you nothing about the secret. Even the smartest AI will guess no better than random chance.
If there IS a leak: The public clue contains a hint. The smart AI will start guessing correctly much more often than random chance.

The paper introduces a framework to measure exactly how much better the AI gets. If the AI's performance jumps significantly, it proves a leak exists.

The Tools: The "Auto-Shop" and the "Calibration"

To make this work, the authors used two powerful tools:

AutoML (Automated Machine Learning): Instead of building a custom AI for every test, they used "AutoML" tools (specifically AutoGluon and TabPFN). Think of these as a high-tech auto-shop that automatically builds the best possible car (AI model) for the job without needing a human mechanic to tweak every bolt.
Calibration: Sometimes, these AI tools get overconfident. They might say, "I'm 99% sure!" when they are actually only 60% sure. This is like a weatherman who always predicts rain but is wrong half the time. The authors added a "calibration" step (like adjusting a thermometer) to ensure the AI's confidence scores are accurate. This was crucial for avoiding false alarms, especially when the data was messy (imbalanced).

The Experiment: The "Timing" Game

The team tested their method on a real-world scenario involving OpenSSL (a common software for secure internet connections).

The Setup: They created two types of servers. One was "vulnerable" (it took slightly longer to process fake messages), and one was "secure" (it took the same amount of time for everything).
The Challenge: They tried to detect the leak in the vulnerable server, even when the time difference was tiny (microseconds) and the data was messy.

The Results

Accuracy: Their new method, using the "AutoML" tools with "calibration," was much better at spotting leaks than the old methods. It could find leaks even when they were very faint or the data was unbalanced.
Speed vs. Precision: One tool (AutoGluon) was fast and practical for real-world use. The other (TabPFN) was incredibly accurate but took a long time to run.
The Verdict: By using these automated tools and fixing the "confidence" of the AI, they created a reliable way to detect information leaks that avoids the pitfalls of older, clunky statistical methods.

In Summary

The paper says: "Don't try to measure the leak directly with a complex ruler. Instead, see if a smart, automated AI can learn the secret from the clues. If the AI learns it, the system is leaking. We found that using modern, automated AI tools with a 'calibration' step is the most accurate and reliable way to catch these digital leaks."

Technical Summary: Information Leakage Detection through Approximate Bayes-optimal Prediction

1. Problem Statement

In data-driven environments, the proliferation of publicly available information and IoT technologies has exacerbated the risk of Information Leakage (IL). IL occurs when sensitive or confidential information is inadvertently exposed to unauthorized parties through observable system data (e.g., network traffic, power consumption, timing delays).

Traditional approaches to detecting IL rely on estimating the Mutual Information (MI) between observable and secret variables. However, these statistical methods face significant hurdles:

Curse of Dimensionality: MI estimation becomes computationally intractable and inaccurate in high-dimensional spaces.
Convergence and Complexity: Existing non-parametric estimators often struggle with convergence rates and computational costs.
Imbalance and Noise: Supervised machine learning (ML) approaches for IL detection (ILD) are often limited to binary classification tasks, struggle with imbalanced real-world datasets, and may produce false negatives or miss specific subclasses of leakage.
Lack of Framework: Current ML-based methods often lack a comprehensive theoretical framework connecting model performance directly to information-theoretic leakage metrics.

2. Methodology

The paper proposes a theoretical framework grounded in statistical learning theory and information theory to quantify and detect IL. The core premise is that the presence of IL is equivalent to the existence of a side-channel, which can be detected by analyzing the learnability of the system's data.

2.1 Theoretical Foundation: LAS and Bayes Predictors

The authors define a Generalized Leakage Assessment Score (LAS), denoted as $\delta(\cdot)$ , which measures the performance difference between two predictors:

Bayes Predictor ( $g_b$ ): The optimal classifier minimizing expected risk (e.g., Log-Loss or 0-1 loss) given the joint distribution of observable ( $X$ ) and secret ( $Y$ ) data.
Marginal Bayes Predictor ( $g_{mc}$ ): The optimal classifier assuming $X$ and $Y$ are independent (i.e., using only the marginal distribution of $Y$ ).

The LAS is defined as the difference in performance (loss or accuracy) between these two.

If $X$ and $Y$ are independent (no leakage), the Bayes predictor converges to the Marginal Bayes predictor, and LAS $\approx 0$ .
If $X$ and $Y$ are dependent (leakage exists), the Bayes predictor significantly outperforms the Marginal Bayes predictor, resulting in LAS $\gg 0$ .

Crucially, when using Log-Loss (Cross-Entropy), the LAS is mathematically equivalent to Mutual Information:
$\delta(\ell_{ll}) = I(X; Y) = E[\ell_{ll}(g_{mc})] - E[\ell_{ll}(g_b)]$

2.2 Estimation via Automated Machine Learning (AutoML)

Since the true Bayes predictor is unknown, the authors approximate it using empirical risk minimizers trained via AutoML tools (TabPFN and AutoGluon). They propose two primary estimation approaches:

Log-Loss Estimation (Cal Log-Loss):
- Estimates MI by calculating the difference in expected Log-Loss between the trained probabilistic classifier ( $g_p$ ) and the marginal classifier ( $g_{mc}^p$ ).
- Calibration: To address the issue of overconfident predictions (especially in imbalanced datasets), the authors employ post-hoc calibration techniques (Isotonic Regression, Platt's Scaling, Beta Calibration, Temperature Scaling, Histogram Binning) to improve the accuracy of probability estimates.
- This approach directly addresses data imbalance, which often causes overestimation in other methods.
Mid-Point Estimation:
- Estimates MI using the relationship between the Bayes error rate and conditional entropy bounds (Hellman-Raviv and Fano bounds).
- The authors note this method can suffer from overestimation in imbalanced datasets, leading to false positives.

2.3 Detection Protocol

To robustly detect IL, the authors propose a statistical testing framework:

Hypothesis Testing: They apply statistical tests (One-sample t-test for MI, Paired t-test for accuracy, Fisher's Exact Test for confusion matrices) to the performance metrics of the top-10 AutoML pipelines.
Null Hypothesis ( $H_0$ ): No IL exists (performance difference is zero).
Multiple Hypothesis Correction: To mitigate false positives arising from testing multiple models, they apply the Holm-Bonferroni correction. A leakage is declared only if a sufficient number of hypotheses are rejected (threshold $\tau \geq 5$ ).

3. Key Contributions

Generalized ILD Framework: A theoretical framework using LAS to quantify IL, unifying classification performance with information-theoretic measures.
MI Estimators via AutoML: Novel methods to estimate MI by approximating the Bayes predictor's Log-Loss and accuracy using AutoML tools (TabPFN, AutoGluon), specifically addressing the challenges of high dimensionality and class imbalance.
Handling Imbalance: Demonstration that Log-Loss estimation, when combined with calibration techniques, effectively handles data imbalance, avoiding the false positives common in Mid-Point estimation.
Robust Detection Protocol: An ILD methodology incorporating Holm-Bonferroni correction to ensure high-confidence detection and minimize false positives in security-sensitive contexts.
Empirical Validation: A comprehensive study comparing the proposed methods against state-of-the-art baselines (MINE, GMM, PC-Softmax) on synthetic and real-world datasets.

4. Experimental Results

4.1 MI Estimation Performance

Experiments were conducted on synthetic datasets generated using Multivariate Normal (MVN) distributions with varying noise levels, dimensions, and class imbalances.

Superiority of TabPFN: The TabPFN IR Cal Log-Loss approach consistently outperformed all baselines (MINE, GMM, PC-Softmax) and AutoGluon-based methods in estimating MI, particularly in high-dimensional and imbalanced scenarios.
Robustness: TabPFN demonstrated strong generalization across varying numbers of classes ( $C$ ), input dimensions ( $d$ ), and noise levels ( $\epsilon$ ).
Calibration Impact: Calibration techniques significantly improved AutoGluon's performance in vulnerable datasets but sometimes degraded performance in non-vulnerable systems, highlighting the context-dependency of calibration.

4.2 ILD on Real-World Data (OpenSSL TLS)

The methods were evaluated on timing side-channel datasets from OpenSSL TLS servers (simulating Bleichenbacher's attacks).

Imbalanced Datasets: TabPFN Cal Log-Loss achieved the highest detection accuracy (up to ~94% for long time delays), significantly outperforming baselines. AutoGluon FET-Median also performed well.
Balanced Datasets: AutoGluon Mid-Point and TabPFN IR Cal Log-Loss detected approximately 99% of leaks.
Comparison: The proposed MI-based approaches generally outperformed classification-based baselines (PTT, FET) in imbalanced scenarios, while PTT-Majority showed strong performance in balanced scenarios.
Efficiency: AutoGluon offered a favorable trade-off between runtime (~~2,400 seconds) and accuracy, whereas TabPFN provided higher accuracy at a higher computational cost (~~30,000 seconds).

5. Significance and Claims

The paper claims to provide a comprehensive and automated framework for Information Leakage Detection that overcomes the limitations of traditional statistical MI estimation and existing ML-based side-channel analysis.

Theoretical Rigor: By grounding IL detection in the performance gap between Bayes and Marginal Bayes predictors, the work establishes a direct link between machine learning generalization and information leakage.
Practical Applicability: The use of AutoML tools allows for the detection of IL in complex, high-dimensional, and imbalanced real-world scenarios (like OpenSSL timing attacks) without requiring manual feature engineering or domain-specific tuning.
Robustness: The integration of statistical correction (Holm-Bonferroni) and calibration techniques ensures that the detection system minimizes false positives, a critical requirement for security applications.
Scalability: The approach is presented as a scalable solution for detecting leaks in modern, data-intensive systems, offering an automated alternative to deep learning models that often require extensive training and tuning.

The authors conclude that their approach, particularly using TabPFN with calibrated Log-Loss, effectively estimates MI and detects timing side-channel leaks, outperforming state-of-the-art methods while providing a robust theoretical basis for future ILD research.