Post-Selection Distributional Model Evaluation

1. Problem Statement

The paper addresses a critical gap in formal model evaluation: the inability to reliably estimate the full distribution of Key Performance Indicators (KPIs) after a model has been pre-selected based on the same data.

• Context: In many applications (e.g., Large Language Model deployment, telecom network slicing), users do not just need to know if a model meets a single fixed target (e.g., "latency < 100ms"). Instead, they need to explore the trade-off between performance and reliability across the entire distribution (e.g., "What is the latency at the 99th percentile?").
• The Challenge:
  1. Unknown Targets: The optimal reliability level (e.g., acceptable failure probability $\gamma$) is often unknown a priori.
  2. Post-Selection Bias: To select the best candidate models from a large pool, practitioners often use the same dataset for both selection (screening models) and evaluation (estimating the KPI distribution).
  3. Failure of Naive Methods: Applying standard confidence intervals (like those based on the Dvoretzky–Kiefer–Wolfowitz inequality) to data that has been used for selection leads to overly optimistic estimates and invalid statistical guarantees (the true distribution falls outside the confidence band more often than expected).
  4. Inefficiency of Splitting: The standard remedy, Sample Splitting (using one part of data for selection and another for evaluation), ensures validity but drastically reduces the sample size available for estimation, leading to wide, uninformative confidence bands.

2. Methodology: PS-DME

The authors propose Post-Selection Distributional Model Evaluation (PS-DME), a framework that enables full data reuse while maintaining rigorous statistical validity.

Core Concept

PS-DME constructs confidence bands for the Cumulative Distribution Function (CDF) of the KPI for pre-selected models, guaranteeing control over the False Coverage Rate (FCR).

• FCR Definition: The expected fraction of selected models for which the confidence band fails to cover the true CDF. The goal is to ensure $FCR \leq \delta$.
• The Mechanism (E-Values):
  • Unlike traditional p-values, e-values are statistics that allow for valid inference even when the significance level is chosen after seeing the data (post-hoc selection).
  • PS-DME utilizes an e-calibrator function $f(\cdot)$ to convert p-values derived from the Kolmogorov-Smirnov statistic into e-values.
  • By inverting the e-value test, the method constructs confidence bands $[L_k(x), U_k(x)]$ for the CDF $F_k(x)$ of each selected model $k$.

The Algorithm

1. Pre-Selection: An arbitrary, data-dependent rule $S(\cdot)$ selects a subset of models $\mathcal{K}$ from $K$ candidates using the full dataset $D$.
2. In-Sample Inference: Using the entire dataset $D$ (not a split), PS-DME computes the empirical CDF $\hat{F}_k(x)$.
3. Band Construction: It constructs uniform confidence bands:
   $$L_k(x) = \max\{0, \hat{F}_k(x) - \varepsilon_k\}$$
   $$U_k(x) = \min\{1, \hat{F}_k(x) + \varepsilon_k\}$$
   Where the bandwidth $\varepsilon_k$ is calculated using the e-calibrator to account for the selection bias:
   $$\varepsilon_k = \sqrt{\frac{1}{2n_k} \log\left( \frac{2}{f^{-1}\left(\frac{K}{\delta|\mathcal{K}|}\right)} \right)}$$
   Here, $f^{-1}$ is the generalized inverse of the e-calibrator, and $|\mathcal{K}|$ is the number of selected models.

Alternative Implementation

The paper also discusses a variant using Berk-Jones (BJ) statistics instead of the DKW inequality. BJ bands are input-dependent (tighter in tails, wider in the center) and can offer better adaptivity for specific distribution shapes, though they require more complex computation.

3. Key Contributions

1. Novel Framework: Introduction of PS-DME, the first general framework for statistically valid distributional assessment after arbitrary data-dependent model pre-selection.
2. Full Data Reuse: Demonstrates that using the entire dataset for both selection and inference is possible without sacrificing validity, provided e-values are used.
3. Theoretical Guarantees: Proves that PS-DME controls the FCR at level $\delta$ for any selection rule.
4. Efficiency Analysis: Provides a theoretical comparison showing that PS-DME yields narrower confidence bands than Sample Splitting (SS-DME) when the sample size is limited or the selection set is small, as the gain from using the full sample outweighs the cost of the post-selection correction factor.
5. Practical Validation: Extensive experiments across synthetic data, LLM decoding, and telecom networks.

4. Experimental Results

The authors validate PS-DME across three distinct domains:

• Synthetic Data (Linear Regression):

  • Setup: 2,000 ridge regression models pre-selected based on mean squared error.
  • Result: PS-DME successfully controlled the FCR (unlike the naive in-sample method). Crucially, PS-DME produced significantly narrower confidence bands than Sample Splitting, especially in low-sample regimes ($n=20$), allowing for the selection of models with better guaranteed performance.

• LLM Decoding (Text-to-SQL):

  • Setup: 43 configurations of the Qwen2.5-7B model (varying temperature, top-p, and decoding strategies like Best-of-N and Iterative Refinement) evaluated on the Spider benchmark.
  • Result: PS-DME consistently provided a lower (better) "guaranteed KPI" (loss) across various reliability targets compared to SS-DME. This demonstrates the ability to reliably select high-quality decoding strategies without overfitting to the validation set.

• Telecom Network (O-RAN):

  • Setup: Evaluation of 18 network configurations for buffer occupancy in a 5G-like environment.
  • Result: PS-DME provided tighter bounds on the CDF of buffer occupancy. In low-data regimes, SS-DME failed to provide useful guarantees due to data scarcity, whereas PS-DME maintained tight, valid bounds.

• Server Latency (HTTP Logs):

  • Setup: Analysis of response time distributions for different request paths.
  • Result: Demonstrated the sensitivity of the confidence band width to the choice of the e-calibrator parameter $\tau$, showing that intermediate values often yield the tightest bands.

5. Significance and Impact

• Paradigm Shift: Moves model evaluation from "pass/fail" against a fixed threshold to a distributional exploration of performance-reliability trade-offs.
• Resource Efficiency: Solves the "data scarcity" problem in model selection. In scenarios where data is expensive or limited (e.g., real-world telecom logs, LLM inference), PS-DME extracts more value from every data point compared to sample splitting.
• Reliability: Enables engineers to make deployment decisions with statistical certainty regarding tail risks (e.g., latency outliers), which is critical for safety-critical systems and Service Level Agreements (SLAs).
• Generality: The framework is agnostic to the specific selection rule or the nature of the KPI, making it applicable to a wide range of machine learning and engineering systems.

In summary, PS-DME offers a mathematically rigorous solution to the "double-dipping" problem in model evaluation, enabling practitioners to safely and efficiently explore the full landscape of model performance and reliability.