Statistical Hypothesis Testing for Information Value (IV)

Technical Summary: Non-Parametric Test for Information Value (IV)

Problem Statement
Information Value (IV) is a standard metric in credit scoring and binary classification for feature selection, often used in conjunction with Weight of Evidence (WoE) to assess the predictive power of variables. However, current industry practices rely on fixed empirical thresholds (e.g., IV > 0.1) to determine feature relevance. The authors critique these thresholds as "magic numbers" lacking theoretical justification. They argue that fixed thresholds fail to account for data-specific characteristics, particularly class imbalance, leading to unreliable feature selection. The paper identifies a gap in the literature: there is no formal statistical hypothesis test for IV, nor are there asymptotic guarantees for its behavior under varying data conditions.

Methodology
The authors propose a formal statistical framework by establishing a connection between IV and the Jeffreys divergence, a symmetric measure of divergence between two probability distributions.

1. Theoretical Formulation:

   • Let $X$ be a predictor and $Y$ be a binary target. The distributions of $X$ given $Y=1$ ($p$) and $Y=0$ ($q$) are defined over $r$ bins.
   • IV is mathematically equivalent to the Jeffreys divergence estimator $J(\hat{p}, \hat{q}) = \sum (\hat{p}_j - \hat{q}_j) \ln(\hat{p}_j / \hat{q}_j)$.
   • The hypothesis test is formulated as $H_0: p = q$ (no predictive power) versus $H_A: p \neq q$ (predictive power exists).

2. Asymptotic Properties:

   • The authors prove the Strong Law of Large Numbers (almost sure consistency) for the IV estimator.
   • They derive the Central Limit Theorem (asymptotic normality) for the estimator. Specifically, as sample sizes $n$ (positive class) and $m$ (negative class) approach infinity, the standardized test statistic converges to a standard normal distribution $N(0,1)$.
   • The variance terms $V_1(p,q)$ and $V_2(p,q)$ are derived to account for the variability in both classes.

3. The J-Divergence Test:

   • Based on the asymptotic normality, a non-parametric hypothesis test is constructed. The test statistic is normalized using estimated variances derived from empirical frequencies.
   • Decisions are made based on p-values and critical values from the standard normal distribution, eliminating the need for arbitrary fixed thresholds.

Key Contributions

• Formal Statistical Framework: This is the first work to present a formal statistical hypothesis test for Information Value, moving beyond heuristic rules.
• Non-Parametric Nature: The test does not assume specific shapes for the underlying empirical distributions, making it robust and model-agnostic.
• Robustness to Imbalance: Theoretical derivation and simulation results demonstrate that the test maintains validity and power even under severe class imbalance, a scenario where traditional IV thresholds often fail (producing high Type-I errors).
• Open-Source Implementation: The authors provide a Python library (statistical-iv) to facilitate reproducibility and adoption.

Experimental Results
The paper validates the methodology through simulation studies and a real-world application:

1. Simulation Studies:

   • Imbalance Sensitivity: In scenarios with high class imbalance (e.g., imbalance rate < 0.01), the traditional criterion (IV > 0.1) exhibits inflated Type-I error rates (false positives). In contrast, the J-Divergence test maintains robust performance across various imbalance levels.
   • Parameter Sensitivity: The test performs optimally with significance levels $\alpha \in [0.001\%, 0.1\%]$ and bin counts $r \in [2, 14]$.
   • Power: The empirical power function of the J-Divergence test consistently outperforms the fixed threshold criterion, particularly when the true divergence is small.

2. Fraud Detection Case Study:

   • Using a Kaggle e-commerce fraud dataset (highly imbalanced, ~0.3% fraud rate), the authors compared three feature selection strategies: no selection, IV > 0.1, and the J-Divergence test.
   • Feature Selection: The J-Divergence test selected 262 features, compared to 220 for the IV > 0.1 criterion. Notably, it identified 42 additional features that the traditional method rejected.
   • Model Performance: When used to train a LightGBM classifier, the J-Divergence test achieved the highest precision (0.88491) compared to the IV > 0.1 method (0.87141) and the unfiltered baseline (0.85496).
   • Efficiency: The proposed method reduced the feature set by over 100 features compared to the unfiltered baseline while improving precision by 3%, suggesting significant reductions in engineering and maintenance costs.

Significance and Claims
The paper claims that the J-Divergence test provides a rigorous, theoretically grounded alternative to the "magic numbers" currently used in industry. Its primary significance lies in its ability to handle class imbalance without inflating false positive rates, a critical requirement in domains like fraud detection and credit scoring.

The authors position the test as a filter-based method suitable for the pre-modeling phase. They explicitly state that it is not intended to replace model-based selection methods (like Lasso or tree-based importance) but to complement them by offering a statistically justified filtering step. The method is described as "model-agnostic," computationally efficient (amenable to parallelization), and interpretable via p-values.

Limitations Acknowledged
The authors note that the test requires features to be categorical or discretized into bins, and its reliability depends on the quality of this discretization. Additionally, the test relies on sufficient sample sizes per bin to satisfy asymptotic assumptions; performance may degrade in datasets with very low sample sizes or extreme sparsity. Finally, the current formulation is limited to binary classification, with multinomial extensions left for future work.