📊 statistics

Hide&Seek: Learning to Explain in an End-to-End Differentiable Network

This paper introduces Hide&Seek, an end-to-end differentiable framework for instance-wise feature selection that jointly learns feature selection and prediction without information leakage by reformulating feature removal as a differentiable operation and stabilizing training through parsimony-weight annealing.

Original authors: Tal Ellinson, Hadi Mohasel Afshar, Sally Cripps

Published 2026-08-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tal Ellinson, Hadi Mohasel Afshar, Sally Cripps

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern world, computers are increasingly asked to make decisions that affect human lives, from approving loans to diagnosing illnesses. These systems, often called black-box models, are powerful because they can find complex patterns in vast amounts of data, but they are also opaque. They can tell us the answer, but they rarely explain how they arrived at it. To trust these systems, scientists and policymakers need to understand which specific pieces of information the computer relied on to make its decision. This is the field of feature selection: the art of identifying the most important variables in a dataset. While early methods looked for features that were important across an entire group of people, a more recent and necessary approach focuses on individual cases. A feature that matters for one patient might be irrelevant for another, just as a specific symptom might signal a heart attack in one person but be a sign of anxiety in another. The challenge has been to build a system that can dynamically decide, for every single individual, which facts are truly relevant without losing accuracy or tricking itself.

Researchers at the University of Technology Sydney have developed a new method called Hide&Seek to solve this problem. Their work addresses a subtle but critical flaw found in previous attempts to teach computers to explain themselves. In earlier systems, the computer would try to explain a decision by temporarily removing certain pieces of information from the input, much like covering up parts of a photograph to see if the image still makes sense. If the computer could still guess the outcome correctly after a piece of information was covered, that piece was deemed unimportant. However, the researchers discovered that these systems often learned to utilize the specific pattern of the "cover" itself. For instance, if a piece of data was always replaced with a zero when it was considered unimportant, the computer would learn that seeing a zero meant a specific type of situation, effectively using the absence of data as a secret code to make its prediction. This meant the system was not actually identifying the important features; it was just memorizing the rules of its own game.

To fix this, the Hide&Seek team redesigned how the computer handles missing information. Instead of replacing a removed feature with a fixed value like zero, they replace it with a random value drawn from the natural distribution of that data. Imagine a system trying to predict house prices. If it removes the "number of bedrooms" feature, previous methods might replace it with zero, which is an obvious signal that the data is missing. Hide&Seek, however, replaces it with a random number that looks like a plausible number of bedrooms found in real life. This makes it impossible for the computer to tell the difference between a real bedroom count and a replaced one, forcing it to rely only on the features it has chosen to keep. This approach prevents the system from exploiting the gaps in the data as a shortcut.

The researchers also introduced a training strategy that changes the computer's priorities over time. At the start of the learning process, the system is told to focus entirely on getting the prediction right, even if it uses many features. As training progresses, the system is gradually encouraged to use fewer features, learning to be more efficient and precise. This two-stage approach helps the computer avoid getting stuck in a bad solution where it relies on too many unnecessary details. The result is a model that is not only faster to train than its predecessors but also more accurate in identifying the true drivers of a decision.

When tested on a variety of scenarios, from synthetic data designed to trick the system to real-world credit card records and medical data, Hide&Seek consistently outperformed existing methods. In experiments involving "switch" features—variables that change the rules of the game depending on their value, such as temperature determining whether a store sells hot chocolate or iced tea—older methods failed to identify the switch as important about half the time. They were fooled by the pattern of the missing data. Hide&Seek, by contrast, correctly identified these critical switches nearly every time. The system also proved robust when dealing with highly correlated data, where many variables move together, a situation that often confuses other algorithms. In tests using images of handwritten digits, the method successfully highlighted the specific strokes that distinguished a three from an eight, whereas other methods produced scattered and inconsistent results.

The implications of this work extend beyond just making better algorithms. By ensuring that the explanations provided by these systems are genuine and not the result of hidden shortcuts, Hide&Seek offers a more reliable way to peer inside the black box. This is crucial for fields like medicine and finance, where understanding the "why" behind a decision is as important as the decision itself. The researchers demonstrated that by changing how information is hidden during the learning process, they could prevent the computer from learning to game the system. This allows for a more honest and transparent interaction between human decision-makers and the artificial intelligence tools they rely on, ensuring that the reasons given for a prediction are based on the actual data, not on the artifacts of the method used to analyze it.

Technical Summary: Hide&Seek: Learning to Explain in an End-to-End Differentiable Network

Problem Statement

Instance-wise feature selection (IWFS) is a critical tool for interpreting labeled data and explaining the predictions of black-box models. Unlike global feature selection, which identifies features important across an entire dataset, IWFS dynamically identifies important features for each specific instance (e.g., a specific patient or student).

Existing state-of-the-art methods (such as L2X, INVASE, and REAL-x) typically employ a selector-predictor framework. In this setup, a selector network identifies a subset of important features, and a predictor network uses this subset to make predictions. However, these pioneering methods face two primary challenges:

Information Leakage: In jointly trained models, when unselected features are replaced with a fixed ablation value (e.g., zero), the predictor may learn to exploit this specific ablation pattern as a signal rather than relying solely on the values of the selected features. This is particularly problematic for "switch features," where the partition induced by a feature (e.g., temperature being above or below a threshold) determines the response, rather than the exact value. The model can encode the partition via the ablation mask, leading to the misidentification of important features.
Differentiability and Training Efficiency: Discrete feature selection is non-differentiable. Methods like L2X use Gumbel-Softmax relaxations, which often require pre-specifying the number of features. Others, like INVASE, use reinforcement learning (actor-critic) to bypass differentiability, resulting in slow training times. REAL-x uses REBAR gradients but requires disjoint training of the predictor, which is not end-to-end differentiable and struggles with the exponential growth of possible feature subsets.

Methodology: Hide&Seek

The authors propose Hide&Seek, an end-to-end differentiable model for instance-wise feature selection that jointly trains a selector and a predictor under a single objective while preventing information leakage.

Core Architecture

The model consists of two feed-forward neural network modules:

Hide (Selector): Takes an input vector $x$ and outputs a continuous mask vector $m \in [0, 1]^d$ , representing the importance of each feature.
Seek (Predictor): Takes a modified input vector $z$ and outputs the predicted label $\hat{y}$ .

Key Innovations

Differentiable Feature Replacement: Instead of discretely removing features (which is non-differentiable), Hide&Seek constructs the input to the predictor as a linear combination of the original signal and a stochastic replacement value:
$z_j = m_j x_j + (1 - m_j) \hat{x}_j$
Here, $m_j$ is a continuous weight learned by the network. This allows for end-to-end differentiability without discrete sampling.
Preventing Information Leakage via Marginal Sampling: To prevent the predictor from exploiting the ablation pattern, Hide&Seek does not replace unselected features with a fixed value (like 0). Instead, it replaces them with random draws from the marginal distribution of the features ( $\hat{x} \sim \prod p(X_i)$ ). This ensures that the modified signal $z$ is distributed according to the original feature distribution, making it impossible for the predictor to distinguish between original and replaced features based on the replacement value itself.
Parsimony-Weight Annealing: To stabilize the joint training of the selector and predictor, the authors introduce an annealing schedule for the sparsity regularization parameter $\lambda$ . The weight on the parsimony term (encouraging fewer features) is quadratically increased over the training epochs ( $\lambda_t = (t/T)^2 \lambda_{max}$ ). This allows the model to prioritize predictive accuracy in early training stages and focus on feature sparsity in later stages, avoiding poor local minima.

Optimization

The model minimizes a loss function combining cross-entropy (prediction accuracy) and an $L_1$ norm penalty on the mask (parsimony):
$\ell = \mathbb{E}[-\log p_\theta(Y|Z)] + \lambda_t \frac{1}{d} \|m\|_1$
Theoretical analysis in the paper demonstrates that standard ablation-based loss functions can be ill-posed proxies for the intended objective, leading to feature misidentification. Hide&Seek's approach of marginal sampling theoretically prevents this leakage.

Key Contributions

Novel End-to-End Framework: Introduction of Hide&Seek, which jointly trains a selector and predictor in a fully differentiable manner, eliminating the need for discrete sampling or disjoint training phases.
Information Leakage Mitigation: A mechanism that replaces unselected features with random draws from the feature distribution rather than fixed values, effectively preventing the predictor from using ablation patterns as information channels.
Training Stabilization: The development of a parsimony-weight annealing schedule that significantly improves performance by balancing prediction accuracy and sparsity throughout the training process.
Theoretical Analysis: A formal proof showing that standard ablation-based objectives can lead to information leakage, specifically deriving a lower bound on the achievable misidentification rate for switch features.

Experimental Results

Hide&Seek was evaluated against eight benchmarks (including INVASE, REAL-x, L2X, SHAP, LIME, Random Forest, and LASSO) across five experimental settings:

Synthetic Data: On six synthetic datasets (including those with switch features), Hide&Seek achieved superior True Positive Rates (TPR) and lower False Discovery Rates (FDR) compared to all other models. It correctly identified switch features (where the partition matters) at a median accuracy of 100%, whereas competing methods like INVASE and L2X struggled (approx. 50-58% accuracy) due to information leakage.
Correlated Data: In settings with high multicollinearity ( $\rho$ up to 0.9), Hide&Seek consistently outperformed other models in feature selection metrics (F1 score) and maintained high switch accuracy (>99.4%).
Credit Default Data: Using a semi-synthetic setup on real-world credit data, Hide&Seek demonstrated robust feature selection capabilities and competitive predictive power (AUROC) compared to INVASE and REAL-x.
MNIST: In image classification, Hide&Seek successfully identified relevant pixel patches (e.g., the "left arcs" distinguishing 3s from 8s) more consistently than LIME, SHAP, and INVASE.
Breast Cancer Subtype Classification: Applied to TCGA microarray data, Hide&Seek identified biologically relevant genes (ESR1 and CCNB2) as the top features, aligning with established literature and outperforming other methods in ranking importance.

Efficiency: Hide&Seek is significantly faster to train than INVASE (approx. 1 minute vs. 1 hour 18 minutes on synthetic data) and comparable to L2X, despite having fewer parameters (approx. 3k vs. 100k for INVASE).

Significance and Claims

The paper claims that Hide&Seek represents a significant advancement in Explainable AI (XAI) by resolving the tension between joint training efficiency and the prevention of information leakage. The authors argue that by reformulating feature removal as a differentiable operation involving distributional replacement rather than discrete ablation, they preserve the benefits of joint selector-predictor training without introducing artificial information channels.

The work highlights that standard training objectives for ablation-based models can be ill-posed, leading to unintended behaviors where models omit genuinely important features to minimize loss. Hide&Seek provides a robust solution that maintains high predictive performance while ensuring that feature importance explanations are not artifacts of the training process. The authors position this as a step toward more transparent and reliable black-box model interpretation, particularly in high-stakes domains like healthcare and finance.