Technical Summary: Hypernetworks for Dynamic Feature Selection

Problem Definition

Dynamic Feature Selection (DFS), also known as active feature learning, addresses machine learning scenarios where features must be acquired sequentially under budget constraints. Unlike standard supervised learning where a fixed feature set is available, DFS requires a model to select a subset of features for each individual sample based on available information.

The core challenge lies in the exponential growth of possible feature acquisition paths ($2^M$ for $M$ features). Existing approaches generally rely on a single shared predictor trained to perform well on average across all possible observation statuses. The authors argue that this "amortization" approach forces a structural compromise: a single set of parameters cannot simultaneously be optimal for every possible feature subset. This leads to suboptimal performance on specific subsets, unreliable sequential decisions, and overfitting to small feature subsets, particularly when acquisition policies are biased. Furthermore, existing methods that condition a shared network on a binary mask (embedding methods) suffer from high structural complexity bounds that scale with the sum of input and task dimensions, rather than independently.

Methodology: Hyper-DFS

To address these limitations, the paper proposes Hyper-DFS, a framework that utilizes a hypernetwork to generate subset-specific classifier parameters on demand.

1. Hypernetwork Architecture

Instead of training a single shared predictor, Hyper-DFS employs a hypernetwork $g_\phi$ that takes the "knowledge status" (the set of observed features) as input and outputs the weights $\theta$ for a primary predictor network $f_\theta$. This allows the model to maintain a distinct, optimal parameter set for each feature subset without the memory overhead of storing $2^M$ separate models.

2. Smooth Feature Set Encoding (Set Transformer)

A critical innovation is the encoding of the feature subset. The authors argue that standard binary masks are poor conditioning signals because they treat all feature positions as interchangeable, failing to capture semantic relationships between features.

• Set Transformer Encoder: The method uses a Set Transformer (specifically Induced Set Attention Blocks) to encode the observed subset.
• Presence/Absence Embeddings: For each feature $i$, the model learns two distinct embeddings: $e^1_i$ (present) and $e^0_i$ (absent). The input to the encoder is constructed by summing these embeddings based on the binary mask.
• Continuous Conditioning Space: This approach maps discrete feature masks into a continuous representation where functionally similar subsets (those inducing similar optimal predictors) are geometrically close. This smoothness helps the hypernetwork generalize to unseen subsets by satisfying Lipschitz constraints, ensuring that small changes in the input mask do not result in drastic, unstable changes in the generated weights.

3. Training Stabilization Strategies

The authors identify and mitigate two specific training instabilities inherent to hypernetworks in the DFS setting:

• Variance Drift: The norm of generated weights can vary significantly with the cardinality of the subset. This is solved by using the presence/absence embedding scheme (which keeps the total number of tokens constant at $M$) combined with $L_2$ normalization of the conditioning vector.
• Gradient Dilution and Collapse: Mixing many different subsets in a single batch can cause gradient cancellation, and the compressor network (used to reduce input dimension for image data) is prone to representation collapse. The authors employ:
  • Pre-training with Restricted Masks: Initially, batches use a fixed, small number of knowledge status vectors (effectively a single subset per batch) to stabilize the primary network and compressor.
  • Linear Learning Rate Warm-up: Prevents the optimizer from taking large steps before the hypernetwork and compressor have co-adapted.
  • Auxiliary Losses: A scale regularizer ensures generated weights maintain appropriate variance, and a variance penalty prevents the encoder or hypernetwork from collapsing into constant maps.

Key Contributions

1. Structural Analysis: The paper demonstrates that existing mask-concatenation DFS approaches are instances of embedding methods with high structural complexity bounds ($\Omega(\epsilon^{-(m_1+m_2)})$), whereas hypernetworks offer a more efficient bound ($O(\epsilon^{-m_1/r})$) by decoupling the conditioning dimension.
2. Hyper-DFS Framework: Introduction of a hypernetwork-based DFS that generates subset-specific classifiers, utilizing Set Transformers to create a smooth, functional-aware conditioning space.
3. Training Solutions: Identification and resolution of variance drift and gradient instability in hypernetwork training for DFS via absence embeddings, normalization, and controlled batch masking.
4. Comprehensive Benchmarking: Evaluation across synthetic, real-world tabular, and image datasets, demonstrating superior performance and generalization.

Experimental Results

The authors evaluated Hyper-DFS against eight state-of-the-art DFS methods (including greedy, reinforcement learning, and multi-model approaches) across diverse datasets.

• Tabular Data: Hyper-DFS (and its ensemble variant, HyperEns) achieved the best average rank and highest mean AUAC-F1 on both synthetic and real-world tabular benchmarks (e.g., Diabetes, Metabric, MiniBooNE). It consistently outperformed strong single-model baselines like RePa and VIP.
• Image Data: On image patch benchmarks (MNIST, FashionMNIST, SVHN, Imagenette, PCam), Hyper-DFS variants achieved the highest scores on four of five datasets, outperforming dedicated multi-model approaches.
• Zero-Shot Generalization: A significant finding is Hyper-DFS's ability to generalize to feature subsets never seen during training. On held-out subsets, Hyper-DFS significantly outperformed single-model baselines (like VIP) and was competitive with or superior to multi-model baselines (like MoE). This suggests the smooth conditioning space effectively captures the relationship between feature subsets and optimal predictors.
• Efficiency: While training a hypernetwork is more expensive than a single network, it remains computationally feasible and scales better than storing hundreds of distinct classifiers.

Significance and Claims

The paper claims that per-subset specialization is a fundamental requirement for optimal DFS, which shared predictors cannot achieve. By using a hypernetwork, the proposed method bridges the gap between the ideal (but intractable) solution of having a unique predictor for every subset and the tractable but suboptimal shared predictor.

The authors emphasize that the smooth conditioning space created by the Set Transformer allows the model to generalize to unseen feature combinations, a critical capability for real-world deployment where acquisition budgets may vary or new feature combinations may arise. The results suggest that Hyper-DFS provides a robust middle ground between amortized single-model approaches and expensive pattern-by-pattern methods, offering substantial improvements in both predictive performance and zero-shot generalization across modalities.