Technical Summary: Twin: Tuning Learning Rate and Weight Decay of Deep Homogeneous Classifiers without Validation

1. Problem Definition

Deep learning models require careful selection of hyper-parameters (HPs), particularly the learning rate (LR) and weight decay (WD), to achieve optimal performance. Traditional hyper-parameter optimization (HPO) pipelines rely on a held-out validation set to evaluate candidate configurations. This approach presents two significant challenges:

1. Data Inefficiency: In data-scarce domains (e.g., medical imaging) or small-data regimes, reserving 10–30% of data for validation significantly reduces the training set size, leading to noisy and unreliable HP estimates.
2. Process Complexity: When no validation set is available, practitioners often resort to a cumbersome two-step process: splitting the training data, tuning HPs on the subset, and then retraining the final model on the full dataset. This is computationally expensive and suboptimal.

Existing validation-free methods, such as early-stopping techniques or generalization-gap predictors, often fail to provide reliable cross-trial ranking metrics or require expensive meta-training. The paper addresses the specific problem of selecting optimal LR and WD configurations using only the training set, eliminating the need for validation data or a two-step retraining process.

2. Methodology: The Twin Pipeline

The proposed method, Twin (Tune without Validation), leverages the theoretical properties of homogeneous classifiers and empirical scaling laws to derive a regime-dependent selection rule.

2.1 Theoretical Foundations

Twin is built upon two complementary insights:

1. Margin Maximization in Homogeneous Classifiers: Citing Lyu & Li (2019), the authors utilize the result that optimizing cross-entropy loss on homogeneous networks via gradient descent leads to a monotonic increase in the smoothed normalized margin once the training loss drops below a threshold (indicating separability).
2. Train-Test Loss Scaling Law: The authors propose an empirical parametric model linking training loss ($L_{train}$) and test loss ($L_{test}$):
   $$L_{test} = B L_{train}^\beta + V L_{train}^{-v} + E$$
   This equation captures three regimes:
   • Divergence/Lazy Learning: Dominated by the decreasing term ($B L_{train}^\beta$), where lower training loss correlates with better generalization.
   • Separable Regime: Dominated by the increasing term ($V L_{train}^{-v}$), where further reduction in training loss (overfitting) leads to increased test loss.

2.2 Regime-Dependent Selection Rule

The core innovation of Twin is a decision rule that switches the selection metric based on the optimization regime, determined by training accuracy:

• Non-Separable Regime (Low Training Accuracy): If no configuration reaches the separability threshold (e.g., 99% training accuracy), the training loss is monotonic with test loss. Twin selects the configuration with the minimum training loss.
• Separable Regime (High Training Accuracy): If configurations reach separability, the margin maximization dynamics induce a monotonic relationship between the parameters' norm ($\rho = \|\theta\|$) and test loss. In this regime, smaller norms indicate better generalization. Twin selects the configuration with the minimum parameter norm among those that are separable.

2.3 Practical Implementation

Twin operates as a validation-free HPO pipeline:

1. Sampler: Performs a logarithmic grid search over LR and WD ranges.
2. Scheduler: Supports early stopping (e.g., modified HyperBand) to reduce computational cost, evaluating trials based on average training loss over a few epochs.
3. Selector: Computes training loss, parameter norms, and training accuracy for all trials. It applies the regime-dependent rule (Algorithm 1) to select the final $(\alpha^*, \lambda^*)$ without ever accessing a validation or test set.

3. Key Contributions

1. Validation-Free Generalization Predictor: The paper introduces a theoretical link between margin maximization in homogeneous classifiers and an empirical train-test loss scaling law. This reveals distinct regimes where either training loss or parameter norms serve as monotonic signals for generalization.
2. The Twin Pipeline: A simple, model-agnostic algorithm that optimizes LR and WD without validation sets. It avoids the two-step tuning process and the need to withhold data.
3. Empirical Validation: A comprehensive study across 37 dataset-architecture configurations (including ConvNets, Vision Transformers, and Feedforward networks) spanning image classification, medical imaging, and small-data regimes.
4. Robustness Analysis: Demonstrations that Twin remains effective under varying grid sparsity, different optimizers (SGD, Adam, AdamW), and different learning rate schedulers.

4. Experimental Results

The authors evaluate Twin against several baselines: an Oracle (selecting HPs based on test accuracy), a Validation Set (SelVS) approach, and sharpness-based selection (m-sharpness).

• Performance vs. Oracle: Across 37 configurations, Twin achieves a Mean Absolute Error (MAE) of 1.28% compared to the Oracle baseline. This indicates Twin can closely approximate the performance of an ideal selector that has access to test data.
• Small Data Regimes: In scenarios with limited data (e.g., 30–80 samples per class), Twin outperforms SelVS. For instance, on a benchmark of small datasets, Twin achieved higher average balanced accuracy (e.g., 76.2% vs. 74.0% for ResNet-50) because SelVS suffers from noise due to small validation subsets.
• Medical Imaging: On the MedMNIST v2 benchmark, Twin matched the performance of SelVS (83.1% vs. 83.2% average accuracy) while eliminating the need to collect or label the ~10% of data typically reserved for validation, offering significant cost savings.
• Comparison to Sharpness: Twin outperformed worst-case m-sharpness selection in 6 out of 8 natural image scenarios, particularly on Vision Transformers where sharpness correlations are weaker.
• Sensitivity: Twin is robust to grid sparsity (performing well even with 60–80% of grid cells removed) and different separability thresholds (optimal at 99% rather than strict 100%).

5. Significance and Claims

The paper claims that Twin offers a simple and effective alternative to traditional HPO, particularly in scenarios where validation data is scarce, expensive, or difficult to collect. By exploiting the intrinsic dynamics of homogeneous classifiers, Twin removes the "cumber-some two-step process" and the need to hold out data.

The authors position Twin not as a universal solution for all hyper-parameters, but as a specialized, theoretically grounded method for tuning Learning Rate and Weight Decay. They emphasize that the method's reliance on margin maximization and scaling laws allows it to generalize across diverse architectures (from small MLPs to large ViTs) and domains (natural images to medical scans). The work aims to stimulate further research into validation-free generalization prediction driven by training dynamics.

Limitations Acknowledged:

• Currently limited to optimizing LR and WD.
• Relies on the assumption of network homogeneity (though empirically effective on standard deep networks).
• Requires the training set to be sufficiently large to allow for a grid search, though it mitigates the need for a separate validation set.