RBF Kernel Parameter Formula for Data Classification Methods

1. Problem Statement

Radial Basis Function (RBF) kernels are fundamental to many machine learning algorithms, particularly Support Vector Machines (SVM) and kernel-based subspace methods like Kernel Optimization Subspaces (KOS). The performance of these methods is critically dependent on the selection of the kernel hyperparameter, denoted as $\gamma$ (or $\sigma$ in the Gaussian formulation).

• Current Limitations: In practice, $\gamma$ is typically determined via computationally expensive procedures such as cross-validation or grid search.
• Drawbacks: These standard tuning methods suffer from:
  • High computational overhead, especially for large-scale or multi-class datasets.
  • Lack of robustness.
  • The requirement to split data into training and validation sets, reducing the amount of data available for actual model training.
  • Sensitivity to improper parameter selection, which can severely degrade generalization performance.

The paper addresses the need for a computationally efficient, analytical formula to directly estimate the optimal RBF parameter without exhaustive search.

2. Methodology

The author proposes a Dual Min-Max (DMM) analytical formula derived from geometric considerations in the induced feature space. The core objective is to find a mapping $\Phi$ that simultaneously:

1. Minimizes intra-class compactness: Reducing the diameter of mapped classes.
2. Maximizes inter-class separability: Increasing the distance between different classes.

Mathematical Derivation

The derivation relies on the properties of the RBF kernel $K(X, Y) = e^{-\gamma \|X-Y\|^2}$:

1. Feature Space Diameter ($D_k$): The squared diameter of a class $C_k$ in the feature space is derived as:
   $$D_k^2(\gamma) = 2 - 2e^{-\gamma D_k^2}$$
   where $D_k$ is the maximum distance between points in the original attribute space for that class.

2. Feature Space Inter-class Distance ($d_{l,k}$): The squared minimum distance between two classes $C_l$ and $C_k$ in the feature space is:
   $$d_{l,k}^2(\gamma) = 2 - 2e^{-\gamma d_{l,k}^2}$$
   where $d_{l,k}$ is the minimum distance between points of different classes in the original space.

3. Optimization Objective: The goal is to minimize the maximum class diameter ($D_{max}$) and maximize the minimum inter-class distance ($d_{min}$). This is formulated as minimizing two functions:

   • $G_1(\gamma) = \gamma D_{max}^2$
   • $G_2(\gamma) = \frac{1}{\gamma d_{min}^2}$

4. Weighted Optimization: A weighted function $G(\gamma) = \lambda G_1(\gamma) + (1-\lambda)G_2(\gamma)$ is constructed. By setting the derivative to zero and finding the Pareto optimum (balanced weighting where $\lambda = 0.5$), the author derives the closed-form solution:
   $$\gamma = \frac{1}{D_{max} d_{min}}$$

5. Robustness Adjustment: To prevent instability when $d_{min}$ approaches zero (due to outliers or mislabeled data), the formula is relaxed to use the average inter-class distance ($d_{avg}$) instead of the minimum:
   $$\gamma = \frac{1}{D_{max} d_{avg}}$$
   (Where $d_{avg}$ is the root-mean-square of all pairwise inter-class distances).

3. Key Contributions

• Analytical Closed-Form Solution: The paper provides a direct mathematical formula to calculate $\gamma$, eliminating the need for iterative grid searches or cross-validation.
• Geometric Justification: The formula is rigorously derived by optimizing the geometric properties (diameter and separation) of data in the high-dimensional feature space induced by the RBF kernel.
• Computational Efficiency: The method drastically reduces the time required to tune the kernel parameter, as it involves only calculating distances in the original input space.
• Data Utilization: Unlike cross-validation, this method utilizes the entire dataset for parameter estimation, potentially improving model robustness.

4. Results and Validation

The proposed DMM formula was validated on real-world datasets from the LIBSVM and OpenML repositories using two classifiers: SVM and KOS (Kernel Optimization Subspaces).

• Accuracy and Precision:
  • The results (Tables 1 and 2) show that the DMM formula achieves performance comparable to the standard learning process (grid search/cross-validation) across various datasets (e.g., Leukemia, USPS, Letter, Shuttle).
  • In several cases (e.g., DNA, BreastCancer), the DMM formula slightly outperformed the standard learning process in terms of accuracy and precision.
• Computational Time:
  • Table 3 demonstrates a significant reduction in processing time.
  • For SVM, time savings ranged from seconds to over an hour (e.g., the "USPS" dataset reduced from ~81 minutes to ~2.5 minutes; "Shuttle" from ~62 minutes to ~2 minutes).
  • For KOS, the reduction was even more dramatic, often dropping from seconds to milliseconds.
• Scalability: The formula using the average inter-class distance ($d_{avg}$) was found to be slightly more effective for datasets with a large number of classes compared to using the strict minimum distance.

5. Significance

This work offers a practical and theoretically grounded alternative to the standard "brute-force" hyperparameter tuning in kernel methods.

• Efficiency: It makes RBF-based classification feasible for large-scale problems where grid search is computationally prohibitive.
• Simplicity: The formula is easy to implement and requires no additional software libraries beyond basic distance calculations.
• Robustness: By avoiding data splitting for validation, it maximizes the utility of available training data.

In conclusion, Remaki demonstrates that a simple geometric analysis of the input data distribution can yield an optimal RBF parameter that rivals complex learning-based tuning strategies while offering orders-of-magnitude improvements in speed.