Genetic Generalized Additive Models

1. Problem Statement

As machine learning systems are deployed in high-stakes domains (e.g., housing valuation, autonomous weapons), there is an urgent need for models that are not only accurate but inherently interpretable. While Generalized Additive Models (GAMs) offer a balance between predictive accuracy and interpretability by decomposing predictions into a sum of univariate functions ($y = \beta_0 + \sum f_j(x_j)$), manually configuring their structure is challenging.

The core problem lies in selecting the appropriate structure and hyperparameters (e.g., linear vs. spline terms, number of basis functions, regularization constants) to control:

• Approximation capabilities: How well the model fits the data.
• Generalization: How well it performs on unseen data.
• Interpretability: The cognitive load required to understand the model.

Current approaches often rely on post-hoc explanation methods, which the authors argue are unreliable. Instead, the paper proposes automating the structural optimization of GAMs to inherently balance predictive accuracy with sparsity, smoothness, and uncertainty, thereby creating models that are transparent and trustworthy without sacrificing performance.

2. Methodology

The authors propose a framework using the multi-objective genetic algorithm NSGA-II (Non-dominated Sorting Genetic Algorithm II) to automatically optimize GAM structures.

2.1 Chromosome Encoding

Each candidate GAM is encoded as a chromosome where genes represent structural hyperparameters for each feature:

• Component Type: None (inactive), Linear, or Spline.
• Spline Parameters: If a spline is chosen, the number of knots (basis functions) and the regularization constant ($\lambda$).
• Scaling: A boolean flag indicating whether to scale the feature.

The algorithm employs smart initialization (Algorithm 1) to sample these parameters with specific probabilities (e.g., 50% chance of spline, 30% linear, 20% none) and runs for 50 generations with a population size of 80.

2.2 Multi-Objective Optimization

NSGA-II optimizes two conflicting objectives simultaneously:

1. Prediction Error (RMSE): Minimized to ensure high accuracy.
2. Complexity Penalty ($C$): A composite metric defined as $C = 0.70 U + 0.30 S$, where:
   • Uncertainty ($U$): The mean width of the 95% confidence intervals of active terms, normalized by the target range. Narrower intervals indicate higher certainty.
   • Sparsity ($S$): The proportion of active terms relative to total features. This encourages the model to ignore irrelevant features.

By minimizing both, the algorithm searches for a Pareto Front—a set of non-dominated solutions representing the best possible trade-offs between accuracy and complexity.

2.3 Experimental Setup

• Dataset: California Housing dataset (8 features, median house value as target).
• Baselines:
  • LinearGAM: A standard GAM where all features use spline functions with a maximum of 25 splines.
  • Decision Tree: Used as a high-variance upper bound.
• Evaluation: 5-fold cross-validation across multiple random seeds. Three representative models are extracted from the Pareto front: the Knee Solution (best trade-off), the RMSE-Optimal, and the Minimal Complexity model.

3. Key Contributions

1. Automated Structural Optimization: Introduced a novel application of NSGA-II to automatically determine the functional form (linear vs. spline) and hyperparameters of GAMs, removing the need for manual tuning.
2. Integrated Complexity Metric: Defined a specific complexity penalty that jointly optimizes for sparsity (feature selection) and uncertainty (confidence interval width), ensuring the resulting models are not just simple, but also "honest" about their predictive certainty.
3. Pareto-Based Model Selection: Demonstrated that evolutionary algorithms can provide a spectrum of solutions rather than a single "best" model, allowing practitioners to choose models based on specific regulatory or operational constraints (e.g., prioritizing interpretability over marginal accuracy gains).
4. Enhanced Interpretability: Showed that multi-objective optimization leads to models that are structurally simpler (fewer active terms) and exhibit wider, more honest confidence intervals in extrapolation regions compared to standard baselines.

4. Results

Experiments were conducted across five random seeds (42, 7, 123, 225, 729).

• Accuracy vs. Complexity Trade-off:
  • The RMSE-optimal GA models consistently outperformed the baseline LinearGAM and Decision Trees in terms of raw prediction error (e.g., Seed 225: GA RMSE 0.6228 vs. LinearGAM 0.6536).
  • Crucially, even the RMSE-optimal GA models were significantly simpler (lower complexity penalty) than the baseline LinearGAM. For instance, in Seed 42, the GA model achieved the lowest RMSE (0.6821) with a complexity penalty of 0.2226, whereas the baseline LinearGAM had a much higher penalty (0.4096) despite worse accuracy.
• The "Knee" Solution: The models selected at the "knee" of the Pareto front offered a balanced performance, often matching or slightly underperforming the RMSE-optimal model but achieving substantially lower complexity (e.g., Seed 42: Penalty 0.1020 vs. Baseline 0.4096).
• Interpretability Gains:
  • Sparsity: The GA models frequently set irrelevant features (like HouseAge in Seed 7) to "Inactive," removing the need to interpret complex, noise-fitting splines found in baseline models.
  • Honest Uncertainty: In features where the GA simplified a function to linear, the resulting confidence intervals were wider at the extremes. The authors argue this is a feature, not a bug, as it signals uncertainty in sparse data regions, preventing the "false precision" often seen in tightly fitted baseline models.

5. Significance and Conclusion

This work demonstrates that interpretability and accuracy are not mutually exclusive when guided by a multi-objective optimization framework. By using NSGA-II to optimize GAM structures, the authors achieved:

• Higher Performance: Models that are more accurate than standard baselines.
• Lower Cognitive Load: Models that are sparser and smoother.
• Trustworthy AI: Models that provide honest uncertainty estimates, crucial for high-stakes decision-making.

The framework provides a generalizable approach for creating transparent AI systems. The authors suggest future work extending this methodology to fuzzy inference systems and cascading rule-based systems, as well as optimizing computational costs through caching strategies. The code is open-source, facilitating reproducibility and further research in trustworthy AI.