🤖 machine learning

Non-linear PCA via Evolution Strategies: a Novel Objective Function

This paper proposes a novel non-linear PCA framework that utilizes Evolution Strategies to optimize neural network-based variable transformations with a granular objective function, thereby achieving superior dimensionality reduction performance while preserving interpretability and natively handling categorical data.

Original authors: Thomas Uriot, Elise Chung

Published 2026-02-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Thomas Uriot, Elise Chung

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Straight Line" Limit

Imagine you have a giant box of mixed-up toys (data). You want to organize them into a few neat piles so you can see the main patterns.

Standard PCA (Principal Component Analysis) is like a strict librarian who only allows you to sort toys by drawing straight lines. If the toys are arranged in a circle, a spiral, or a complex 3D shape, the librarian can't see the pattern. They just see "mess" because they are forced to draw straight lines through it.

Kernel PCA (kPCA) is a smarter librarian who can draw curved lines. But there's a catch: they draw the lines in a secret, invisible dimension. You can't see why they sorted the toys the way they did, and it's very hard to explain their logic to anyone else. Also, they get confused if you give them toys with labels like "Red," "Blue," or "Large" (categorical data) because they don't know how to measure the distance between a "Red" toy and a "Blue" toy.

The Solution: A "Shape-Shifting" Librarian

The authors propose a new method that acts like a shape-shifting librarian.

The Transformation (The Neural Network): Before sorting, this librarian can magically reshape every single toy individually. A round ball might get squished into a cube; a long stick might get bent into a curve. They do this using Neural Networks (computer programs that learn patterns).
The Goal: The goal is to reshape the toys so that when the librarian finally draws their straight lines (Standard PCA), the lines capture the most important patterns possible.

The Secret Sauce: "Evolution Strategies"

Here is the tricky part: The librarian can't use a standard calculator to figure out the best way to reshape the toys because the math is too messy (it's "non-differentiable").

Instead, they use Evolution Strategies, which works like natural selection:

Imagine you have a population of 50 librarians, each trying to reshape the toys slightly differently.
You test them all. The ones who do the best job of organizing the toys get to "reproduce."
Their "children" (new librarians) inherit their reshaping tricks but with tiny random tweaks.
You repeat this process over and over. Eventually, you evolve a librarian who is a master at reshaping the toys perfectly for sorting.

The Big Innovation: The "Granular" Scorecard

The paper introduces a new way to grade the librarians.

The Old Way (Global Objective): You give the whole team one single grade based on how well the entire box is sorted. It's like saying, "Good job, team!" but you don't know which librarian actually did the heavy lifting.
The New Way (Partial/Granular Objective): The authors created a scorecard that grades each toy's transformation individually. They ask: "How much did this specific reshaping help the overall sorting?"
- Analogy: Imagine a sports team. The old method just looks at the final score. The new method looks at how many points each player contributed.
- Result: This gives a much stronger signal to the "evolution" process. It tells the librarians exactly which reshaping tricks are working and which aren't, leading to much faster and better results, especially when you have many different types of toys (high-dimensional data).

Handling "Categorical" Toys

One of the biggest headaches in data science is dealing with categories (like "Yes/No," "Small/Medium/Large," or "Dog/Cat").

The Old Problem: Usually, you have to turn "Dog" into a long list of zeros and ones (one-hot encoding). If you have 1,000 different breeds, your box suddenly has 1,000 columns. This explodes the size of the data and breaks the sorting machine.
The New Solution: This method treats "Dog" as a single concept. It learns that "Golden Retriever" and "Labrador" are close to each other, while "Snake" is far away. It keeps the data compact and handles mixed types (numbers, categories, and rankings) all at once without exploding the size of the box.

The Results

The authors tested this on:

Fake Data: Shapes like nested circles and spheres where the patterns are clearly non-linear.
Real Data: Medical records, credit checks, and wine datasets.

The Findings:

Their method captured more information (explained more variance) than both the old straight-line method (PCA) and the secret-curved-line method (kPCA).
The "Granular Scorecard" (the new objective function) worked significantly better than the old "Team Grade" method.
The results remained interpretable. Because they only reshaped one toy at a time, you can still look at the final map and say, "Ah, the 'Red' toys are close to the 'Round' toys because of this specific transformation." You can use standard tools (like biplots) to visualize the results, which you can't easily do with Kernel PCA.

Summary

The paper presents a new way to simplify complex data. Instead of forcing data into straight lines or hiding it in secret dimensions, it uses an evolutionary process to "reshape" the data first. By grading each reshaping move individually, they get a much smarter, more efficient, and easier-to-understand result that handles all types of data (numbers and categories) without getting overwhelmed.

Technical Summary: Non-linear PCA via Evolution Strategies

Problem Statement

Principal Component Analysis (PCA) is a dominant dimensionality reduction technique valued for its interpretability and deterministic nature. However, its reliance on linear transformations limits its ability to capture complex, non-linear structures inherent in real-world data. Existing non-linear extensions face significant trade-offs:

Kernel PCA (kPCA): While it captures non-linearity, it sacrifices interpretability (principal components do not map directly to original variables) and struggles with hyperparameter selection.
Auto-encoders: These lack the intrinsic variance maximization and orthogonality properties of PCA, and their latent variables are often difficult to interpret.
Handling Categorical Data: Standard PCA and many NLPCA methods fail to natively handle categorical or ordinal variables. Converting these to one-hot encodings causes a "dimensional explosion," where categorical levels dominate the variance contribution, skewing the analysis. Furthermore, existing methods like Multiple Factor Analysis (MFA) require one-hot encoding, leading to instability in datasets with many unique levels.

The authors identify a gap for a framework that unifies the interpretability of PCA with the flexibility of neural networks, capable of handling mixed data types without dimensional explosion.

Methodology

The proposed framework, Non-linear PCA via Evolution Strategies (ES), consists of three core components:

1. Parameterization of Transformations

Instead of learning a global non-linear mapping, the method parametrizes a separate transformation function $\Phi_l$ for each original variable $X^{(l)}$ .

Architecture: Each $\Phi_l$ is implemented as a simple feed-forward neural network.
Categorical Handling: For categorical variables, the input is one-hot encoded, but the network outputs a single scalar value. This ensures that each original variable (whether numerical or categorical) maps to exactly one transformed dimension, preventing the dimensional explosion associated with standard one-hot encoding in PCA or MFA.
Interpretability: The one-to-one mapping allows the resulting components to be traced back to specific original variables.

2. Optimization via Evolution Strategies

The objective function involves eigendecomposition of the covariance matrix, which is non-differentiable. Consequently, standard backpropagation cannot be used.

Algorithm: The authors employ Evolution Strategies (ES), a derivative-free black-box optimization method.
Process: The parameters of the neural networks are perturbed with Gaussian noise. The objective value is computed for each perturbation, and the parameters are updated based on a gradient approximation derived from the weighted noise samples. This bypasses the differentiability constraints of standard eigendecomposition.

3. Novel Objective Functions

The paper introduces two objective functions to maximize the explained variance of the first $k$ principal components:

Global Objective ( $F^k_{global}$ ): Maximizes the sum of variances of the first $k$ components. This provides a single scalar signal for the entire system.
Partial (Granular) Objective ( $F^k_l$ ): A novel contribution that decomposes the global variance to maximize the individual variance contribution of each transformed variable.
- Based on Proposition 3.1, the variance contribution $c_{j,l}$ of variable $l$ to eigenvalue $j$ is calculated.
- The objective maximizes the sum of these individual contributions for each variable separately.
- Scaling: To ensure fair comparison with the global objective, the global objective is scaled down by the number of variables $p$ to match the magnitude of the learning step in ES.

Key Contributions

Non-Linear Extension via Neuroevolution: The paper proposes a robust NLPCA framework where variable transformations are parameterized by neural networks and optimized using ES. This allows for arbitrary complex mappings while avoiding the differentiability issues of standard eigendecomposition.
Granular Variance Maximization: The introduction of the partial objective function is a primary contribution. By isolating and maximizing the specific variance contribution of each variable, the method provides a stronger, more robust learning signal than global variance maximization, particularly in higher dimensions.
Unified Data Handling: The method natively handles numerical, categorical, and ordinal variables simultaneously. By mapping each categorical variable to a single transformed column, it avoids the dimensional explosion of one-hot encoding, bridging the gap between Kernel PCA and traditional component-based methods like MFA.

Experimental Results

The method was evaluated on three synthetic datasets (nested circles, spheres, alternate stripes) and five real-world datasets (credit, ionosphere, breast cancer, heart, wine) from OpenML.

Explained Variance: The proposed method (ES-PCA) consistently outperformed linear PCA across all datasets.
Comparison with kPCA: The ES-PCA method, particularly with the partial objective (ES-Partial), significantly outperformed Kernel PCA (kPCA) on real-world datasets for $k=2$ $k = 2$ (cumulative variance of the first two components).
- Note: The kPCA comparison was favorable to kPCA as the best result out of four kernels (rbf, cosine, polynomial, sigmoid) was reported without fine-tuning, whereas ES-PCA used default parameters.
Objective Performance: The partial objective consistently yielded higher explained variance and faster convergence compared to the global objective. The performance gain of the partial objective over the global objective correlated strongly with dataset dimensionality ( $\rho=0.87$ for $k=1$ and $\rho=0.98$ for $k=2$ ).
Interpretability: Qualitative results on synthetic datasets demonstrated that the learned transformations were intuitively interpretable (e.g., capturing radius differences in spheres or periodicity in stripes). The method preserved the ability to use standard PCA visualization tools like biplots.

Significance and Claims

The paper claims to address the "gap" in NLPCA research identified in previous reviews: the need for a general framework with parameterized transformations that maintains interpretability.

Interpretability: Unlike auto-encoders or kPCA, this method preserves the one-to-one mapping between original and transformed variables, allowing analysts to use standard tools (biplots) to visualize feature contributions and non-linear relationships.
Robustness to Data Types: By handling categorical variables without one-hot encoding, the method offers a more stable and efficient approach for real-world, non-curated datasets where categorical levels may be numerous or inconsistent.
Learning Signal: The authors assert that the granular partial objective provides a superior learning signal, enabling the model to learn more effectively in higher-dimensional spaces where global objectives may struggle to provide specific guidance for individual transformations.

The authors remain modest regarding the current state of the work, noting that the explained variance is still increasing with the number of generations, suggesting further gains are possible. They also acknowledge that while the method handles outliers better than standard PCA in some contexts, robust PCA techniques could be integrated in future work.