Spectral Analysis of Molecular Features: When Richer Features Do Not Guarantee Better Generalization

This paper challenges the common heuristic that richer spectral features guarantee better generalization by demonstrating through a comprehensive spectral analysis of kernel ridge regression on molecular benchmarks that the relationship between spectral richness and performance is highly dependent on the specific representation and task, with simpler features like ECFP often outperforming complex transformer or 3D descriptors in low-data regimes.

The Gist

The Big Question: Does "More Information" Always Mean "Smarter Predictions"?

Imagine you are trying to teach a computer to guess the properties of a molecule (like how hot it gets or how much energy it holds). To do this, you have to describe the molecule to the computer using a "feature list."

In the world of machine learning, there is a popular belief (a "heuristic") that the more detailed and complex your feature list is, the better the computer will perform. It's like thinking that if you give a chef a recipe with 1,000 ingredients instead of 10, the dish will inevitably taste better.

This paper puts that belief to the test in the world of chemistry. The authors asked: If we look at the mathematical "spectrum" (the distribution of importance) of these feature lists, does a "richer" (more complex) spectrum always lead to better predictions?

The short answer is: No. Sometimes, having a "richer" list of features actually makes the model worse, or has no effect at all.

---

The Cast of Characters: How We Describe Molecules

The researchers tested four different ways to describe molecules, like four different dialects for describing a house:

1. ECFP (The Fingerprint): Think of this as a checklist of specific Lego bricks used to build the molecule. It's a hand-crafted, rule-based list.
2. Transformers (The AI Translator): These are pre-trained AI models (like a smart language model) that have read millions of chemical descriptions. They output a "latent feature" vector, which is like a summary sentence the AI wrote about the molecule.
3. Global 3D (The Whole House Photo): This describes the entire molecule as a single 3D shape, like a photograph of the whole house from the outside.
4. Local 3D (The Room-by-Room Tour): This describes the molecule by looking at the immediate neighborhood of every single atom, like a tour guide describing every room in the house individually.

The Experiment: Listening to the "Spectrum"

The authors didn't just look at the final score (how accurate the prediction was). They looked at the spectrum of the data.

The Analogy: Imagine the feature list is a symphony orchestra.

• Rich Spectrum: A full orchestra with many instruments playing different notes, creating a complex, layered sound.
• Poor Spectrum: A single violin playing one note.

The common belief was: "The more instruments (richer spectrum) you have, the better the music (prediction)."

The researchers analyzed the "music" of each molecular description method to see if the complexity of the sound matched the accuracy of the prediction.

The Surprising Results

The study found that the "richer is better" rule is not universal. It depends entirely on which dialect (representation) you are using.

1. The Fingerprint (ECFP): The Rule Holds

For the hand-crafted "Lego checklist" (ECFP), the old rule worked. The more complex the spectrum, the better the prediction.

• Analogy: If you have a detailed checklist of Lego bricks, having more specific details about the bricks helps you build the house correctly.

2. The AI Translators (Transformers): It's a Mixed Bag

For the AI-generated summaries, the relationship was messy. Sometimes a richer spectrum helped, sometimes it hurt, and often it didn't matter.

• Analogy: The AI translator might be giving you a very detailed, complex summary, but that complexity doesn't necessarily help you guess the house's temperature.

3. The 3D Descriptions: The Rule Flips!

This was the biggest surprise.

• Global 3D: Mixed results.
• Local 3D (Room-by-Room): Here, the rule reversed. The richer the spectrum (the more complex the description of every atom's neighborhood), the worse the prediction became.
• Analogy: Imagine trying to guess the house's temperature. If you have a "rich" description that lists the temperature of every single screw, nail, and dust particle in every room, the computer gets confused and makes a worse guess. It turns out, you only need a tiny bit of that information to get it right.

The "Truncation" Test: How Much Do We Actually Need?

To prove this, the researchers did a "Truncation Test." They asked: How much of the "orchestra" do we need to keep to get 95% of the correct answer?

• For Local 3D (Room-by-Room): They found that less than 2% of the information was needed. In some cases, they only needed 0.02% of the data to get a near-perfect prediction.
  • Metaphor: You don't need to listen to the whole symphony to know the song; you only need the first two notes. Adding the rest of the orchestra just creates noise.
• For Fingerprints and Transformers: These required much more of the "orchestra" (sometimes nearly the whole thing) to get the same level of accuracy.

The "Noise" Problem

Why does a "richer" spectrum sometimes hurt? The paper suggests that in complex representations (like the Local 3D ones), the extra "richness" often comes from noise or irrelevant details.

• The Analogy: If you are trying to find a specific person in a crowd, a "rich" description might include the color of their socks, the brand of their shoes, and the weather outside. This extra data doesn't help you find them; it just distracts you. The computer tries to learn from this noise, gets confused, and makes a mistake.

The Takeaway

The paper concludes that the popular idea in self-supervised learning—that "richer features always yield better generalization"—is false for molecular chemistry.

• Context Matters: A "rich" spectrum is great for some types of data (like Fingerprint lists) but can be harmful for others (like Local 3D descriptions).
• Less is More: For many 3D molecular descriptions, a very small, simple slice of the data is actually all you need. The "long tail" of complex, rich features often just adds noise that hurts performance.

In short: Don't assume that a more complex, information-heavy model is automatically smarter. Sometimes, the simplest, most focused description is the most accurate.

Technical Summary

Technical Summary: Spectral Analysis of Molecular Features

Problem Statement

Accurate molecular property prediction is critical for materials discovery, yet the relationship between the quality of molecular representations and model generalization remains poorly understood, particularly in low-data regimes where kernel methods excel. While the self-supervised learning (SSL) community has adopted a prevailing heuristic that "richer feature spectra yield better generalization," this principle has not been rigorously tested in the context of molecular chemistry. Existing evaluations of molecular representations (such as fingerprints, 3D descriptors, and pre-trained embeddings) rely almost exclusively on downstream test-set performance, obscuring fundamental questions about how well a kernel captures the intrinsic structure of the target function. This paper addresses the gap in understanding the spectral properties of molecular feature embeddings and their impact on generalization.

Methodology

The authors present the first comprehensive spectral analysis of Kernel Ridge Regression (KRR) across diverse molecular representations evaluated on the QM9 dataset and three MoleculeNet benchmarks (ESOL, FreeSolv, and Lipophilicity).

Representations Analyzed:
The study evaluates four categories of features:

1. ECFP-based Kernels: 13 hand-crafted kernels (e.g., Tanimoto, Dice, Sogenfrei) applied to Extended Connectivity Fingerprints.
2. Pre-trained Features: Embeddings from transformer-based models (SELFIESTED, SELFormer, ChemBERTa, MLT-BERT) and a GNN-based model (GROVER).
3. Global 3D Descriptors: Coulomb Matrix (CM), Bag of Bonds (BOB), and SLATM.
4. Local 3D Descriptors: SOAP, FCHL19, and ACSF.

Spectral Metrics:
To quantify spectral richness, the authors compute four metrics on the empirical covariance of the feature spaces:

• Polynomial decay rate ($\alpha$): A smaller $\alpha$ indicates a richer spectrum.
• Spectral Shannon Entropy (SSE): Higher values indicate richer spectra.
• Intrinsic Dimension (ID): Higher values indicate richer spectra.
• Stable Rank (SR): Higher values indicate richer spectra.

Experimental Design:

• Kernel Probing: The authors introduce "Kernel Probing" (KP), applying KRR to pre-trained features. This generalizes linear probing (LP), which is shown to be a special case of KP with a linear kernel.
• Ablation Studies: Feature ablation is performed by removing dimensions from embeddings or features from fingerprints to observe changes in the eigenvalue spectrum and downstream performance.
• Truncation Analysis: The authors employ Truncated Kernel Ridge Regression (TKRR) to determine the fraction of eigenvalues ($r/N$) required to recover 95% and 99% of the maximum predictive performance ($R^2$).

Key Contributions

1. Comprehensive Spectral Analysis: The paper provides the first systematic evaluation of the spectral properties of molecular kernels and SSL features, correlating them with predictive performance across multiple datasets.
2. Kernel Probing (KP): The authors propose and apply Kernel Probing, a method that utilizes KRR on SSL features, demonstrating improved performance over standard linear probing baselines.
3. Truncated Threshold Quantification: The study extends the concept of truncated kernels to ECFP-based representations, quantifying the minimal fraction of eigenvalues necessary to recover high performance, thereby challenging the necessity of long-tailed spectra for generalization.
4. Ablation Robustness: The work analyzes how spectral metrics respond to feature removal, distinguishing between the degradation of essential predictive features and the pruning of redundant noise.

Results

The central finding contradicts the common SSL heuristic that richer spectra inherently improve generalization. The correlation between spectral richness and performance is highly dependent on the representation type:

• ECFP Kernels: Show a strictly positive correlation between spectral richness and performance. Among these, the Sogenfrei kernel exhibits the best spectral metrics and second-lowest Mean Absolute Error (MAE), while the standard Tanimoto kernel (despite being the preferred cheminformatics kernel) retains more information in lower-ranked eigenvectors but does not always yield the best performance.
• Transformer-based Features: Exhibit mixed behavior. The spectral decay rate ($-\alpha$) shows a weak positive correlation, while SSE, ID, and SR show weak negative trends. No statistically significant relationship was found across the board.
• Global 3D Features: Show mixed behavior. The decay rate ($-\alpha$) correlates positively with performance, but other metrics (SSE, ID, SR) show weak negative correlations.
• Local 3D Features: Exhibit consistently negative correlations across all metrics. Increased spectral richness in local 3D representations does not improve accuracy and may be detrimental.

Truncation Findings:

• Local 3D Representations: For thermodynamic targets, fewer than 2% of eigenvalues (and occasionally as few as 0.02%) are sufficient to recover 95% of performance. This indicates that the predictive power is concentrated in the top eigenvalues, and the "tail" of the spectrum contributes little to generalization.
• ECFP and Transformer Features: These require significantly more eigenvalues to achieve similar recovery rates. For example, predicting the HOMO–LUMO gap often requires nearly the full spectrum for ECFP-based kernels.

Ablation Insights:

• ECFP-based kernels (Tanimoto, Dice) are robust to feature loss, maintaining stable spectra even with significant feature removal.
• Dense pre-trained embeddings (e.g., SELFIESTED) are highly robust; their spectra decay smoothly even after removing hundreds of dimensions.
• In specific cases (e.g., BOB with Laplacian kernel), increasing ablation reduced spectral metrics (SSE, ID, SR) while improving downstream MAE, suggesting that the metrics can detect the removal of redundant noise.

Significance and Claims

The paper claims to challenge the universal applicability of the "richer features yield better generalization" heuristic in molecular chemistry. The authors assert that:

• Spectral richness is not a universal predictor of downstream performance; its utility is contingent on the specific molecular representation and the target property.
• For local 3D representations, a "rich" spectrum (long tail) is not necessary for generalization and may even hinder it by facilitating overfitting to noise, a phenomenon analogous to the benefits of Tikhonov regularization or spectral truncation.
• The alignment between the geometry of the representation and the kernel function is more critical than the raw capacity (richness) of the embedding.
• The proposed Kernel Probing method offers a practical, improved baseline for evaluating SSL models in chemistry.

The study concludes that while spectral analysis provides critical insights into model behavior, the assumption that maximizing spectral richness is always beneficial is flawed in the context of molecular property prediction. This offers new guidance for selecting representations and kernels in label-limited scientific tasks.