GoodRegressor: A Hierarchical Inductive Bias for Navigating High-Dimensional Compositional Space

The paper introduces GoodRegressor, a hierarchical symbolic regression framework that balances predictive performance and interpretability by using depth-controlled expansion to navigate vast compositional spaces, achieving state-of-the-art results in materials science while revealing system-specific optimal interaction depths.

The Gist

The Big Problem: The "Black Box" vs. The "Messy Kitchen"

Imagine you are trying to bake the perfect cake. You have a list of ingredients (flour, sugar, eggs, temperature, time).

• Old AI (The Black Box): You give a super-smart robot all your ingredients and the recipe for a good cake. The robot learns to bake amazing cakes, but when you ask, "Why did you add extra sugar?" it says, "I just know it works." It's a black box. You get a great result, but you don't understand the why.
• Simple Math (The White Box): You try to write a simple formula like Cake = Flour + Sugar. It's easy to understand, but it fails because real baking is complex. Maybe the sugar needs to interact with the eggs before the flour is added. Simple math misses these hidden connections.

The Challenge: Scientists face this with materials (like batteries or superconductors). They have thousands of ingredients (atoms, temperatures, pressures). They need a model that is smart enough to find complex recipes but clear enough to explain the physics.

The Solution: GoodRegressor (The "Lego Architect")

The author, Seong-Hoon Jang, built a new tool called GoodRegressor. Think of it as a Lego Architect that builds models in a very specific, disciplined way.

Instead of throwing Lego bricks at the wall and hoping a castle forms (which is how some AI works), GoodRegressor builds the castle floor by floor.

1. The "Depth" Concept (Building the Tower)

Imagine the ingredients are Lego bricks.

• Level 1 (Shallow): You just stack bricks on top of each other. (e.g., "More heat = faster reaction"). This is simple but often wrong.
• Level 2 (Deeper): You start connecting bricks side-by-side. (e.g., "Heat + Pressure = faster reaction").
• Level 3 (Deep): You build complex structures where bricks interact in weird ways. (e.g., "If Heat is high AND Pressure is low, THEN the reaction explodes, BUT only if the brick is red").

GoodRegressor controls exactly how deep the tower goes. It doesn't just guess; it systematically builds models of increasing complexity.

2. The "Goldilocks" Zone (Not Too Shallow, Not Too Deep)

The paper discovered a fascinating rule: Deeper isn't always better.

• Too Shallow: The model is too simple. It misses the magic interactions. (Like trying to bake a cake with just flour).
• Too Deep: The model gets too complicated. It starts memorizing the specific cake you baked yesterday instead of learning the general rules of baking. It "overfits" and fails on new cakes.
• Just Right: Every material system has a "sweet spot" or an optimal depth.
  • Analogy: Think of it like tuning a radio. If you turn the dial too far left, you get static. Too far right, you get static. There is one specific frequency where the music is crystal clear. GoodRegressor finds that frequency for every material.

How It Works: The "Jungle Run"

The paper describes the algorithm as a "Jungle Run." Imagine you are in a massive jungle (the search space) looking for a hidden treasure (the perfect formula).

• The Problem: The jungle is so big (10^457 possible paths!) that you can't walk every path.
• The Trick: GoodRegressor doesn't walk randomly. It uses a map (lexicographical order). It walks in a strict, organized grid pattern, checking specific spots efficiently.
• The "Swap" and "Transit": If it finds a good spot, it tries swapping a tree for a bush or changing the path slightly to see if the view gets better. It keeps refining the path until it finds the best view.

The Three Test Cases (The "Material Trios")

The author tested this on three different types of materials, and each had a different "personality" regarding how deep the model needed to be:

1. Oxygen-Ion Conductors (The "Sensitive" One):

   • Analogy: Like a delicate violin.
   • Result: It needed a specific, narrow depth to work. If the model was too simple or too complex, the music (prediction) fell apart. This tells us the physics here is tightly coupled and precise.

2. NASICONs (The "Relaxed" One):

   • Analogy: Like a campfire.
   • Result: It worked well even with a shallow model. You didn't need to dig deep to find the heat. The ingredients interact in a simpler way, so a basic model was almost as good as a complex one.

3. Superconducting Oxides (The "Complex" One):

   • Analogy: Like a chaotic jazz band.
   • Result: It needed a deep, broad model. The ingredients interact in many layers. You had to go deep to understand the music, but even then, there was a limit before it got too messy.

Why This Matters

1. Transparency: Unlike "Black Box" AI, GoodRegressor gives you the actual formula. You can read it and say, "Ah, I see! The material works because of this specific interaction."
2. Efficiency: It doesn't waste time searching the whole jungle. It knows exactly where to look based on the "depth" of the problem.
3. New Science: By finding the "optimal depth" for a material, scientists can learn something new about the material itself. If a material needs a deep model, it means the physics is complex and entangled. If it needs a shallow model, the physics is simpler.

The Bottom Line

GoodRegressor is a new way to teach computers to do science. Instead of just guessing the answer, it builds a hierarchical, step-by-step explanation that is both accurate and easy for humans to understand. It teaches us that in science, the "best" model isn't always the most complex one; it's the one that matches the complexity of the universe it is trying to describe.

Technical Summary

1. Problem Statement

Scientific machine learning faces a fundamental trade-off between predictive performance and interpretability.

• Black-box models (e.g., Neural Networks, XGBoost, Random Forests) often achieve high accuracy but lack structural transparency, making it difficult to extract physical insights or governing laws.
• White-box models (e.g., linear regression, standard symbolic regression) offer explicit functional forms but often fail to capture the hierarchical and nonlinear entanglement of descriptors found in complex physical systems (e.g., materials science).
• The Core Challenge: In high-dimensional compositional spaces (like materials design), target properties emerge from complex, multi-level interactions among descriptors. The search space for symbolic expressions grows explosively (e.g., $\sim 10^{457}$ candidate structures), making exhaustive search computationally intractable. Existing symbolic regression methods often rely on stochastic mutation (evolutionary algorithms) or sparsity constraints, which may miss deep hierarchical structures or suffer from instability.

2. Methodology: GoodRegressor

The author proposes GoodRegressor, a hierarchical depth-controlled symbolic regression framework designed to navigate this massive search space while maintaining interpretability.

A. Core Concept: Hierarchical Inductive Bias

Instead of treating interaction depth as a byproduct of feature expansion, GoodRegressor treats interaction depth as an explicit, controllable structural axis. It enforces a lexicographically ordered, multi-level construction of nonlinear descriptor interactions. This acts as a "design principle" to systematically assemble complex descriptors without relying solely on stochastic variation.

B. Algorithmic Workflow

The framework operates through a structured pipeline involving five modules: Parser, Designer, Curator, Regressor, and Designer (Post-processing). The core regression algorithm consists of four iterative steps (a "Run-through, Swap, Transit, Pick" cycle):

1. Run-through (Jungle):

   • The algorithm divides the massive combinatorial search space lexicographically across multiple CPU cores (MPI-parallelized).
   • It samples specific combinations of $n_t$ active variables from a pool of $N$ candidates at fixed intervals ("jumping-jack-flash").
   • It identifies the best provisional model ($M_{ic}$) for each core based on validation $R^2$ and strict statistical constraints (F-test and t-test with $p < 0.05$).

2. Swap:

   • A local refinement step where the least significant variable (highest $p$-value) in the current best model is swapped with inactive variables to improve the fit.

3. Transit:

   • Scalar transformations (e.g., $\log, \exp, \sin, \text{erf}, \text{polynomials}$) are applied to active variables to capture nonlinearities. The best transformation is retained.
   • Note: The "Swap" and "Transit" steps alternate until convergence.

4. Pick (Depth Control & Expansion):

   • This is the critical hierarchical step. The algorithm reduces the number of active variables ($n_t \to n_t - 1$) but expands the candidate pool.
   • The new pool includes the original features, scalar-transformed features, and their multiplicative/divisional interactions generated in the previous step.
   • This forces the model to rely on higher-order composite terms to explain the data as the linear term count decreases, effectively deepening the symbolic hierarchy.
   • This cycle repeats until the average cross-validated $R^2$ ($\langle R^2 \rangle_{100}$) no longer improves.

5. Bagging (Ensemble):

   • The entire pipeline is repeated $N_f$ times (typically 10) with different random train-validation splits (8:2 ratio).
   • The resulting individual models are stacked into a consensus model ($M_{f, N_f}$) via least-squares fitting, enhancing robustness and reproducibility.

3. Key Contributions

• Explicit Depth Control: Introduces interaction depth as a structural hyperparameter that governs the "grammar" of scientific hypothesis generation, rather than just a capacity limit.
• Tractable Search in Massive Spaces: Demonstrates that disciplined, lexicographically ordered construction can navigate search spaces of $\sim 10^{457}$ candidates, a scale previously considered intractable for symbolic regression.
• Interaction-Depth Evolution: Proposes a new diagnostic metric where the evolution of predictive performance against interaction depth reveals the intrinsic hierarchical complexity of a dataset.
• Reproducibility: Uses stacking ensembles to mitigate the stochastic nature of symbolic regression, ensuring stable and reproducible physical insights.

4. Results and Case Studies

The framework was tested on three high-complexity materials datasets:

A. Oxygen-Ion Conductors ($E_a$ and Pre-exponential Factor $A$)

• Dataset: 483 samples, 358 initial features. Search space $\sim 10^{457}$.
• Performance: GoodRegressor outperformed all black-box models (XGBoost, LightGBM, etc.) and other symbolic baselines (SISSO, PySR, $\Phi$-SO).
  • $E_a$: $\langle R^2 \rangle = 0.726$ (vs. best black-box $\le 0.667$).
  • $A$: $\langle R^2 \rangle = 0.654$.
• Depth Signature: Showed a sharply defined optimal window ($n_t \approx 13-18$). Performance degraded significantly if interaction depth was disabled, indicating tightly coupled, complex descriptor entanglement.
• Insight: Identified that low charge disorder, low oxygen ratio, loose packing, and high shear modulus lower activation energy.

B. NASICONs (Na-ion Conductivity)

• Dataset: 180 samples, 211 features. Search space $\sim 10^{124}$.
• Performance: Outperformed black-box models ($\langle R^2 \rangle = 0.862$).
• Depth Signature: Showed a broad, shallow dependence with an optimum near $n_t \approx 20$. Disabling interactions caused minimal performance loss.
• Insight: Suggests that Na-ion conductivity in NASICONs is governed by weaker descriptor coupling and is structurally more tractable than oxygen-ion conduction.

C. Superconducting Oxides ($T_c$)

• Dataset: 1,358 samples (filtered), 20+ features. Search space $\sim 10^{430}$.
• Performance: Matched state-of-the-art black-box models ($\langle R^2 \rangle = 0.536$) while retaining interpretability, outperforming white-box baselines.
• Depth Signature: Exhibited a broadened peak with a distinct optimal window ($n_t \approx 13-17$). Disabling interactions caused substantial performance degradation.
• Insight: Indicates strong but partially redundant hierarchical entanglement in superconducting mechanisms.

D. Materials Discovery

• The framework successfully predicted promising new materials:
  • Oxygen-ion: Apatite-type $La_{9.5}Si_{5.5}Al_{0.5}O_{26}$ with predicted $E_a \approx 494$ meV.
  • NASICON: Zr-free $Na_{3.4}Y_{0.4}Hf_{1.6}Si_2PO_{12}$ with high conductivity.
  • Superconductors: Identified candidates with predicted $T_c$ up to 275 K (though noted as potentially mathematical artifacts, the compositional trends were chemically sensible).

5. Significance and Conclusion

• New Taxonomy of Complexity: The paper establishes that scientific datasets are not just "simple" or "complex" but possess distinct interaction-depth signatures. The sharpness of the optimal depth window and the penalty for disabling interactions serve as an empirical taxonomy for hierarchical complexity.
• Design Principle for Interpretable AI: GoodRegressor shifts the focus from algorithmic competition to inductive-bias design. It proves that for systems with deeply entangled nonlinear physics, structured hierarchical construction is necessary to achieve both accuracy and interpretability.
• Scalability: By combining lexicographic ordering with depth control, the method makes symbolic regression viable for high-dimensional, real-world scientific problems where search spaces are astronomically large.

In summary, GoodRegressor provides a robust, reproducible, and interpretable framework for scientific discovery, demonstrating that controlling the "depth" of symbolic interactions is key to unlocking the physical laws hidden within complex compositional data.