Explainable Hierarchical Deep Learning Neural Networks (Ex-HiDeNN)

This paper introduces Ex-HiDeNN, a novel two-step neural architecture that combines hierarchical deep learning with symbolic regression to efficiently discover accurate, interpretable closed-form expressions from limited data, outperforming traditional methods across dynamical system identification and various engineering applications.

The Gist

Imagine you are a detective trying to solve a mystery. You have a massive pile of clues (data) left at a crime scene, but they are messy, scattered, and hard to read. Your goal isn't just to guess what happened; you want to find the exact rule or formula that explains the crime so clearly that anyone can read it and say, "Ah, that makes perfect sense!"

This is the challenge the paper "Explainable Hierarchical Deep Learning Neural Networks" (Ex-HiDeNN) tackles.

Here is the story of how they solved it, broken down into simple concepts and analogies.

The Problem: The "Black Box" vs. The "Messy Notebook"

In the world of Artificial Intelligence (AI), there are two main types of detectives:

1. The "Black Box" (Deep Neural Networks): These are super-smart detectives who can look at a messy crime scene and guess the outcome with incredible accuracy. But, if you ask them how they figured it out, they just shrug. They can't explain their logic. They are like a genius chef who makes a perfect dish but refuses to write down the recipe. You can eat the food, but you can't cook it yourself.
2. The "Symbolic Regression" (The Rule Finder): These detectives try to write down the exact recipe (a math formula) from the start. They are very transparent and easy to understand. However, they are often slow, get confused by messy data, and struggle when the crime scene is huge or complex. They might give up or write a recipe that is too complicated to be useful.

The Goal: We need a detective who is as smart as the "Black Box" but can also write a simple, clear recipe like the "Rule Finder."

The Solution: The "Ex-HiDeNN" Detective Team

The authors created a new two-step team called Ex-HiDeNN. Think of it as a partnership between a Master Sketch Artist and a Mathematical Editor.

Step 1: The Master Sketch Artist (C-HiDeNN-TD)

First, the team uses a special type of AI (a neural network) to look at the messy data.

• The Analogy: Imagine the data is a blurry, noisy photograph of a landscape. The AI doesn't try to write a poem about it yet. Instead, it acts like a sketch artist. It draws a smooth, clean, perfect line-art version of that landscape. It removes the noise (the graininess) and connects the dots perfectly.
• The Magic Trick: This artist is special because it can tell you how the landscape is built. It can look at the drawing and say, "Hey, the left side of this hill depends only on the wind, and the right side depends only on the rain. They don't really mix!"
• The Score: The team calculates a "Separability Score." This is like a report card that tells them: "Is this problem simple enough to be broken into small, independent pieces, or is it a giant, tangled knot?"

Step 2: The Mathematical Editor (Symbolic Regression)

Once the sketch is perfect and the team knows how "tangled" the problem is, they hand the clean drawing to the Mathematical Editor (a tool called PySR).

• The Strategy:
  • If the problem is simple (High Separability): The Editor breaks the drawing into tiny, single-variable pieces (like "Wind" and "Rain"). It writes a tiny, simple formula for each piece and then multiplies them together. It's like building a Lego tower one brick at a time.
  • If the problem is medium: The Editor writes a few small formulas and adds them together.
  • If the problem is a tangled knot (Low Separability): The Editor looks at the whole drawing at once and tries to find a complex formula that fits the whole picture.
• The Result: Because the Editor is working with the clean sketch (not the noisy original data), it finds the perfect math formula much faster and more accurately than if it had to guess from the messy data directly.

Why This Matters: Real-World Superpowers

The paper shows this team solving three real engineering mysteries that were previously very hard to crack:

1. The Fatigue Mystery (Metal Fatigue):

   • The Problem: Engineers wanted to know exactly how long a piece of 3D-printed steel would last before breaking. There were 25 different factors (temperature, chemical makeup, cooling speed, etc.).
   • The Result: Ex-HiDeNN didn't just predict the answer; it wrote a single, readable equation. It told engineers exactly how carbon, nickel, and cooling rates interact. It was like finding the "secret sauce" recipe for durable steel.

2. The Hardness Mystery (Micro-indentation):

   • The Problem: How hard is a material? Usually, you have to test it physically.
   • The Result: The team found a formula that predicts hardness based on other properties. It was 25 times more accurate than previous methods. It's like being able to guess the hardness of a diamond just by looking at its color and weight, without ever scratching it.

3. The Physics Mystery (Yield Surfaces):

   • The Problem: How does sand or soil behave under pressure?
   • The Result: They discovered a classic physics law (the Matsuoka-Nakai yield surface) directly from data, without anyone telling them the law existed. It's like an AI rediscovering Newton's laws of gravity just by watching apples fall.

The Big Picture

Think of Ex-HiDeNN as a translator.

• Data is a foreign language full of noise and confusion.
• Black Box AI is a fluent speaker who can answer questions but won't teach you the language.
• Ex-HiDeNN listens to the foreign language, cleans it up, and then translates it into English (a simple math formula) that anyone can read, understand, and use.

In short: It takes the "magic" out of AI and turns it into "math" we can trust. It gives us the best of both worlds: the brainpower of deep learning with the clarity of a textbook equation.

Technical Summary

1. Problem Statement

The paper addresses the critical trade-off in machine learning between expressivity (the ability to model complex, non-linear relationships) and interpretability (the ability for humans to understand the model's logic).

• Black-box models (e.g., Deep Neural Networks, CNNs) offer high expressivity but lack transparency, making them unsuitable for safety-critical engineering applications where physical laws and causality must be verified.
• Symbolic Regression (SR) methods (e.g., Genetic Programming, PySR) produce interpretable, closed-form mathematical expressions but struggle with:
  • Combinatorial explosion: The search space grows exponentially with the number of variables and operators.
  • Noise sensitivity: They often overfit to noisy data.
  • Scalability: They frequently fail to recover simple expressions in high-dimensional or sparse datasets.
• The Gap: There is a need for a framework that can efficiently discover accurate, parsimonious, closed-form expressions from complex, high-dimensional, and potentially noisy engineering data without succumbing to the limitations of pure symbolic regression.

2. Methodology: The Ex-HiDeNN Framework

Ex-HiDeNN is a novel, two-stage hybrid architecture that combines a structured, interpolating neural network with symbolic regression. The process is adaptive, guided by a separability score.

Stage 1: Surrogate Modeling with C-HiDeNN-TD

The first stage employs C-HiDeNN-TD (Convolutional Hierarchical Deep Learning Neural Network with Tensor Decomposition).

• Architecture: It uses a hierarchical structure with tensor decomposition to split the functional space into one-dimensional sub-spaces. It utilizes a "patch function" (inspired by meshfree methods) to interpolate data.
• Properties: The resulting surrogate is continuously differentiable and interpolating (passes exactly through training points).
• Role: It acts as a noise filter and a dimensionality reducer. Instead of applying symbolic regression directly to raw data, the algorithm samples from this smooth surrogate.

Stage 2: Separability Analysis and Adaptive Sampling

A key innovation is the Hessian-based Separability Score ($S^\otimes$).

• Calculation: The algorithm computes the Hessian matrix of the learned surrogate. It analyzes the ratio of diagonal terms to off-diagonal terms in the log-Hessian to determine if the function is multiplicatively separable (i.e., $f(x,y) \approx f_1(x)f_2(y)$).
• Adaptive Strategy: Based on the score $S^\otimes$, the framework selects a sampling strategy for the second stage:
  • High Separability ($S^\otimes \gtrsim 0.95$): Samples each dimension independently. Symbolic regression is run on each 1D slice, and the results are multiplied. This reduces the search space from exponential to linear.
  • Moderate Separability ($0.6 \lesssim S^\otimes \lesssim 0.95$): Samples per "mode" (tensor component). The final expression is a sum of products (tensor decomposition form).
  • Low Separability ($S^\otimes \lesssim 0.6$): Samples the full multi-dimensional space globally. Symbolic regression is performed on the full dataset, but the surrogate ensures the sampling is intelligent and noise-robust.

Stage 3: Symbolic Regression (PySR)

The sampled data from Stage 2 is fed into PySR (a high-performance evolutionary symbolic regression engine). Because the input data is derived from a smooth, differentiable surrogate, PySR operates on a cleaner signal, significantly reducing the risk of overfitting to noise and shrinking the effective search space.

3. Key Contributions

1. Novel Hybrid Architecture: Introduction of Ex-HiDeNN, which marries the scalability of tensor-decomposed neural networks with the interpretability of symbolic regression.
2. Pointwise Separability Score: Development of a metric ($S^\otimes$) derived via automatic differentiation of the surrogate's Hessian to quantify local multiplicative separability, enabling adaptive sampling strategies.
3. Robustness to Noise: The framework demonstrates superior noise filtering capabilities compared to direct symbolic regression, as the C-HiDeNN-TD surrogate captures the underlying signal while smoothing stochastic noise.
4. Engineering Discovery: Successful application to three complex engineering problems with unknown or complex physics, outperforming state-of-the-art baselines.

4. Results and Validation

Benchmark Performance

• Accuracy: On the Vladislavleva benchmark suite (8 functions), Ex-HiDeNN recovered exact functional forms for 5 out of 8 cases (V1, V2, V3, V5, V6) and achieved orders-of-magnitude lower error than reference methods (Genetic Algorithms, KAN, PySR alone) on the remaining cases.
• Noise Robustness: In tests with Gaussian noise ($\sigma$ up to 0.32), Ex-HiDeNN showed a 2-to-10-fold improvement in RMSE compared to PySR alone, due to the interpolating nature of the surrogate.
• Dynamical Systems: Successfully recovered the exact equations of the chaotic Lorenz system from time-series data, with coefficients accurate to two significant figures.

Engineering Applications

1. Fatigue Life in Additive Manufacturing:
   • Data: 25 input parameters, 437 experiments (extremely sparse: ~1.27 samples per dimension).
   • Result: Discovered a closed-form equation with $R^2 = 0.9767$ on the test set and a relative error of 2.8%. The expression revealed intuitive metallurgical relationships (e.g., logarithmic dependence on Carbon, exponential decay with Copper).
2. Vickers Hardness Prediction:
   • Data: Micro-indentation data with 6 input parameters.
   • Result: Outperformed the SISSO method significantly. Ex-HiDeNN achieved $R^2 = 0.9999$ and an RMSE of 0.066 GPa, compared to SISSO's $R^2 = 0.9552$ and RMSE of 1.66 GPa (a 25-fold reduction in error).
3. Matsuoka-Nakai Yield Surface:
   • Data: 13,200 data points for pressure-sensitive granular materials.
   • Result: Discovered a constitutive law with $R^2 = 0.9992$ and 2.3% relative error. The derived expression correctly captured the lode-angle dependence and pressure scaling inherent in the Matsuoka-Nakai criterion.

5. Significance and Conclusion

Ex-HiDeNN represents a significant step toward trustworthy AI in science and engineering. By automating the discovery of closed-form expressions, it bridges the gap between data-driven modeling and physical interpretability.

• Efficiency: It mitigates the "curse of dimensionality" in symbolic regression by exploiting separability, making high-dimensional problems tractable.
• Actionability: The resulting models are not just predictors but mathematical laws that can be integrated into simulation pipelines, used for optimization, and subjected to formal analysis (e.g., bifurcation studies).
• Future Work: The authors plan to integrate more robust surrogate models (like KHRONOS) and explore gradient-based symbolic regression to further leverage the differentiability of the neural surrogates.

In summary, Ex-HiDeNN provides a scalable, noise-robust, and highly accurate method for "white-box" model discovery, proving that complex engineering phenomena can be distilled into simple, human-understandable mathematical expressions.