Self-scaling tensor basis neural network for Reynolds stress modeling of wall-bounded turbulence

This paper proposes a self-scaling tensor basis neural network (STBNN) that utilizes an invariant velocity-gradient normalization to achieve robust, geometry-independent Reynolds stress modeling for wall-bounded turbulence, demonstrating superior accuracy and generalization across Reynolds numbers and unseen flow configurations compared to existing data-driven and traditional closure models.

The Gist

The Big Picture: Predicting the Chaos of Wind and Water

Imagine you are trying to predict how a river flows around a rock, or how wind swirls around a skyscraper. In the world of engineering, this is called turbulence. It's the chaotic, swirling mess of water or air that happens when things move fast.

To design safe bridges, efficient airplanes, or better cars, engineers need to simulate this turbulence on computers. But simulating every single tiny swirl is like trying to count every grain of sand on a beach—it takes too much computing power and time.

So, engineers use a shortcut called RANS (Reynolds-Averaged Navier-Stokes). Instead of tracking every tiny swirl, they look at the "average" flow and try to guess what the missing, chaotic swirls are doing. This guess is called the Reynolds Stress.

The Problem: The old shortcuts (mathematical formulas) are like using a generic map for every terrain. They work okay on flat roads, but when the terrain gets weird—like a steep hill or a sharp turn—they get lost. They fail to predict where the flow will separate from a wall or how it will swirl back.

The Old AI Solution: The "Smart" Map with a Flaw

Recently, scientists started using Neural Networks (a type of AI) to learn these missing swirls from high-quality data. One popular method was called TBNN (Tensor Basis Neural Network).

Think of TBNN as a very smart student who memorized a textbook on fluid dynamics. It learned the rules of how water moves. However, this student had a bad habit: it relied too heavily on a specific ruler to measure things.

In the old model, the AI used a "turbulence ruler" (based on energy and dissipation) to scale its predictions. The problem is, near a wall (like the side of a pipe), this ruler breaks down. It's like trying to measure the height of a skyscraper with a ruler that shrinks when you get close to the ground. Because the ruler was broken near the walls, the AI couldn't generalize. If you trained it on a small pipe, it would fail miserably on a big pipe or a different shape.

The New Solution: The "Self-Scaling" AI (STBNN)

The authors of this paper, Yuan and Li, invented a new AI called STBNN (Self-Scaling Tensor Basis Neural Network).

Here is the magic trick: They gave the AI a ruler that scales itself.

Instead of using a fixed, fragile ruler, the new AI looks at the local flow and says, "Okay, the water is moving fast and spinning here; I will adjust my scale to match this specific moment."

• The Analogy: Imagine you are a chef cooking soup.
  • Old AI (TBNN): You have a recipe that says "Add 1 cup of salt." But if you are cooking a tiny bowl of soup or a giant vat, 1 cup is either too much or too little. The recipe fails.
  • New AI (STBNN): You have a recipe that says "Add salt until it tastes right relative to the size of the pot." You automatically adjust the amount based on what you see in the pot right now.

By using a "self-scaling" method based on the speed and spin of the water (velocity gradients), the new AI doesn't need to know the size of the pipe or the distance to the wall to do its job. It just looks at the local physics and adapts.

What Did They Test?

They put this new AI to the test in two very different scenarios:

1. The Flat Pipe (Plane Channel): They trained the AI on pipes of certain sizes and then asked it to predict flow in pipes of completely different sizes (some much smaller, some much larger).

   • Result: The old AI got confused and made big mistakes. The new AI (STBNN) nailed it, predicting the flow with 99% accuracy, even for sizes it had never seen before.

2. The Bumpy Road (Periodic Hills): They simulated water flowing over a series of hills. Some hills were gentle; some were steep. They trained the AI on gentle hills and asked it to predict flow over steep, unseen hills.

   • Result: This is where turbulence usually breaks old models (because the water separates and swirls wildly). The old AI got the shape of the swirls wrong. The new AI predicted exactly where the water would detach from the hill and swirl back, matching the "perfect" computer simulations (DNS) almost perfectly.

Why Does This Matter?

Think of this as moving from memorizing to understanding.

• The old AI was memorizing specific answers for specific questions. If you asked a question it hadn't seen, it guessed poorly.
• The new AI (STBNN) learned the underlying physics of how to scale things. It understands the "rules of the game" rather than just the "score."

The Bottom Line:
This new method allows engineers to use AI to predict complex fluid flows (like air over a wing or water in a pipe) with much higher confidence. It works for different sizes of pipes and different shapes of obstacles without needing to be retrained every time. It's a step toward making computer simulations of the real world as reliable as a physical wind tunnel, but much faster and cheaper.

Technical Summary

1. Problem Statement

Despite the success of machine learning in turbulence modeling, existing Tensor Basis Neural Networks (TBNN) face significant limitations when applied to wall-bounded turbulent flows:

• Lack of Generalization: Standard TBNNs struggle to generalize across different Reynolds numbers ($Re$) and geometries. They often fail when tested on flow conditions outside their training distribution.
• Scaling Issues: Traditional TBNNs rely on the turbulent time scale $\tau = k/\varepsilon$ (where $k$ is turbulent kinetic energy and $\varepsilon$ is the dissipation rate) for normalizing input tensors. Near walls, $\varepsilon$ is difficult to predict accurately without empirical damping functions or wall-distance inputs ($d_w$). This introduces regime-dependent uncertainty and breaks the model's ability to scale consistently across different flow regimes.
• Physics Inconsistency: The reliance on empirical coefficients and wall-distance inputs limits the physical interpretability and frame indifference of the models, leading to poor performance in complex separated flows.

2. Methodology: Self-Scaling Tensor Basis Neural Network (STBNN)

The authors propose a Self-Scaling Tensor Basis Neural Network (STBNN) that reformulates the input representation to achieve intrinsic, geometry-independent scaling without relying on empirical coefficients or wall-distance data.

• Invariant Velocity-Gradient Normalization:
  Instead of using the turbulent time scale $k/\varepsilon$, the STBNN normalizes the mean strain-rate ($S$) and rotation-rate ($\Omega$) tensors using an invariant velocity-gradient magnitude derived from the first two invariants of the velocity-gradient tensor:
  $$\tilde{S} = \frac{S}{\sqrt{\|S\|^2 + \|\Omega\|^2}}, \quad \tilde{\Omega} = \frac{\Omega}{\sqrt{\|S\|^2 + \|\Omega\|^2}}$$
  Here, $\| \cdot \|$ denotes the Frobenius norm. This provides an intrinsic local time scale that naturally balances strain and rotation effects.

• Tensor Basis Formulation:
  The Reynolds stress anisotropy tensor ($b_{ij}$) is modeled as a linear combination of five isotropic tensor basis functions constructed from the self-scaled tensors ($\tilde{S}, \tilde{\Omega}$):
  $$b = \sum_{n=1}^{5} g_n(\tilde{\lambda}, q; \theta) \tilde{T}^{(n)}$$

  • Inputs: The neural network takes scalar invariants ($\tilde{\lambda}_1 \dots \tilde{\lambda}_5$) of the self-scaled tensors and auxiliary physical features ($q$) as input.
  • Architecture: A fully connected feed-forward network with 5 hidden layers (20 neurons each) using GELU activation functions.
  • Invariance: The formulation preserves Galilean and rotational invariance because the normalization factor is constructed solely from scalar invariants of the velocity gradient.

• Training Strategy:

  • A Priori: Trained to minimize the Mean Squared Error (MSE) between predicted and Direct Numerical Simulation (DNS) deviatoric Reynolds stresses.
  • A Posteriori: Integrated into RANS solvers (OpenFOAM) using a $k$-corrective-frozen strategy. The turbulent kinetic energy ($k$) and dissipation ($\varepsilon$) are provided by a baseline $k-\varepsilon$ model, while the STBNN replaces only the stress anisotropy. To ensure numerical stability, the nonlinear stress component is treated explicitly via Helmholtz-Hodge decomposition.

3. Key Contributions

1. Intrinsic Scaling Mechanism: The introduction of a self-scaling normalization based on velocity-gradient invariants eliminates the need for empirical damping functions and wall-distance inputs, resolving the generalization bottleneck of standard TBNNs.
2. Frame-Indifferent Formulation: The model maintains strict adherence to physical symmetries (Galilean and rotational invariance) while providing a unified scaling across different flow regimes.
3. Robust Generalization: The framework demonstrates the ability to generalize to unseen Reynolds numbers and complex geometries without retraining, a capability previously lacking in data-driven RANS closures.

4. Results

The model was evaluated using Plane Channel Flows (varying $Re_\tau$) and Periodic Hill Flows (varying geometry steepness $\alpha$).

A. A Priori Tests (Direct Comparison with DNS)

• Plane Channel Flows:
  • Accuracy: STBNN achieved correlation coefficients >99.9% and relative errors <2% for all Reynolds stress components across training and validation sets ($Re_\tau$ from 550 to 10,000).
  • Generalization: Unlike the baseline TBNN (which showed errors >90% in validation) and algebraic models (LEVM/QEVM), STBNN maintained near-perfect accuracy at unseen Reynolds numbers, correctly capturing near-wall scaling and logarithmic-layer behavior.
• Periodic Hill Flows:
  • Geometry Generalization: For unseen hill steepness ($\alpha = 0.8, 1.2$), STBNN maintained correlations >98% and relative errors <20%.
  • Comparison: The baseline TBNN showed significant accuracy degradation (correlations dropped to ~0.46 for $R_{33}$) on unseen geometries, whereas STBNN remained robust.
  • Flow Features: STBNN accurately reproduced stress peaks and spatial distributions in separation bubbles and reattachment regions, whereas LEVM failed to capture anisotropy and TBNN showed distortions near reattachment.

B. A Posteriori Tests (RANS Simulations)

• Mean Velocity Profiles: In channel flows, STBNN predictions matched DNS profiles closely across all $Re_\tau$, correctly capturing the log-layer slope. Baseline TBNN significantly overpredicted velocity in the outer layer.
• Separated Flows: In periodic hill flows, STBNN accurately predicted the separation bubble size and reattachment location.
  • LEVM underestimated separation.
  • QEVM showed near-wall discrepancies.
  • TBNN exhibited non-physical oscillations near the reattachment point.
  • STBNN provided stable, physically consistent predictions.

5. Significance

This work represents a significant step forward in data-driven turbulence modeling by addressing the critical issue of generalizability.

• Physical Consistency: By deriving the scaling mechanism from first principles (velocity gradient invariants) rather than empirical fits, the model bridges the gap between data-driven flexibility and physical rigor.
• Industrial Applicability: The ability to train on low-Reynolds-number or simple geometries and successfully apply the model to high-Reynolds-number or complex 3D separated flows makes this approach highly attractive for practical engineering CFD applications where DNS data is scarce.
• Future Impact: The proposed framework offers a pathway to develop robust, universal Reynolds-stress closures that do not require case-specific tuning, potentially reducing the reliance on expensive high-fidelity simulations for model calibration.