Uncertainty Quantification in PINNs for Turbulent Flows: Bayesian Inference and Repulsive Ensembles

This paper develops and evaluates probabilistic extensions of Physics-Informed Neural Networks (PINNs), specifically Bayesian inference, Monte Carlo dropout, and repulsive deep ensembles, to provide reliable uncertainty quantification for inverse turbulent flow problems, demonstrating that Bayesian methods yield the most consistent estimates while repulsive ensembles offer a computationally efficient alternative.

The Gist

Imagine you are trying to recreate a complex, swirling storm inside a computer. You have a few scattered weather reports (data) and the laws of physics (the rules of how air moves). Your goal is to fill in the gaps and predict exactly what the wind and pressure are doing everywhere, even where you have no measurements.

This is the challenge of Turbulent Flow Modeling. It's notoriously difficult because the math is messy, and the data is often sparse (like trying to guess the shape of a whole cloud by looking at just a few raindrops).

For a long time, scientists used Physics-Informed Neural Networks (PINNs) to solve this. Think of a PINN as a super-smart student who is given a test. The student has to memorize the few weather reports they have and follow the strict rules of physics. If they get the answer right, they pass.

The Problem:
The old PINN students were overconfident. They would give you a single, precise answer for the wind speed, but they wouldn't tell you how sure they were. If they were guessing wildly in an area with no data, they would still say, "I'm 100% certain!" This is dangerous in engineering. If you are designing a bridge or a plane, you need to know where the model is shaky so you don't build something that fails.

The Solution:
This paper introduces a new way to teach these students how to admit when they are unsure. The authors tried three different "classroom strategies" to quantify uncertainty (how much the model might be wrong):

1. The Bayesian Approach (The "Many-Worlds" Student)

• The Analogy: Imagine instead of one student, you have a single student who is incredibly indecisive. They run the simulation thousands of times, slightly changing their internal logic each time, just to see how the answer might wiggle.
• How it works: This is called Bayesian PINN. It treats the network's weights (its knowledge) as a range of possibilities rather than fixed numbers. It uses a sophisticated sampling method (like a random walk through a maze) to explore all the different ways the physics could fit the data.
• The Result: This is the gold standard. It gives the most honest answer. It says, "Here is my best guess, and here is a 'confidence bubble' around it. If the bubble is huge, you know the model is guessing." It even figured out the pressure field perfectly, even though no one ever told it what the pressure was!

2. The Dropout Method (The "Sleepy" Student)

• The Analogy: Imagine a student who is forced to take a nap during the test. Every time they answer a question, they randomly forget a few facts they learned. You ask them the same question 1,000 times, and every time they forget something different.
• How it works: This is MC Dropout. By randomly "turning off" parts of the network during the test, it forces the model to make slightly different predictions each time. The spread of these answers tells you the uncertainty.
• The Result: It's fast and cheap, but it's a bit clumsy. In this paper, it tended to be too conservative. It would say, "I'm not sure at all!" even when it had plenty of data. It's like a student who is so afraid of being wrong that they refuse to guess, even when they should.

3. The Repulsive Ensemble (The "Debate Club")

• The Analogy: Imagine you hire 10 different students to solve the problem.
  • The Old Way (Vanilla Ensemble): You just give them all the same textbook and let them study alone. The problem? They all end up thinking the exact same thing. They all converge on the same "wrong" answer because the physics rules are so strong. They have zero diversity.
  • The New Way (Repulsive Ensemble): You add a rule: "You must disagree with your friends!" You force the students to stay apart from each other in their thinking. If two students start to think too similarly, you push them apart.
• How it works: The authors created a "Repulsive Deep Ensemble." They trained 10 networks but added a special penalty if the networks started to produce the same output. This forces them to explore different solutions.
• The Result: This was a huge success for speed. It was much faster than the Bayesian method but still gave a good idea of the uncertainty for the main wind speeds. However, for the trickiest parts (the "Reynolds stress," which is like the chaotic swirls of the storm), it wasn't quite as honest as the Bayesian method.

The Big Takeaway

The paper tested these methods on two scenarios:

1. A simulated cylinder: Where they knew the "true" answer (like a perfect simulation).
2. Real experimental data: Where they used actual wind tunnel measurements (which are noisy and imperfect).

The Verdict:

• If you need the absolute truth and can wait: Use the Bayesian PINN. It's the most reliable, honest, and well-calibrated. It tells you exactly where it's guessing.
• If you need a quick answer for the main flow: Use the Repulsive Ensemble. It's like a fast, efficient debate club that gives you a good estimate of the wind, even if it's a little less precise on the chaotic swirls.
• Don't use the old "Vanilla" Ensembles: Without the "repulsive" rule to force them to disagree, they all collapse into a single, overconfident, and potentially wrong answer.

In short: The authors figured out how to make AI models for weather and fluid dynamics stop lying about how sure they are. They gave us tools to know when to trust the computer and when to double-check the math.

Technical Summary

1. Problem Statement

The paper addresses the critical challenge of epistemic uncertainty quantification in Physics-Informed Neural Networks (PINNs) applied to turbulent flow modeling, specifically for Reynolds-Averaged Navier–Stokes (RANS) equations.

• Context: RANS is the industry standard for turbulence modeling due to computational efficiency but suffers from a "closure problem" where the Reynolds stress tensor must be modeled.
• The Gap: Existing PINN formulations for inverse RANS problems (inferring velocity, pressure, and Reynolds stresses from sparse data) are typically deterministic. They provide a single point estimate without quantifying the uncertainty inherent in ill-posed inverse problems.
• Consequence: Without uncertainty estimates, it is impossible to distinguish between regions where predictions are data-constrained and regions driven by the network's inductive bias. This limits the reliability of PINNs for safety-critical applications and experimental design.

2. Methodology

The authors propose and systematically evaluate three probabilistic extensions of PINNs to address these limitations:

A. Bayesian PINNs (BPINNs)

• Framework: Places a prior distribution over network weights and computes the posterior distribution conditioned on data and physics constraints.
• Inference: Uses Hamiltonian Monte Carlo (HMC) via the No-U-Turn Sampler (NUTS).
• Key Innovations:
  • Tempered Multi-Component Likelihood: To prevent the posterior from becoming overly confident (cold posterior effect) in high-dimensional spaces, the authors employ a tempered likelihood. They assign distinct tempering exponents ($\beta$) to velocity data, Reynolds stress data, and PDE residuals ($\beta_d > \beta_f > \beta_r$) to reflect their relative reliability.
  • Two-Stage Inference: A Maximum A Posteriori (MAP) pre-training phase (using Adam and L-BFGS) initializes the sampler, followed by NUTS sampling.
  • Post-Hoc Recalibration: A scaling factor is applied to the posterior standard deviation to ensure the predictive intervals achieve nominal coverage (e.g., 95%).

B. MC Dropout PINNs

• Framework: Interprets dropout at test time as approximate variational inference.
• Process: A single network is trained with dropout layers; during inference, multiple stochastic forward passes generate an ensemble of predictions to estimate mean and variance.
• Limitation: The uncertainty estimates are limited by the expressiveness of the variational family induced by dropout.

C. Repulsive Deep Ensemble PINNs (RDE-PINNs)

• Framework: Trains an ensemble of $M$ independent networks but enforces diversity to prevent "ensemble collapse" (where all members converge to the same solution).
• Repulsion Mechanisms:
  • Parameter-Space Repulsion: Penalizes similarity between weight vectors ($\theta_i$ and $\theta_j$).
  • Function-Space Repulsion: Penalizes similarity between network outputs (flow fields) at specific collocation points.
• Objective: The total loss includes the standard PINN loss (data + physics) plus a repulsive term ($\lambda_{rep} L_{rep}$).
• Finding: Function-space repulsion is shown to be superior for physics-informed problems as it directly enforces diversity in the predicted physical quantities.

3. Key Contributions

1. Repulsive Deep Ensembles for PDEs: First application of repulsive deep ensembles specifically tailored for systems of PDEs (RANS), demonstrating that function-space repulsion is essential for maintaining diversity in predicted flow fields.
2. Enhanced BPINN Formulation: Development of a robust BPINN framework featuring tempered multi-component likelihoods and post-hoc recalibration. This addresses the scalability and calibration issues of standard Bayesian PINNs in highly underdetermined settings.
3. Systematic Comparison: A comprehensive evaluation of BPINN, MC Dropout, and Repulsive Ensembles across a hierarchy of test cases:
   • Pedagogical: Van der Pol oscillator.
   • DNS Data: Turbulent flow past a cylinder at $Re = 3,900$.
   • Experimental Data: Turbulent flow past a cylinder at $Re = 10,000$ (using Particle Image Velocimetry data).
4. Quantitative Trade-off Analysis: Provides clear metrics on the trade-offs between computational cost, predictive accuracy, and uncertainty calibration quality.

4. Results

Van der Pol Oscillator (Pedagogical)

• BPINN: Delivered the most reliable, well-calibrated uncertainty estimates.
• Function-Space RDE: Provided a computationally efficient approximation with good calibration.
• Parameter-Space RDE: Failed to provide reliable calibration (underestimated uncertainty) because weight diversity did not translate to functional diversity.
• Vanilla Ensembles & MC Dropout: Vanilla ensembles collapsed (zero variance); MC Dropout was overly conservative and poorly calibrated.

Turbulent Flow ($Re = 3,900 & 10,000$)

• Predictive Accuracy: Both BPINN and Function-Space RDE achieved high accuracy in reconstructing mean velocity and pressure fields from sparse data.
• Uncertainty Calibration:
  • BPINN: Consistently provided the best-calibrated uncertainty across all variables (velocity, pressure, and Reynolds stresses). After recalibration, it achieved near-nominal 95% coverage. It successfully inferred pressure fields without direct data supervision.
  • Function-Space RDE: Achieved comparable accuracy for primary variables (velocity) but underestimated uncertainty for closure terms (Reynolds stresses). Coverage for closure terms dropped significantly (e.g., ~70% vs. target 95%).
  • Vanilla Ensembles: Suffered catastrophic collapse for Reynolds stress closures, with coverage as low as 1.8% for $f_y$, rendering the uncertainty estimates physically meaningless.
• Computational Cost: BPINN is computationally expensive (MCMC sampling), whereas Function-Space RDE offers a scalable, efficient alternative for primary flow variables.

5. Significance and Conclusion

• Practical Guidance: The paper establishes that for data-driven turbulence modeling, the choice of UQ method depends on the priority:
  • High Fidelity/Calibration: Use Bayesian PINNs (with tempering and recalibration) when rigorous uncertainty quantification is required, especially for ill-posed quantities like Reynolds stresses or pressure.
  • Efficiency: Use Function-Space Repulsive Ensembles when computational cost is a constraint and the primary interest is in mean flow variables, acknowledging that closure term uncertainty may be less reliable.
• Scientific Impact: The work highlights that ensemble diversity must be enforced in function space, not just parameter space, to effectively capture epistemic uncertainty in PDE-constrained inverse problems. It provides a robust framework for moving PINNs from deterministic solvers to reliable probabilistic tools for engineering design and safety analysis.