📊 statistics

A Multi-fidelity Estimator of the Expected Information Gain for Bayesian Optimal Experimental Design

This paper introduces an unbiased, variance-minimizing multi-fidelity estimator for Expected Information Gain (MF-EIG) within the approximate control variate framework, which achieves significant computational efficiency and variance reduction for Bayesian optimal experimental design by reparameterizing the objective function to decouple expectations from data models.

Original authors: Thomas E. Coons, Xun Huan

Published 2026-01-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Thomas E. Coons, Xun Huan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a scientist trying to figure out the best way to run an experiment to learn the most about a mysterious system. Maybe you're trying to understand how a new engine works, or how a virus spreads. You have a computer model that simulates this system, but the model is incredibly complex. Running it once takes a long time and costs a lot of money.

This is the world of Optimal Experimental Design (OED). The goal is to pick the perfect conditions for your experiment so you get the maximum amount of new knowledge. To measure "knowledge," scientists use a number called the Expected Information Gain (EIG). Think of EIG as a "learning score." The higher the score, the more you learn.

The Problem: The "Perfect" Model is Too Expensive

The catch is that calculating this "learning score" is incredibly hard. It's like trying to guess the average height of every person in a city by measuring just one person at a time. To get a good answer, you have to run your expensive computer model thousands of times. If your model takes an hour to run, you'd need to wait years to get a good answer.

Scientists have tried using "cheaper," less accurate models (low-fidelity models) to speed things up. But if you just use the cheap model, your answer might be wrong (biased). It's like trying to guess the weight of a gold bar using a bathroom scale; it's fast, but the result is useless.

The Solution: The "Multi-Fidelity" Team

The authors of this paper, Thomas Coons and Xun Huan, created a new method called MF-EIG (Multi-Fidelity Expected Information Gain). They didn't just swap the expensive model for a cheap one. Instead, they built a team of models working together.

Here is how they did it, using a simple analogy:

The Analogy: The Master Chef and the Sous Chefs
Imagine you are trying to judge the quality of a very complex, expensive dish (the High-Fidelity model). You can't taste it enough times to be sure because it takes hours to cook.

The Master Chef (High-Fidelity): Makes the perfect dish but takes 10 hours.
The Sous Chefs (Low-Fidelity): Make a "good enough" version of the dish in 10 minutes. They aren't perfect, but they taste somewhat similar to the Master Chef's dish.

The authors' method is like a tasting committee.

They ask the Master Chef to cook the dish a few times (expensive but accurate).
They ask the Sous Chefs to cook the dish many, many times (cheap and fast).
They compare the Sous Chefs' results to the Master Chef's results. Because the Sous Chefs are correlated with the Master Chef (they are all making the same dish, just with different techniques), the committee can use the Sous Chefs' massive amount of data to "correct" the Master Chef's small amount of data.

The result? They get an answer that is just as accurate as if they had only used the Master Chef, but they did it 10 to 100 times faster.

The Secret Sauce: Repackaging the Problem

To make this teamwork work, the authors had to do some clever math "repackaging."
Normally, calculating the learning score requires simulating the actual data you would get from an experiment (like a sensor reading). This is the expensive part.
The authors realized they could rewrite the math so that instead of simulating the sensor reading, they simulate the noise (the random errors) that goes into the sensor.

Before: "Simulate the weather, then see what the thermometer reads." (Expensive)
After: "Simulate the random wind gusts, then calculate what the thermometer would have read." (Cheaper and easier to mix with the Sous Chefs).

This change allowed them to use a statistical trick called Approximate Control Variates (ACV). Think of this as a way to mathematically cancel out the errors. If the cheap models are consistently "off" by a certain amount compared to the expensive model, the math can subtract that error out, leaving a perfect answer.

What They Found

They tested this method on two problems:

A Math Puzzle: A standard test case with a tricky, non-linear equation.
Real-World Physics: A simulation of turbulent air flow over a flat plate (like air moving over an airplane wing). They were trying to figure out the best place to put a pressure sensor to learn about the air turbulence.

The Results:

Unbiased: Their method didn't cheat. The final answer was just as accurate as the expensive method.
Massive Speedup: They reduced the "noise" (variance) in their results by 10 to 100 times compared to using the expensive model alone.
Smart Reuse: They found that if they let the cheap models "reuse" some of the same random numbers used by the expensive model, the teamwork became even more effective, cutting the error in half again.

The Bottom Line

This paper introduces a smart way to run experiments. Instead of waiting years for a super-computer to tell you the best way to test something, you can use a team of fast, slightly imperfect models to guide a few perfect, slow models. The result is that you find the best experimental setup much faster, saving time and money, without losing accuracy.

The authors have even made their code available for others to use, so scientists can apply this "teamwork" strategy to their own complex problems, from engineering to climate science.

Technical Summary: A Multi-fidelity Estimator of the Expected Information Gain for Bayesian Optimal Experimental Design

Problem Statement

Optimal Experimental Design (OED) aims to identify experimental conditions that maximize the information gained about model parameters. In a Bayesian framework, the standard objective is the Expected Information Gain (EIG), defined as the mutual information between model parameters and observation data. While analytically tractable for linear models with Gaussian priors, the EIG is intractable for nonlinear models. Consequently, it must be estimated numerically, typically via Nested Monte Carlo (NMC) sampling.

The primary challenge lies in the computational cost of these estimations. Evaluating the EIG requires repeated assessments of the data model (solving governing physical equations) under various parameter and noise realizations. For high-fidelity models, such as those involving complex fluid dynamics, this process becomes computationally prohibitive. While surrogate or reduced-order models can accelerate computation, simply substituting them introduces bias. Existing multi-fidelity methods, such as Multilevel Monte Carlo (MLMC), offer variance reduction but rely on strict assumptions regarding cost-accuracy relationships and convergence to a "truth model," limiting their flexibility.

Methodology

The authors propose a novel Multi-fidelity Expected Information Gain (MF-EIG) estimator built upon the Approximate Control Variate (ACV) framework. The methodology addresses two critical hurdles: the incompatibility of standard EIG formulations with ACV requirements and the need for variance reduction without introducing bias.

1. Reparameterization of the EIG

A fundamental requirement for the ACV framework is that the input samples for the estimator can be generated independently of the expensive model evaluations. Standard NMC estimators sample observation data $Y$ directly, which necessitates solving the forward model $Y = h(E; \Theta, \xi)$ for every sample.

To resolve this, the authors reparameterize the EIG by changing the integration variable from the observation data $Y$ to the underlying random data noise $E$ . Assuming the data model $h$ is injective and continuously invertible, the likelihood and evidence terms are rewritten in terms of $E$ and the Jacobian of the inverse mapping. This transformation allows the expectation to be taken over $Z = \{E, \Theta\}$ , where samples of $E$ and $\Theta$ can be drawn directly without evaluating the forward model. The utility function is consequently reformulated as $u_E(\xi, E, \Theta)$ , making it compatible with ACV.

2. The MF-EIG Estimator

The estimator combines a high-fidelity utility model ( $u_0$ ) with an ensemble of $M$ low-fidelity utility models ( $u_1, \dots, u_M$ ). These models can differ in:

Data Model Fidelity: Using different forward models (e.g., coarser grids, simplified physics).
Numerical Approximation: Using different inner-loop sample sizes or quadrature rules to estimate the intractable evidence term.

The MF-EIG estimator is defined as:
$\tilde{U}(\xi, z; \alpha, A) := \hat{U}_0(\xi, z_0) + \sum_{m=1}^M \alpha_m \left( \hat{U}_m(\xi, z^*_m) - \hat{U}_m(\xi, z_m) \right)$
where $\hat{U}_m$ are standard Monte Carlo estimators for each fidelity level, $z$ represents the allocated sample sets, and $\alpha$ are control variate weights. The estimator is unbiased with respect to the high-fidelity mean ( $u_0$ ) and minimizes variance under a given computational budget by optimizing the sample allocation $A$ and weights $\alpha$ .

3. Practical Enhancements

The paper details specific implementations for additive and scaled noise data models. It further introduces two practical enhancements:

Inner-loop Sample Size Optimization: The number of inner-loop samples ( $N_{in,m}$ ) for low-fidelity models is optimized to balance correlation and computational cost, as too few samples degrade correlation while too many yield diminishing returns.
Sample Reuse: To enhance correlation between utility models, the authors propose reusing inner-loop samples of the parameters $\Theta$ across different fidelity levels. This strategy significantly boosts the correlation structure required for effective variance reduction.

Key Contributions

Novel MF-EIG Estimator: The derivation of an unbiased multi-fidelity EIG estimator under the ACV framework that accommodates general ensembles of models without requiring specific cost-accuracy convergence assumptions.
Reparameterization Strategy: A mathematical reformulation of the EIG that decouples the expectation from the data model, enabling the application of ACV techniques to OED problems.
Optimization Techniques: The development of strategies for optimizing inner-loop sample sizes and implementing sample reuse to maximize variance reduction.
Implementation: The integration of the estimator into the MXMCPy toolbox, allowing for flexible allocation strategies that encompass existing MFMC and MLMC methods as special cases.

Numerical Results

The authors validate the MF-EIG estimator using two numerical examples:

Case 1: Nonlinear Benchmark

A one-dimensional nonlinear problem with additive and scaled noise.

Variance Reduction: The MF-EIG estimator achieved a 6–8 times variance reduction compared to single-fidelity NMC using a naive approach.
Optimization Impact: Optimizing inner-loop sample sizes provided an additional 6–12% improvement.
Sample Reuse: Implementing sample reuse yielded the most significant gains, achieving 17–23 times variance reduction (approximately double the improvement of the non-reuse case).

Case 2: Turbulent Flow Calibration

A multi-dimensional problem involving the calibration of Shear-Stress Transport (SST) turbulence closure model parameters within a Reynolds-averaged Navier–Stokes (RANS) framework. Four solvers with varying grid resolutions served as fidelity levels.

Variance Reduction: The MF-EIG estimators achieved one to two orders of magnitude (19–71 times) variance reduction across the design space compared to single-fidelity NMC.
Optimal Design Performance: At the specific optimal design location, the variance reduction was even more pronounced, reaching 94 to 233 times.
Physical Insights: The results highlighted that the most informative designs lie near the leading edge of a flat plate within the boundary layer, where the correlation between solvers is high but the physical sensitivity is maximal. The estimator successfully distinguished the optimal design with high confidence, whereas the single-fidelity estimator struggled due to large variance.

Significance and Claims

The paper claims that the MF-EIG estimator provides a robust, unbiased, and highly efficient alternative to traditional single-fidelity EIG estimation. By leveraging the ACV framework, the method minimizes variance under a fixed computational budget without the restrictive convergence assumptions required by MLMC.

The authors emphasize that the estimator is unbiased with respect to the high-fidelity mean, ensuring that the optimization of experimental designs is not skewed by the multi-fidelity approximation. The results demonstrate that the method can reduce computational costs by one to two orders of magnitude while maintaining accuracy. The paper concludes that inner-loop sample reuse is a critical technique for maximizing the benefits of the multi-fidelity approach, and that optimizing inner-loop sample sizes offers moderate but valuable additional gains.

The work positions MF-EIG as a practical tool for complex engineering applications, such as turbulent flow modeling, where high-fidelity simulations are expensive, and efficient experimental design is crucial.