An information-matching approach to optimal experimental design and active learning

This paper introduces a scalable, convex optimization-based information-matching approach using the Fisher Information Matrix to select optimal training data that specifically constrains parameters relevant to downstream quantities of interest, thereby enabling precise predictions with minimal data across diverse scientific fields and active learning applications.

The Gist

Imagine you are trying to teach a robot to predict the weather. You have a massive library of historical data: temperature, humidity, wind speed, cloud cover, barometric pressure, and even the number of birds flying south.

The Problem:
You can't just feed the robot everything. Collecting all that data is expensive, slow, and sometimes impossible (like measuring the temperature inside a volcano). Plus, the robot might get confused. It might spend all its brainpower trying to perfectly memorize the exact number of birds, which doesn't actually help it predict if it will rain tomorrow. In the world of math, these confusing, unnecessary details are called "sloppy" parameters.

The Old Way:
Traditional methods for choosing data are like a chef who says, "I need to taste every single ingredient in the pantry to make sure I know exactly how much salt, pepper, and sugar is in the whole house." They try to make every number in their model perfect. This is inefficient and often overkill.

The New Way (Information Matching):
This paper introduces a smarter strategy called Information Matching. Think of it like a detective solving a specific crime.

1. Define the Goal: First, the detective asks, "What exactly do I need to know?" Maybe they just need to know who committed the crime, not the exact brand of shoes the suspect was wearing.
2. The "Fishing Net" Analogy: Imagine you are fishing.
   • Old Method: You cast a giant net that catches everything in the ocean—fish, seaweed, old boots, and plastic bottles. You then spend hours sorting through the trash to find the fish.
   • New Method (Information Matching): You look at the fish you want to catch (the "Quantities of Interest"). You then design a net with holes exactly the right size to let the seaweed and boots pass through, but trap the fish. You only catch what you need.

How It Works in Real Life:
The authors use a mathematical tool called the Fisher Information Matrix (think of it as a "usefulness meter").

• Scenario 1: Power Grids (The Electrical Map):
  Imagine a city's power grid. You need to know the voltage at every street corner to keep the lights on. But installing sensors (PMUs) at every single corner costs millions.

  • The Solution: The algorithm figures out that if you put sensors at just three specific intersections, you can mathematically "see" the voltage everywhere else. It ignores the sensors that don't add new information, saving huge amounts of money.

• Scenario 2: Underwater Sound (The Ocean Echo):
  Imagine trying to find a lost submarine using sound. The ocean is messy; the water temperature and the type of sand on the sea floor change how sound travels.

  • The Solution: Instead of trying to map the entire ocean's temperature and sand composition (which is impossible), the algorithm picks specific spots to place microphones. These spots are chosen specifically because the sound patterns there will tell you exactly where the submarine is, without needing to know the exact temperature of the water 10 miles away.

• Scenario 3: Building Materials (The Lego Set):
  Scientists want to predict how a new material (like a super-strong metal) will behave. They need to run expensive computer simulations to train their models.

  • The Solution: Instead of running 2,000 different simulations, the algorithm says, "You only need to run these 7 specific simulations." These 7 are the "golden" ones that contain all the necessary information to predict the material's strength. The other 1,993 are just noise.

The "Active Learning" Loop:
The paper also describes a "learning loop." Imagine you are taking a test.

1. You guess the answers.
2. The teacher (the algorithm) looks at your guess and says, "You are shaky on Question 5. Let's study Question 5 specifically."
3. You study Question 5, update your knowledge, and take the test again.
4. The teacher says, "Great, now you're shaky on Question 12."
5. You repeat until you can answer the specific questions that matter with perfect confidence, without wasting time studying the whole textbook.

The Big Takeaway:
This method is a game-changer because it stops us from trying to be perfect at everything. Instead, it focuses on being perfect at what matters.

It tells us: "You don't need to know everything about the system to predict the outcome. You just need the right few pieces of information." This saves time, money, and computing power, making it possible to build better models for everything from climate change to new medicines, using a tiny fraction of the data we thought we needed.

Technical Summary

1. Problem Statement

Mathematical models in scientific fields often rely on training data to infer parameters. However, data collection is frequently expensive, resource-intensive, or physically constrained. Furthermore, many models (particularly "sloppy" models) contain numerous parameters that are practically unidentifiable or redundant.

• The Core Issue: Traditional Optimal Experimental Design (OED) and Active Learning (AL) methods often focus on minimizing the uncertainty of all model parameters (e.g., via A-, D-, or E-optimality based on the Fisher Information Matrix).
• The Limitation: In many predictive applications, precise estimation of every parameter is unnecessary. The primary goal is often to predict specific Quantities of Interest (QoIs) with high precision. If the parameters relevant to the QoIs are not constrained, prediction uncertainty remains high, even if other irrelevant parameters are well-constrained. Conversely, constraining irrelevant parameters wastes resources.
• The Challenge: How to select a minimal subset of training data that provides just enough information to constrain the specific parameter combinations required for precise QoI predictions, while ignoring unidentifiable or irrelevant parameters.

2. Methodology: Information-Matching

The authors propose an Information-Matching criterion that aligns the information content of the training data with the information required to achieve a target precision for the QoIs.

Theoretical Framework

• Fisher Information Matrix (FIM): The method utilizes the FIM, $I(\theta)$, which quantifies the information about parameters $\theta$ contained in the data. For weighted least squares, $I(\theta) = \sum w_m J_f^T J_f$, where $J_f$ is the Jacobian of the model $f$ with respect to parameters.
• QoI FIM: A target FIM, $J(\theta)$, is defined based on the desired precision (covariance $\Sigma$) of the QoIs, calculated as $J(\theta) = J_g^T \Sigma^{-1} J_g$, where $J_g$ is the Jacobian of the QoI mapping $g$.
• The Optimization Problem: The goal is to find a weight vector $w$ (representing the selection and precision of data points from a candidate pool) that minimizes the number of data points used while ensuring the training data's information dominates the QoI's required information.
  $$\begin{aligned} & \text{minimize} & & \|w\|_1 \\ & \text{subject to} & & w_m \geq 0 \\ & & & I(\theta) = \sum w_m I_m(\theta) \succeq J(\theta) \end{aligned}$$
  • Objective: Minimizing the $\ell_1$-norm encourages sparsity, selecting the fewest necessary data points.
  • Constraint: The matrix inequality $I \succeq J$ (meaning $I - J$ is positive semidefinite) ensures that the parameter uncertainty derived from the selected data is smaller than (or equal to) the uncertainty required to meet the QoI target precision.

Active Learning Loop

The method is integrated into an iterative Active Learning loop (Algorithm 1):

1. Initialize parameters and a pool of candidate inputs.
2. Compute the QoI FIM ($J$) and candidate data FIMs ($I_m$) at current parameters.
3. Solve the convex optimization problem to find optimal weights.
4. Generate labels (ground truth) for data points with non-zero weights.
5. Update model parameters using the new data.
6. Repeat until convergence.

3. Key Contributions

1. Shift in Objective: Moves OED from "minimizing global parameter variance" to "matching information to QoI precision." This acknowledges that in sloppy models, only specific parameter subspaces matter for prediction.
2. Convex Formulation: The problem is formulated as a convex optimization problem (Semidefinite Programming), making it scalable to large models and datasets.
3. Theoretical Guarantee: The authors prove (Theorem 1) that if the information-matching constraint is satisfied, the propagated uncertainty of the QoIs is guaranteed to be within the target precision (up to third-order terms), even if the full parameter set is not uniquely identifiable.
4. Handling Ill-Conditioning: By focusing only on the subspace relevant to QoIs, the method bypasses numerical instability issues often associated with ill-conditioned FIMs in sloppy models.

4. Results and Applications

The method was validated across three distinct scientific domains:

• Power Systems (PMU Placement):

  • Task: Determine optimal locations for Phasor Measurement Units (PMUs) to observe the state of an IEEE 39-bus power grid.
  • Result: The method successfully identified the minimal set of buses required for full observability, matching previous literature results. Crucially, it also solved the "partial observability" problem, identifying optimal sensor sets for specific sub-networks without needing to observe the entire grid, demonstrating flexibility in defining QoIs.

• Underwater Acoustics (Source Localization):

  • Task: Localize two sound sources in a shallow ocean using passive hydrophones, despite unknown environmental parameters (sediment, temperature).
  • Result: The method selected receiver locations covering only 5% of the candidate grid. It achieved the target localization accuracy without needing to fully invert (precisely estimate) the complex ocean environmental parameters, focusing only on the parameter combinations necessary for source location.

• Materials Science (Interatomic Potentials):

  • Task: Develop a Stillinger-Weber potential for Molybdenum Disulfide (MoS$_2$) using Active Learning.
  • Result: Starting from a pool of 2,000 atomic configurations, the algorithm selected only 7 configurations to train the potential. These 7 configurations were sufficient to constrain the potential parameters to predict the energy-strain relationship with the target precision (10% of the full-model prediction).
  • Robustness: Tests showed that while the specific selected configurations varied with initial parameter guesses, the final prediction uncertainty consistently satisfied the target constraint.

5. Significance and Future Impact

• Efficiency: The approach drastically reduces the cost of data acquisition by identifying that a very small subset of data is often sufficient for precise predictions.
• Interpretability: By focusing on relevant parameter combinations, it improves the interpretability of the model training process.
• Scalability: The convex nature of the optimization allows the method to scale to large datasets and complex models, including potential applications in machine learning.
• Broad Applicability: The framework is applicable to any domain with "sloppy" models where the goal is prediction rather than full parameter identification, such as biology, neuroscience, geology, and atmospheric science.

In summary, this paper introduces a mathematically rigorous and practically efficient method to bridge the gap between data collection costs and predictive accuracy, ensuring that experimental resources are spent only on the information necessary to achieve specific scientific goals.