Experimental Design for Missing Physics

This paper proposes a sequential experimental design technique that optimally discriminates between plausible model structures identified by universal differential equations and symbolic regression to efficiently discover missing physics in process systems, demonstrated through its application to a bioreactor.

The Gist

Imagine you are a detective trying to solve a mystery, but you only have a partial map of the crime scene. You know the general layout of the building (the laws of physics), but there's a specific room where the action happens that you can't see into. You know something is happening in there, but you don't know the rules of that room.

This is exactly the problem scientists face with bioreactors (like giant vats used to grow bacteria or yeast). They know how the liquid moves and how the tank fills up, but they don't fully understand the "secret sauce" that makes the bacteria grow. This missing piece of knowledge is called "Missing Physics."

Here is how the authors of this paper solved the mystery, broken down into simple steps:

1. The "Black Box" Detective (Universal Differential Equations)

First, the scientists needed a way to guess what was happening in that invisible room. They used a tool called a Universal Differential Equation (UDE).

Think of a UDE as a super-smart, shape-shifting robot.

• The scientists tell the robot: "We know how the tank fills up, but we don't know how the bacteria grow. You figure out the growth rule."
• The robot uses a Neural Network (a type of AI) to try millions of different growth rules until it finds one that fits the data it has seen so far.
• The Problem: The robot is great at guessing, but it's a "black box." It gives you the answer, but it speaks in a language of numbers and weights that no human can understand. It's like the robot saying, "The bacteria grow because of 4,502 specific math operations," which isn't very helpful for a scientist.

2. The Translator (Symbolic Regression)

To make the robot's answer useful, the scientists used a second tool called Symbolic Regression.

Think of this as a translator or a poet.

• The translator looks at the robot's complex, messy answer and tries to rewrite it into a simple, elegant sentence (a mathematical formula) that a human can read.
• Instead of "4,502 operations," the translator might say, "Ah, the bacteria grow according to the Monod Equation (a famous formula in biology)."
• However, the translator doesn't just give you one answer. It gives you a shortlist of the top 10 most likely sentences that could explain the data. Some might be simple, some complex, but they all fit the data the robot has seen so far.

3. The "Taste Test" (Optimal Experimental Design)

Here is the tricky part: The translator gave them 10 different sentences. Which one is the true rule? They can't tell just by looking at the data they already have. They need to run a new experiment to see which sentence is right.

But running a random experiment is like tasting soup without a spoon; you might miss the flavor. You need to design the experiment specifically to force a disagreement between the top 10 sentences.

The authors developed a method to act like a tough judge:

• They ask: "If I add a huge amount of food to the tank right now, which of these 10 sentences predicts the bacteria will grow the fastest? Which predicts they will die?"
• They then design the experiment (controlling the flow of food) to create a scenario where the top 10 sentences disagree wildly.
• By running the experiment in this specific way, they can immediately see which sentences are wrong. The ones that predicted the wrong outcome get crossed off the list.

4. The Loop (The "Smart" Cycle)

The process is a cycle, like a video game level:

1. Guess: The robot (UDE) learns from current data.
2. Translate: The poet (Symbolic Regression) writes down the top 10 possible rules.
3. Test: The judge (Optimal Design) creates a specific experiment to see which rule is wrong.
4. Repeat: They run the experiment, get new data, and start over.

The Result: Why It Matters

The authors tested this on a bioreactor.

• Random Approach: If they just ran random experiments (like flipping a coin to decide how much food to add), they ran 5 different sets of experiments and never found the true rule (the Monod equation).
• Smart Approach: Using their "Judge" method, they only needed 3 experiments to find the true rule with high confidence.

The Big Picture

In simple terms, this paper teaches us how to stop guessing and start strategically learning. Instead of throwing data at a wall and hoping something sticks, the scientists built a system that asks the right questions to eliminate the wrong answers quickly.

It's the difference between trying to find a needle in a haystack by grabbing random handfuls of hay, versus using a magnet that is specifically designed to pull the needle out in just three tries.

Technical Summary

1. Problem Statement

In the analysis and control of process systems (e.g., bioreactors), mathematical models often suffer from incomplete knowledge of the underlying physics. While conservation laws and transport phenomena are known, specific functional relationships (e.g., reaction kinetics) may be unknown. These gaps are termed "missing physics."

Traditional Model-Based Design of Experiments (MbDoE) focuses on:

1. Parameter Precision: Optimizing data to estimate parameters within a known model structure.
2. Model Discrimination: Distinguishing between a finite, pre-defined set of candidate structures.

The Gap: Neither approach works when the functional form of a part of the model is entirely unknown. Furthermore, machine learning techniques like Universal Differential Equations (UDE) require high-quality, informative data to recover these unknown structures, but standard experimental designs do not account for the specific needs of discovering unknown functions.

2. Methodology

The authors propose a sequential experimental design framework that integrates Universal Differential Equations (UDE), Symbolic Regression (SR), and Optimal Experimental Design (OED) to iteratively discover missing physics.

A. System Formulation

The system is modeled as a set of Ordinary Differential Equations (ODEs) where a specific term $\phi$ represents the unknown physics:
$$\frac{dx}{dt} = f(t, x, \phi(g(x)), u(t))$$

• $x$: Dynamic states (e.g., substrate, biomass).
• $u(t)$: Controllable inputs (e.g., feed rate).
• $\phi(g(x))$: The missing physics function, where $g(x)$ is a known preprocessing function (e.g., selecting specific states).

B. The Iterative Loop

The methodology operates in a sequential loop consisting of four main steps:

1. Universal Differential Equation (UDE) Training:

   • The unknown function $\phi$ is replaced by a Neural Network (NN).
   • The UDE is trained on available experimental data to minimize the loss between predicted and measured states.
   • Note: NNs are data-hungry and lack interpretability.

2. Symbolic Regression (SR):

   • To make the NN interpretable, SR is applied to the trained NN.
   • SR searches the space of mathematical expressions (using tree structures and genetic algorithms) to find human-readable equations that approximate the NN's output.
   • This generates a set of plausible model structures (candidates for the true physics).

3. Optimal Experimental Design (Model Discrimination):

   • Instead of optimizing for parameter precision, the design optimizes for discrimination between the plausible structures generated by SR.
   • Criterion: A modified T-optimal design criterion is used. Since no "ground truth" is known, the algorithm maximizes the average pairwise distance between the predicted outputs of all candidate models at future measurement times.
   • Control Optimization: The input $u(t)$ is optimized (restricted to piecewise constant functions) to maximize this discrimination metric. The goal is to find conditions where the candidate models diverge most significantly.

4. Data Gathering & Retraining:

   • The optimal experiment is performed, generating new data.
   • The UDE is retrained with the new data, and the cycle repeats, refining the set of plausible models.

3. Key Contributions

• Novel Sequential Framework: The paper introduces a specific workflow for discovering unknown functional forms (missing physics) rather than just tuning parameters or choosing between fixed models.
• Integration of ML and OED: It bridges the gap between data-driven discovery (UDE + SR) and rigorous experimental design, ensuring that data collection is specifically targeted to resolve ambiguities in the discovered mathematical structures.
• Discrimination Criterion for Unknown Structures: The authors develop a specific objective function (Eq. 8) that maximizes the divergence between a set of learned candidate models, enabling the system to "prune" incorrect hypotheses efficiently.
• Open Source Implementation: The methodology is implemented in Julia (ModelingToolkit, DifferentialEquations, SymbolicRegression) and made publicly available.

4. Results (Bioreactor Case Study)

The method was tested on a fed-batch bioreactor where the specific growth rate $\mu(C_s)$ was unknown (True form: Monod equation).

• Initial State: An initial experiment with a zero control signal provided data only for low substrate concentrations. The UDE and SR identified several plausible models, but they diverged significantly in the medium-to-high concentration range (where no data existed).
• Second Experiment: The OED algorithm determined that the highest possible feed rate would maximize the difference between the candidate models. This experiment gathered data at high substrate concentrations, effectively eliminating models that predicted constant or linear growth.
• Third Experiment: The algorithm identified a "medium" concentration range where remaining candidates still disagreed. The optimal control created a "staircase" feed profile to maintain the reactor in this specific region, further discriminating between the remaining hypotheses.
• Outcome: After three sequential experiments, the Monod equation emerged as the top-ranked model with the highest score.
• Comparison: When compared to 5 random experiments (with controls drawn from a uniform distribution), the optimal design successfully recovered the Monod equation, whereas the random experiments failed to identify the correct structure in any of the 5 trials.

5. Significance and Future Directions

• Scientific Impact: This approach moves beyond "black box" machine learning. It uses ML to propose hypotheses and OED to rigorously test and refine them, resulting in interpretable, physics-compliant models.
• Efficiency: It demonstrates that significantly fewer experiments are needed to discover complex physics when the experimental design is optimized for structural discrimination rather than random data collection.
• Future Work:
  • Extending the method to systems with multiple unknown inputs/outputs.
  • Combining missing physics discovery with parameter estimation (multi-objective design).
  • Automatic Hyperparameter Tuning: Currently, hyperparameters for the NN and SR are fixed. Future research aims to create a fully online system where these hyperparameters are tuned automatically during the experiment.

In summary, the paper presents a robust, iterative framework that leverages the flexibility of neural networks and the interpretability of symbolic regression, guided by optimal experimental design, to solve the critical problem of discovering unknown physical laws from data.