Neural-ISAM: A hybrid in-situ machine learning approach for complex manifold-based combustion models in LES of turbulent flames

This paper introduces Neural-ISAM, a hybrid in-situ machine learning method that dynamically replaces pruned regions of adaptive tabulation databases with trained neural networks to significantly reduce memory requirements while maintaining accuracy in large-eddy simulations of complex turbulent flames.

The Gist

Imagine you are trying to simulate a complex, swirling fire on a supercomputer. To do this accurately, the computer needs to know the exact temperature, chemical makeup, and pressure of the air at millions of tiny points every single second.

The Problem: The "Too Big to Carry" Library
Traditionally, scientists solve this by creating a massive "library" of pre-calculated answers for every possible fire scenario. Think of this like a giant encyclopedia where every page is a different fire condition.

• The Issue: As the fire models get more realistic (adding soot, radiation, complex chemistry), this encyclopedia becomes so huge it won't fit in the computer's memory. It's like trying to carry the entire Library of Congress in your backpack while running a marathon.

The First Solution: The "Just-in-Time" Notebook (ISAM)
To fix the memory issue, scientists developed a method called ISAM. Instead of carrying the whole library, the computer only writes down the answers it actually needs while it runs the simulation. It keeps these answers in a smart, organized notebook (a binary tree).

• How it works: If the computer needs an answer it hasn't seen before, it calculates it and writes it down. If it sees a similar situation later, it uses a quick shortcut (a linear guess) based on what it wrote down.
• The New Problem: Even this notebook gets too full if the fire is very complex. The computer runs out of space again.

The New Solution: The "Smart Summarizer" (Neural-ISAM)
This paper introduces Neural-ISAM, a hybrid approach that combines the "just-in-time" notebook with Artificial Intelligence (Neural Networks).

Here is the analogy:
Imagine your notebook is getting too heavy. You decide to hire a smart assistant (the Neural Network) to summarize specific chapters of your notebook.

1. Scanning for Summaries: The computer scans its notebook to find sections that are very crowded with data (lots of similar fire conditions).
2. Training the Assistant: For these crowded sections, the computer takes the data and trains a small, compact AI model to "memorize" that specific chapter.
3. The Swap: Once the AI is trained, the computer deletes the heavy pages of the notebook for that section and replaces them with the tiny AI model.
   • The Result: The AI model is like a tiny flash drive that holds the same information as a thick book. This drastically shrinks the memory footprint.

How the Training Works (The "Safe Zone" Trick)
The paper highlights a clever way to train these AI assistants without needing to pre-calculate millions of scenarios:

• The computer looks at the "safe zones" (called Ellipsoids of Accuracy) it already calculated in its notebook.
• It generates new training data by sampling points inside these safe zones.
• Because these points are inside the safe zones, the computer doesn't need to do expensive new calculations; it just uses its existing shortcuts to generate the training data.
• The AI learns to mimic the notebook's behavior in that specific area, and then the notebook pages are deleted.

The Results: What Happened?
The authors tested this on two types of turbulent flames (Sandia Flame D and a Sooting Flame).

• Memory Savings:

  • For the simpler flame, they reduced memory usage by about 14% to 20%.
  • For the complex "sooting" flame (which has more variables like soot and heat loss), they reduced memory by 34% to 38%.
  • Crucial Finding: If they tried to summarize too much (pruning too aggressively), the AI models actually took up more space than the original notebook because the models had to be too complex. They had to find a "Goldilocks" zone.

• Speed vs. Accuracy:

  • Accuracy: The results were very accurate. The AI summaries matched the original calculations almost perfectly, with only tiny, barely noticeable errors in specific chemical amounts.
  • Speed: There is a trade-off.
    • Training: It takes time to train the AI assistants (the "summarizing" step).
    • Running: Once trained, looking up an answer in the AI model takes slightly longer (about 10 microseconds) than looking it up in the original notebook (about 5 microseconds). However, because the AI is so much smaller, it fits in the computer's fast memory, preventing the simulation from crashing due to lack of space.

In Summary
Neural-ISAM is a method that lets scientists run complex fire simulations that would otherwise be too big for their computers. It does this by letting the computer build a database as it goes, and then periodically replacing the heaviest parts of that database with tiny, trained AI models. This saves massive amounts of memory, allowing for more realistic simulations, though it requires a bit more computing power to run the AI models during the simulation.

Technical Summary

Technical Summary: Neural-ISAM

Problem Statement
Manifold-based combustion models are widely used in Large Eddy Simulations (LES) of turbulent flames to reduce computational costs by projecting thermochemical states onto lower-dimensional manifolds. Traditionally, these models rely on precomputed and pretabulated solutions. However, as manifold complexity increases (e.g., higher dimensions, complex chemistry, non-adiabatic effects), the memory requirements for these tables become prohibitive.

Alternative approaches include In-Situ Adaptive Tabulation (ISAT) and In-Situ Adaptive Manifolds (ISAM), which build databases dynamically during simulation, storing only encountered states. While ISAM reduces memory compared to full pretabulation, memory usage can still grow excessively for complex models. Conversely, machine learning (ML) methods, particularly neural networks, offer memory-compact representations of complex functions. However, standard ML approaches require pregenerating training datasets covering the entire parameter space, which defeats the purpose of in-situ efficiency for high-dimensional problems where not all states are encountered.

Methodology: Neural-ISAM
This work introduces Neural-ISAM, a hybrid framework that couples adaptive tabulation with in-situ machine learning to address the memory limitations of ISAM while avoiding the precomputation requirements of standard ML. The method operates as follows:

1. Initial ISAM Operation: The simulation begins using the standard ISAM approach, where ISAT stores manifold solutions in a binary tree structure. Each leaf contains a solution, its Jacobian, and an Ellipsoid of Accuracy (EOA) defining the region where linear extrapolation is valid.
2. Pruning Candidate Identification: Periodically, the binary tree is traversed to identify candidate nodes for pruning. A node is selected if it satisfies two user-specified thresholds:
   • Coverage Density ($\rho$): A metric quantifying the density of EOAs within a node's region relative to the bounding box volume. High density ensures sufficient training data.
   • Leaf Count ($N_L$): A minimum number of leaves below the node to ensure the pruned region is significant enough to warrant replacement.
3. In-Situ Training Data Generation: For selected candidate nodes, training data is generated by sampling random points within the node's Axis-Aligned Bounding Box (AABB). Points are accepted only if they fall within an EOA of a leaf beneath the candidate node. This allows the generation of training data via linear extrapolation from existing leaves, requiring no new explicit manifold calculations.
4. Neural Network Training:
   • Data Transformation: Thermochemical profiles are transformed by subtracting mean profiles and scaling residuals by standard deviation. For non-adiabatic cases involving heat loss, an inverse hyperbolic sine ($\text{arcsinh}$) scaling is applied to handle large dynamic ranges.
   • Architecture Optimization: Bayesian Optimization is used to tune the number of layers and an expansion factor determining neuron counts per layer.
   • Replacement: Once trained, the binary tree structure beneath the candidate node is removed (pruned), and the trained neural network is assigned to that node. Future queries in this region are resolved via neural network evaluation rather than tree traversal.

Key Contributions

• Hybrid Framework: The development of Neural-ISAM, which dynamically replaces portions of an ISAT database with neural networks, effectively combining the "learn-as-you-go" capability of ISAT with the memory compactness of neural networks.
• In-Situ Training Strategy: A novel method for generating training data without precomputation, utilizing the existing ISAT structure to sample and validate data points within specific regions of the manifold space.
• Adaptive Pruning Criteria: The introduction of coverage density and leaf-count thresholds to determine optimal regions for pruning, balancing memory reduction against prediction accuracy.

Results
The method was evaluated using LES of two turbulent flames: Sandia Flame D (nonpremixed, 1D manifold input) and the Sandia Sooting Flame (non-adiabatic, soot formation, 2D manifold input).

• Memory Reduction:
  • For Sandia Flame D, pruning reduced database memory by 14–20%, depending on the thresholds.
  • For the more complex Sandia Sooting Flame, high thresholds ($N_{L,thresh} = 350$) achieved a memory reduction of approximately 34–38%. However, setting $N_{L,thresh}$ too low (e.g., 90) resulted in increased memory usage due to the overhead of numerous small neural networks.
• Accuracy:
  • Conditional statistics for temperature and species mass fractions generally matched the baseline ISAM results.
  • Slight underpredictions were observed for $Y_{OH}$ and $Y_{CO}$ in cases with aggressive pruning.
  • In the Sooting flame, accuracy for the heat loss parameter ($H$) degraded in regions with sparse training data (between EOAs), particularly when using low coverage density thresholds. Higher coverage density thresholds ($\rho_{thresh} = 0.8$) improved accuracy but reduced the number of pruned regions.
• Computational Performance:
  • Training: Training time was significant but manageable, scaling with the number of neural networks trained (higher for lower $N_{L,thresh}$).
  • Runtime: Post-pruning, the time per timestep increased slightly compared to the baseline ISAM. This is because a single neural network evaluation (10 $\mu$s) is computationally more expensive than a primary ISAT retrieve via linear extrapolation (5 $\mu$s). The increase in timestep duration correlated with the number of neural network evaluations required per step.

Significance and Claims
The paper claims that Neural-ISAM successfully addresses the trade-off between memory usage and model complexity in manifold-based combustion modeling. It demonstrates that replacing portions of adaptive tabulation databases with neural networks can significantly reduce memory footprints, particularly for complex, high-dimensional manifolds where traditional tabulation fails.

The authors modestly note that while memory is reduced, there is a computational cost associated with neural network evaluations. They identify that prediction accuracy can suffer in regions of sparse data (between EOAs), suggesting that future work is needed to improve model performance in these specific areas and to optimize the evaluation speed of the neural networks within the ISAT framework. The method is presented as a viable path forward for simulating increasingly complex combustion systems without the prohibitive memory costs of full pretabulation or the precomputation requirements of standard machine learning approaches.