Assessing the Numerical Stability of Physics Models to Equilibrium Variation through Database Comparisons

This paper evaluates the numerical stability of physics models by comparing a large database of manually reconstructed DIII-D kinetic equilibria against automated CAKE and JAKE tools, finding that while scalar parameters agree well, profile quantities like bootstrap current show significant discrepancies, though ideal kink stability classifications remain robust in 90% of cases.

The Gist

Imagine you are trying to bake the perfect chocolate cake. You have a recipe (the physics of the plasma), but the ingredients you measure (the data from sensors) are a bit fuzzy, and the oven temperature fluctuates.

In the world of nuclear fusion, scientists use giant donut-shaped machines called tokamaks to try to create star-like energy. To make this work, they need to know the exact shape and behavior of the super-hot plasma inside. This is called finding the "equilibrium."

For decades, experts have been manually building these "recipes" by hand, tweaking numbers until the model looks right. It's like a master chef tasting the soup and adding a pinch of salt here or a dash of pepper there. But this is slow, and different chefs might make slightly different versions of the same soup.

Recently, scientists built automated robots (called CAKE and JAKE) to write these recipes instantly. The big question this paper asks is: "If we let the robots do the cooking, will the cake still taste the same? And if the recipe changes slightly, will the cake collapse?"

Here is a simple breakdown of what they found:

1. The "Manual" vs. The "Robots"

The researchers took a massive database of 596 different plasma "cakes" (shots) from the DIII-D tokamak.

• The Manual Group: These were the "Master Chefs." They spent hours or days perfecting each recipe.
• The Robot Group (CAKE): This is the new, fast automated tool. It does the same job in minutes.
• The Robot Group (JAKE): This is another automated tool, but it's a bit rougher around the edges, like a newer, less experienced robot.

2. The Taste Test (Scalar Parameters)

First, they compared the basic stats of the cakes: How big is it? How much pressure is inside? How strong is the magnetic field?

• The Result: For the big, obvious numbers (like the size of the donut or the total current), the robots and the chefs agreed very well. It's like if you asked three people to measure a table; they'd all say it's about 6 feet long.
• The Problem: When they looked at the details—like the temperature right at the edge of the cake or the specific flow of electricity inside—the robots and chefs started to disagree. The "JAKE" robot was especially messy, producing recipes that were quite different from the chefs. Even the "CAKE" robot had some differences in the fine details.

3. The Stability Test (Will the Cake Collapse?)

This is the most critical part. In fusion, if the plasma shape is slightly off, the whole thing can become unstable and crash (like a wobbly tower of Jenga blocks). The scientists used two different "crash detectors" to see if the robots' recipes would cause a disaster.

• Detector A (The "Kink" Detector - DCON): This checks if the plasma will twist and snap.

  • The Finding: The robots and the chefs agreed 90% of the time. If the chef said "Safe," the robot usually said "Safe."
  • The Catch: Even a tiny difference in the recipe could change the stability score. However, the robots were generally reliable enough for this specific test.

• Detector B (The "Tearing" Detector - STRIDE): This checks if the plasma will rip apart like a piece of paper.

  • The Finding: This was a disaster. The robots and chefs disagreed wildly. Sometimes the chef said "Safe," and the robot said "Rip apart!" The numbers were off by factors of 10 or even 100.
  • Why? This detector is extremely sensitive to the shape of the ingredients (the gradients). Since the robots calculated the ingredient shapes slightly differently than the chefs, the "rip" detector went haywire. It's like a very sensitive smoke alarm that goes off if you just toast a piece of bread too darkly.

4. The "Knob" Experiment

To see how sensitive these robots are, the scientists tweaked the settings on the CAKE robot (like changing the temperature at the edge or how smooth the curves are).

• The Result: A tiny tweak to a setting caused huge changes in the final recipe. It's like turning a dial on a radio just a fraction of a millimeter and suddenly hearing a completely different station. This proves that the "fine print" of the recipe matters a lot.

The Big Takeaway

The paper concludes that while automated tools (like CAKE) are amazing for speed and consistency, they aren't perfect yet.

• Good News: For general safety checks, the robots are doing a great job.
• Bad News: For detailed, high-stakes physics (like predicting exactly when the plasma might tear), the robots can give very different answers than the human experts.

The Lesson for the Future:
Scientists shouldn't just rely on one "recipe." If you are making a decision about a nuclear reactor, you should check the recipe with the human chef, the CAKE robot, and maybe even the JAKE robot. If they all agree, you're probably safe. If they disagree, you need to be very careful.

In short: Automation is fast and mostly accurate, but in the high-stakes world of fusion energy, we still need to double-check the fine print to make sure the cake doesn't collapse.

Technical Summary

1. Problem Statement

High-fidelity kinetic equilibria are fundamental to tokamak modeling, serving as the foundation for downstream physics analyses such as turbulence transport, turbulence, and Magnetohydrodynamic (MHD) stability. However, constructing these equilibria traditionally relies on manual workflows performed by physics experts, which are:

• Time-consuming: Requiring hours of labor per shot.
• Subject to User Bias: Different experts prioritize different physics goals (e.g., core safety factor vs. pedestal pressure), leading to inconsistencies.
• Lacking Uncertainty Quantification (UQ): It is difficult to quantify the general uncertainty arising from reconstruction techniques when relying on single, manually curated datasets.

As fusion devices become more complex (e.g., ITER, DEMO), the risk of underestimating uncertainties in equilibrium reconstruction could lead to costly errors in predicting plasma stability and performance. While automated tools (CAKE, JAKE) have been developed to standardize this process, their numerical stability and impact on critical physics outputs compared to "gold standard" manual reconstructions had not been rigorously quantified at a large database scale.

2. Methodology

The authors conducted a comprehensive database comparison using data from the DIII-D tokamak involving three distinct equilibrium reconstruction sources:

1. Manual Expert Equilibria: A historical dataset of 1,336 unique timeslices from 596 shots (2012–2023) reconstructed by experts using the PyD3D suite. These were filtered for convergence (Picard iteration error $< 10^{-8}$).
2. CAKE (Consistent Automatic Kinetic Equilibrium): An automated tool using a workflow of profile fitting, ONETWO kinetic constraints, and EFIT optimization. A corresponding dataset was generated for the same shots.
3. JAKE: An automated workflow based on the kineticEFITtime module in OMFIT, generated for a specific 2019 campaign.

Analysis Approach:

• Scalar Parameter Comparison: The authors compared scalar parameters (e.g., plasma current $I_p$, toroidal field $B_T$, safety factor $q$, pressure profiles) between datasets using percent differences.
• Sensitivity Analysis: They scanned specific CAKE settings (e.g., LCFS electron temperature, spline knot locations, GS error minimization) to determine how internal parameter variations affect the final equilibrium.
• MHD Stability Assessment: The reconstructed equilibria were fed into two stability codes:
  • DCON: To calculate $\delta W$ (ideal kink stability).
  • STRIDE: To calculate $\Delta'$ (classical tearing mode stability).
• Case Studies: Artificial scans of physical parameters ($\beta_N$, $l_i$, triangularity) were performed to establish a baseline for how much $\delta W$ changes due to physical variations versus reconstruction variations.

3. Key Contributions

• Large-Scale Database Benchmark: This is one of the first studies to compare thousands of equilibrium pairs (Manual vs. Automated) rather than isolated case studies.
• Quantification of Reconstruction Uncertainty: The paper provides statistical metrics (Mean $\pm$ Standard Deviation) for the variance introduced by different reconstruction methods across a wide range of plasma scenarios.
• Stability Code Sensitivity: It explicitly links variations in equilibrium reconstruction to the numerical stability of downstream MHD codes, distinguishing between robust metrics ($\delta W$) and highly sensitive ones ($\Delta'$).
• Identification of Critical Parameters: The study isolates specific reconstruction settings (e.g., LCFS temperature, FF' spline knots) that disproportionately influence equilibrium profiles and stability metrics.

4. Key Results

A. Scalar and Profile Parameters

• Global Parameters: Good agreement was found for global shape and current parameters (e.g., $I_p$, $B_T$, Major Radius $R_0$) between Manual and CAKE.
• Profile Quantities: Significant disagreement was observed in profile-dependent quantities, particularly bootstrap current ($j_{boot}$), edge pressure ($p_{edge}$), and safety factor ($q_0$).
• JAKE vs. CAKE: The JAKE automated tool showed substantially higher variance and lower fidelity compared to both Manual and CAKE datasets, particularly in global parameters like $R_0$.
• CAKE Sensitivity: Small changes in CAKE settings (e.g., LCFS electron temperature or spline knot locations) resulted in significant shifts in $R_0$, $l_i$, and pedestal profiles, highlighting the sensitivity of automated workflows to input assumptions.

B. MHD Stability Analysis

• Ideal Kink Stability ($\delta W$ via DCON):
  • The $\delta W$ calculation is relatively robust.
  • 90% agreement: In 90% of cases, the stability classification (stable vs. unstable) remained unchanged between Manual and CAKE equilibria.
  • Correlation with Error: There is a strong correlation between the Grad-Shafranov (GS) reconstruction error and $\delta W$ values; lower GS error generally yields higher (more stable) $\delta W$.
  • Magnitude of Change: Physical scans showed $\delta W$ changes of $0.5–1.0$ for significant plasma parameter changes. Differences in $\delta W$ between Manual and CAKE were often within this range, suggesting that reconstruction method is a comparable source of uncertainty to physical parameter variation.
• Tearing Mode Stability ($\Delta'$ via STRIDE):
  • High Numerical Instability: The $\Delta'$ calculation is extremely sensitive to equilibrium variations. Values for the same shot could differ by factors of 10 to 100 between reconstruction methods.
  • Classification Agreement: Despite the magnitude variance, the sign of $\Delta'$ (stabilizing vs. destabilizing) agreed in approximately 75% of cases.
  • Implication: The high sensitivity makes $\Delta'$ unreliable for precise tearing mode growth rate modeling (Modified Rutherford Equation) without rigorous uncertainty quantification.

5. Significance and Conclusions

• Validation of Automation: Automated tools like CAKE provide a consistent, high-throughput alternative to manual reconstruction, with results that are generally consistent with expert manual reconstructions for global stability metrics.
• Uncertainty Quantification (UQ) Necessity: The study demonstrates that reconstruction method is a major source of uncertainty. For critical profile quantities (like bootstrap current and pedestal pressure), relying on a single equilibrium reconstruction is insufficient.
• Recommendations for Future Work:
  • Multi-Equilibrium Analysis: Physics analyses should utilize multiple equilibrium reconstructions (e.g., Manual + CAKE) for a single plasma shot to explicitly quantify uncertainty.
  • Focus on Robust Metrics: While ideal kink stability ($\delta W$) is robust enough for general classification, tearing mode stability ($\Delta'$) requires extreme caution and potentially better-constrained profiles.
  • Workflow Improvement: The results highlight specific areas (e.g., LCFS temperature handling, spline constraints) where automated workflows need refinement to reduce variance in profile quantities.

In summary, this paper establishes that while automated equilibrium reconstruction is a viable and necessary step for large-scale fusion data analysis, the resulting uncertainties in profile quantities can significantly impact downstream physics conclusions, necessitating a shift toward uncertainty-aware modeling practices in tokamak research.