The Closure Challenge: a benchmark task for machine learning in turbulence modelling

This paper introduces "The Closure Challenge," a standardized open-source benchmark comprising curated datasets, evaluation code, and specific test cases designed to establish a common metric for evaluating and advancing machine learning models in Reynolds-averaged Navier-Stokes turbulence modeling.

The Gist

Imagine you are trying to teach a robot how to predict how wind blows around a car, a plane, or even a building. This is a huge challenge in physics called turbulence modeling. For decades, scientists have been trying to use Machine Learning (AI) to make these predictions faster and more accurate.

However, there was a major problem: everyone was playing a different game.

Some researchers used their own secret training data, others used different rules to grade their work, and there was no standard way to say, "Hey, your AI is actually better than mine." It was like having three chefs trying to prove they make the best soup, but one uses a secret recipe, another uses a different pot, and they all judge the taste differently. Progress was slow because nobody could fairly compare results.

The Solution: "The Closure Challenge"

This paper introduces a new, standardized cooking competition for AI scientists working on fluid dynamics. The authors (a team from MIT, TU Delft, and Sorbonne University) have built a "playground" called The Closure Challenge.

Here is how it works, using simple analogies:

1. The "Test Kitchen" (The Benchmark)

Think of the challenge as a standardized test kitchen.

• The Ingredients: The organizers provide a specific set of "test cases" (complex wind scenarios like flow over a hill, inside a square pipe, or around a NASA-designed bump).
• The Rule: You are forbidden from tasting the test dishes while you are practicing. You must train your AI on other data, and then see how well it predicts these specific, unseen test cases.
• The Goal: To see which AI can generalize best. Can it learn the rules of wind from one situation and apply them to a totally new, tricky situation?

2. The "Recipe Book" (The Datasets)

To make it fair and easy to start, the organizers have opened a massive library of "training recipes" (high-fidelity data).

• They provide the "ground truth" (the perfect, real physics data) and even the "mean velocity gradients" (which are like the detailed instructions on how the wind speed changes from one point to the next).
• Why this matters: Before this, if you wanted to train an AI, you had to build your own lab and gather your own data. Now, the data is free and ready to use, lowering the barrier to entry so more people can join the race.

3. The "Scorecard" (Evaluation)

In the past, people measured success in confusing ways. Now, there is a single, clear scorecard.

• The score measures the average error in the AI's prediction.
• The Analogy: Imagine the AI is trying to hit a target with a dart. The score tells you how far off the dart landed from the bullseye, on average.
• The Winner: The person with the lowest score wins. A score of 0.05 means the AI is off by only 5% on average.

Why This Matters

This isn't just a one-time contest; it's a living leaderboard.

• The "Hall of Fame": The paper shows the current top three teams. They are already achieving scores that suggest their AI is making very accurate predictions (around 6% error).
• The Future: The goal is to make this the standard for the entire field. Just like how protein scientists use specific benchmarks to prove their new drugs work, fluid dynamicists will now use "The Closure Challenge" to prove their AI models are ready for the real world.

In a Nutshell

The authors are saying: "Stop reinventing the wheel. We've built a standardized track, provided the cars (data), and set up the timing system (scoring). Now, let's see who can actually drive the fastest and most accurately."

This challenge aims to speed up innovation, ensuring that when we finally use AI to design better planes or predict weather, we know the models are actually good, not just lucky.

Technical Summary

Based on the provided paper, here is a detailed technical summary of "The Closure Challenge."

1. Problem Statement

The field of Machine Learning (ML) augmented Reynolds-Averaged Navier-Stokes (RANS) turbulence modelling has seen significant growth over the last decade. However, progress is hindered by a critical lack of standardization:

• Absence of Benchmarks: There is no open-source benchmark dataset with clear evaluation criteria.
• Comparison Difficulty: Comparing new techniques against existing ones requires significant manual effort due to disparate training datasets, codebases, and local infrastructures used by different research groups.
• Generalization Gap: Current methodologies often struggle with generalization across different Reynolds numbers and geometries, but this is difficult to quantify without a unified test suite.

2. Methodology and Framework

The authors propose The Closure Challenge, a continuously running, open-source benchmark designed to standardize the evaluation of ML models in RANS turbulence modelling.

• Core Philosophy: Unlike traditional competitions tied to specific conferences, this is an ongoing challenge intended to serve as a permanent "leaderboard" for the field, similar to benchmarks in protein structure prediction or weather forecasting.
• Task Definition:
  • Objective: Predict flow fields on specified Computational Fluid Dynamics (CFD) meshes for a series of test cases.
  • Training Flexibility: To lower the barrier to entry, the challenge does not mandate a specific training or validation dataset. Submitters may use their own data and code.
  • Strict Constraint: It is strictly forbidden to train or validate on the test cases.
• Evaluation Metric:
  • The score is calculated as the average fractional velocity error across all test cases.
  • Formula: $Score = \frac{1}{N_{cases}} \sum_{c} \frac{1}{N_c |\bar{U}|_c} \sum_{i=1}^{N_c} |\tilde{U}_i - U_{true,i}|$
  • Where $\tilde{U}$ is the predicted velocity, $U_{true}$ is the reference (DNS/LES) velocity, and $|\bar{U}|_c$ is the mean velocity magnitude for the case. This normalization ensures cases with different characteristic velocities contribute equally.
  • Interpretation: A lower score indicates better agreement with reference data (e.g., 0.05 implies a ~5% average error).

3. Key Contributions

• Standardized Test Suite: The challenge provides a curated collection of high-fidelity test cases focusing on two critical generalization axes: Reynolds number and geometry.
  • Periodic Hills: Various parametric variations (different hill heights and lengths) at $Re=5600$.
  • Square Duct: Variations in Aspect Ratio (AR) and friction Reynolds number ($Re_\tau$).
  • NASA Wall-Mounted Hump: A complex 3D separation case.
• Data Accessibility:
  • Provides DNS/LES ground truth data, including mean velocity gradients (crucial for many current ML frameworks).
  • Includes k-ω SST RANS predictions as a baseline.
  • Offers an open-source Python evaluation package (available on PyPI) for calculating scores.
• Community Infrastructure: A central GitHub repository (rmcconke/closure-challenge-benchmark) hosts the datasets, code, and a dynamic leaderboard.

4. Results (As of March 2026)

The paper reports on the initial phase of the challenge, synchronized with the ERCOFTAC workshop, featuring three early submissions:

Rank	Authors	Overall Score	Notes
1	Reissmann, Fang, and Sandberg	0.0595	Best performance.
2	Wu and Zhang	0.0624
3	Montoya, Oulghelou, and Cinnella	0.0779	Pretrained model only (no fine-tuning on challenge data).

• Current Scope: The benchmark is fully implemented for 2D flows.
• Future Scope: 3D datasets (Wing-body junction, Ahmed body, FAITH hill) are available on the page, with plans to introduce a dedicated 3D leaderboard in the future.

5. Significance

• Accelerating Innovation: By removing the friction of data preparation and evaluation setup, the challenge allows researchers to focus purely on model architecture and physics-informed learning strategies.
• Raising the Bar: The challenge is designed to be "hard," specifically targeting generalization failures that plague current ML-RANS models. It aims to filter out methods that only work on specific training distributions.
• Establishing a Standard: The authors aim for this benchmark to become the de facto standard for evaluating new ML frameworks in RANS, ensuring that future claims of "state-of-the-art" performance are backed by rigorous, comparable metrics.
• Open Science: By providing high-fidelity gradients and ground truth in a standardized format, the challenge democratizes access to high-quality turbulence data for the broader ML community.