The IAEA Fusion Data Lake Project -- Accelerating AI and Big Data Applications through Open Science and FAIR Data

This paper presents the IAEA's Fusion Data Lake project, a five-year initiative under the AI for Fusion Coordinated Research Project that aims to accelerate AI and big data applications in fusion energy by developing a FAIR-compliant data infrastructure featuring an international catalogue, centralized storage, and global data federation to enhance the accessibility and visibility of experimental datasets.

The Gist

Imagine the world of nuclear fusion (the technology that aims to replicate the power of the sun) as a massive, global construction project. Scientists in different countries are building different parts of a giant machine, but they are all speaking different languages and using different blueprints. They have tons of data—logs, measurements, and simulations—but it's scattered in silos, making it hard to see the big picture.

This paper introduces a solution called the Fusion Data Lake, a project run by the IAEA (the International Atomic Energy Agency). Think of it as building a giant, universal library and construction site where all these scattered pieces can finally come together.

Here is a breakdown of the project using simple analogies:

1. The Problem: Too Many Cooks, Too Many Kitchens

Right now, fusion scientists are like chefs in different kitchens. One chef in the UK has a recipe book (data) on a specific type of stove. Another in Japan has a different book for a different stove. They want to use Artificial Intelligence (AI) to figure out how to cook the perfect meal (a stable fusion reaction), but the AI can't learn if it can't see all the recipes at once. The data is there, but it's locked away in different formats, making it hard to share.

2. The Solution: The "Fusion Data Lake"

The IAEA is building a Fusion Data Lake. Imagine a massive, digital reservoir where water (data) from all these different rivers (institutions) flows into one place.

• The Goal: To make sure the water is clean, labeled, and easy to drink (or use) for anyone, anywhere.
• The "FAIR" Principle: This stands for data that is Findable, Accessible, Interoperable (works with other systems), and Reusable. Think of it as organizing a messy garage so you can find your tools instantly, know who owns them, and use them without breaking anything.

3. How It Works: The Three Pillars

The project is built on three main services, which act like the tools in this new library:

• The Catalogue (The Index): A searchable list of every experiment and measurement from around the world. It's like a library card catalog that tells you exactly what book is on the shelf.
• The Federation (The Bridge): Instead of moving all the books to one building, they build a bridge. You can search the library, and if the book is in Japan, the system fetches it for you without you needing to travel there. It connects different computer systems so they talk to each other.
• The Storage (The Warehouse): A temporary holding area for data that hasn't been connected to the bridge yet, ensuring nothing gets lost while waiting to be organized.

4. The Tech Stack: The Engine Room

To make this happen, they are using a modern "toolkit" of technologies:

• Snowflake: The heavy-duty engine that processes the data (like a super-fast factory assembly line).
• Microsoft Azure: The giant warehouse where the data sits safely.
• ETL Pipeline: This is the conveyor belt. It Extracts data from the source, Transforms it into a standard format (so a Japanese log looks like a UK log), and Loads it into the system. They created a "recipe" for this process so adding new data sources is easy and doesn't require rebuilding the whole machine.

5. The Proof of Concept: Testing the Waters

The team is building this in three steps, like testing a new app before launching it to the world:

• Phase I (The Pilot): They connected the UK's MAST data. They proved they could pull data from one place and show it on a screen.
• Phase II (The Scale-Up): They added data from Japan (LHD) and the USA (MIT). This was the big test: Can the system handle three different types of data coming in at once? Yes! It successfully standardized them all.
• Phase III (The Expansion): They are adding data from China (HL-2A) and polishing the user interface (the website and tools) so real scientists can start using it comfortably.

6. The Rules of the Road: Data Governance

You can't just let anyone dump trash in a public park; you need rules. The project is creating a Terms of Service (a rulebook) to ensure:

• Permission: You can only upload data you have the right to share.
• Credit: If you use someone else's data, you must give them credit (citation), just like in a school paper.
• Privacy Levels: They have a "traffic light" system for access:
  • 🟢 Public: Anyone can see it.
  • 🟡 Internal: Only IAEA members can see it.
  • 🟠 Restricted: Only specific approved institutions can see it.
  • 🔴 Closed: Only specific individuals approved by the owner can see it.

7. The Future: A Global Team Effort

The paper concludes that this project is a game-changer. By bringing together 22 institutions from 11 countries, the IAEA is creating a foundation where AI can finally learn from all fusion experiments, not just one.

In a nutshell: They are building a universal translator and a shared digital warehouse for fusion scientists. This allows AI to learn faster, helping humanity unlock clean, limitless energy sooner than we could have on our own.

Technical Summary

Based on the provided paper, here is a detailed technical summary of the IAEA Fusion Data Lake (FDL) Project.

1. Problem Statement

The development of fusion energy is increasingly reliant on Artificial Intelligence (AI) and Machine Learning (ML) for tasks such as scenario design, plasma control, and physics insights. However, the scaling of these AI/ML applications is currently hindered by a lack of modern data infrastructure and metadata-rich datasets.

• Fragmentation: Fusion data is currently siloed across various institutions (e.g., UKAEA, NIFS, MIT) with incompatible formats and access protocols.
• Barriers to Entry: The absence of a unified, programmable interface makes it difficult to create "machine-agnostic" models that can generalize across different fusion devices.
• Need for FAIR Data: There is an urgent requirement to adhere to FAIR principles (Findable, Accessible, Interoperable, Reusable) to support the next generation of digital twins and surrogate models.

2. Methodology and Platform Architecture

The IAEA is addressing these challenges through the Fusion Data Lake (FDL) project, a component of the five-year "AI for Fusion" Coordinate Research Project (CRP). The methodology involves building a hybrid cloud platform that balances centralized storage with decentralized data federation.

A. Service Goals

The platform is designed to deliver three core services:

1. Data Catalogue: A query-able central index of global fusion experiments and diagnostic signals.
2. Data Federation: A single access point to a decentralized network of repositories, allowing users to access data without it physically migrating to the FDL infrastructure.
3. Centralized Storage: Interim short-to-medium-term storage for data that cannot yet be federated.

B. Technology Stack

The architecture leverages the IAEA's enterprise technology stack:

• Compute & Database: Snowflake for Python data pipelines and SQL database servers.
• Storage & DevOps: Microsoft Azure for storage and Azure DevOps for CI/CD.
• Application Layer: C# and .Net for web application development.
• Data Pipeline: A modern Medallion Architecture (Bronze/Silver/Gold layers) utilizing a standard ETL (Extract, Transform, Load) approach.
  • Innovation: A metadata-driven ingestion pipeline uses configuration files to manage dataset transformation and mapping to the data model, ensuring reusability and reducing maintenance effort for new sources.

C. Data Modeling

• Minimal Metadata Model: A simplified schema categorizing metadata into three tables: "core," "icf" (Inertial Confinement Fusion), and "mcf" (Magnetic Confinement Fusion).
• Ontology Alignment: The model is being developed in alignment with the ITER IMAS Data Dictionary to ensure interoperability and ease of adoption.

3. Key Contributions

• Global Coordination: The IAEA leverages its unique diplomatic and scientific position to facilitate collaboration across 22 institutions in 11 countries, bridging the gap between magnetic and inertial fusion communities.
• Hybrid Access Model: The project successfully demonstrates a "federation" approach where data remains at the source (e.g., UKAEA's MAST) but is accessible via the FDL portal, reducing data duplication while ensuring accessibility.
• Governance Framework: The paper introduces a provisional Data Governance Strategy including:
  • Terms of Service (ToS): Covering uploads (authority declaration), downloads (usage compliance), and attribution/citation.
  • Tiered Access Levels: A four-tier privacy model (Public, Internal, Restricted, Closed) managed via IAEA NUCLEUS accounts and OpenAthens authentication, allowing institutions to control data sharing based on their policies.

4. Results (Proof of Concept Phases)

The project has progressed through three Proof of Concept (PoC) phases to validate scalability and functionality:

• Phase I (Federation Concept):
  • Built the initial pipeline on Snowflake.
  • Successfully federated the MAST (UK) shot log via REST API.
  • Result: Demonstrated that users could access MAST data through the FDL interface without the data physically residing in the FDL storage.
• Phase II (Scalability & Diversity):
  • Integrated two additional, distinct datasets: LHD (Japan) and Alcator C-Mod (USA/MIT).
  • Ingestion Patterns: LHD data was synchronized via Azure Blob Storage; Alcator C-Mod data utilized a combination of PSFC tools (disruption-py), MDS Plus, and the IAEA multicloud platform.
  • Result: Proved the platform can ingest diverse datasets from different institutions using unique patterns while conforming to a single FDL data model.
• Phase III (Expansion & Maturity - Ongoing/Pre-release):
  • Incorporating the HL-2A (China) catalogue.
  • Maturing front-end interfaces (API and Web) to support Terms of Service compliance for early access users.

5. Significance and Future Outlook

The Fusion Data Lake represents a critical infrastructure milestone for the global fusion community.

• Enabling AI/ML: By providing metadata-rich, multimachine datasets via a programmable API, the FDL removes the primary bottleneck for training robust AI models that can scale to future devices like ITER and DEMO.
• Standardization: The alignment with ITER IMAS and the development of a unified metadata model sets a precedent for global data interoperability in fusion science.
• Sustainability: The metadata-driven ingestion and governance frameworks ensure the platform is maintainable and scalable as new institutions and devices join the network.

The project is currently transitioning from PoC to a pre-release version, aiming to become the foundational data service for the next generation of fusion research and development.