Training on Data Analysis Reproducibility via Containerization with Apptainer

This paper presents training materials and resources developed by the HEP Software Foundation to equip physicists with Apptainer containerization skills, thereby enhancing the reproducibility, portability, and collaboration of High Energy and Nuclear Physics analyses.

The Gist

Imagine you are a brilliant chef who has created a perfect, award-winning recipe. You write it down on a piece of paper and send it to a friend in a different country.

Your friend tries to cook it, but it fails. Why? Because your kitchen had a specific brand of oven, a rare spice only found in your local market, and a special knife your friend doesn't own. The recipe didn't fail; the environment failed.

In the world of science, specifically High Energy Physics (like studying the Big Bang or nuclear reactions), researchers face this exact problem. They run complex computer analyses that produce huge amounts of data. If they send their "recipe" (code and data) to a colleague, it often breaks because the colleague's computer is set up differently. This is the infamous "It works on my machine" problem.

This paper is about a training course designed to fix that problem using a technology called Apptainer (formerly known as Singularity). Here is the breakdown in simple terms:

1. The Solution: The "Digital Lunchbox"

Think of a computer program not just as a set of instructions, but as a digital lunchbox.

• Old Way: You send the instructions and hope your friend has the exact same ingredients (software libraries) and tools (operating system settings) in their kitchen.
• Apptainer Way: You pack the instructions, plus every single ingredient, tool, and even the specific temperature of the kitchen, into a single, sealed lunchbox.
• The Magic: No matter whose kitchen (computer) your friend opens this lunchbox in, the food cooks exactly the same way. The lunchbox is self-contained.

2. Why This Training Matters

The authors (a team of scientists from universities and labs across the US and Europe) realized that science is becoming too complex to rely on "hope." They created a training module to teach physicists how to pack these "digital lunchboxes."

• The Goal: To make sure that a physics experiment done today can be perfectly recreated 10 or 20 years from now, even if the computers and software have completely changed.
• The Benefit: It saves time, reduces frustration, and ensures that scientific results are trustworthy and reproducible.

3. What Did They Teach?

The training course is like a cooking class for scientists. It covers:

• Finding Lunchboxes: How to download pre-made lunchboxes from a giant online library (like a digital grocery store).
• Cooking Inside: How to run commands and do work inside the lunchbox without messing up the main kitchen.
• Building Your Own: How to take a recipe and pack it into a new lunchbox from scratch.
• Sharing: How to pass ingredients back and forth between the main computer and the lunchbox.
• Advanced Moves: How to set up a lunchbox that runs a service (like a 24/7 server) in the background.

They used real-world tools that physicists actually use, like ROOT (a massive data analysis toolkit) and Python, to show that this isn't just theory—it works for real science.

4. Did It Work? (The Taste Test)

The team tested this training on 360 scientists between 2023 and 2025.

• Before the class: Most scientists had never heard of Apptainer. They were like people who had never seen a lunchbox before.
• After the class: The scientists felt much more confident. They learned how to use the basic commands to pull, run, and build these containers.
• The Verdict: The students said the lessons were the "just right" difficulty—not too easy, not too hard—and they loved the hands-on exercises.

5. The Big Picture

This paper isn't just about software; it's about preserving knowledge.

In the past, if a scientist left a lab or a computer system was upgraded, years of complex analysis could be lost forever because no one could figure out how to run the old code again. With Apptainer, the "lunchbox" preserves the entire environment. As long as the lunchbox exists, the science inside it remains alive, reproducible, and shareable with anyone, anywhere in the world.

In short: This training teaches scientists how to pack their work into unbreakable, portable time capsules, ensuring that the discoveries of today can be understood and built upon by the scientists of tomorrow.

Technical Summary

Based on the provided paper, here is a detailed technical summary of the training module on Data Analysis Reproducibility via Containerization with Apptainer.

Problem Statement

High Energy Physics (HEP) and Nuclear Physics analyses face significant challenges regarding reproducibility, preservation, and portability. The complexity of modern analyses, coupled with rapidly evolving software dependencies and computing environments, leads to the "works on my machine" problem. This hinders the ability to reuse, reinterpret, and verify research outcomes over time. There is a critical need for a framework that encapsulates the entire analysis environment (dependencies, libraries, configurations) to ensure that scientific results remain valid and executable years after their initial creation, regardless of changes in the host operating system or hardware.

Methodology

The authors developed a comprehensive training module titled "Introduction to Apptainer/Singularity" as part of the HEP Software Foundation (HSF) Training Center. The methodology involves:

1. Technology Selection: The training focuses on Apptainer (formerly Singularity), a container platform designed specifically for scientists. Unlike traditional containers, Apptainer:

   • Runs without requiring superuser (root) permissions, making it ideal for shared HPC clusters and cloud environments.
   • Uses single-file container images for easy distribution and archiving.
   • Preserves user permissions (the user inside the container is the same as outside), addressing security concerns for system administrators.
   • Integrates seamlessly with resource managers and distributed computing environments.

2. Curriculum Design: The module is structured into specific "episodes" (chapters) designed for both live instruction and asynchronous self-study. Each episode includes learning objectives, exercises with solutions, and summaries. Key topics include:

   • Fundamentals: Concepts of containers vs. virtual machines and Apptainer design goals.
   • Image Management: Pulling images from libraries (Sylab, Docker Hub) and running commands.
   • Container Creation: Building containers from existing images and creating them from scratch using Definition Files.
   • Data Handling: Sharing files between the host and container via bind mounts.
   • Advanced Deployment: Running detached instances (e.g., Jupyter servers) and deploying containers to GitHub Packages via GitHub Actions.
   • Domain Specifics: Examples utilize tools common to HEP, such as ROOT, PyROOT, Pythia8, Uproot, and RooFit.

3. Implementation & Evaluation:

   • The training was delivered during HSF and IRIS-HEP events between 2023 and 2025.
   • Pedagogical Approach: A mix of pre-recorded lectures, hands-on exercises, and live mentoring (via Slack) to troubleshoot setup issues and review code.
   • Assessment: A two-step survey process (pre- and post-training) was implemented to measure knowledge gain and satisfaction. An anonymous verification system ensured response integrity.

Key Contributions

• Standardized Training Material: Created a modular, open-access curriculum that bridges the gap between theoretical containerization concepts and practical application in HEP/Nuclear Physics workflows.
• Domain-Specific Examples: Provided concrete, reproducible examples using the ROOT data analysis framework and Python libraries, demonstrating how to containerize complex physics event generators and data readers.
• Workflow Integration: Demonstrated how to convert existing Docker images to Apptainer for HPC environments and how to automate container building and sharing using GitHub Actions and the GitHub Container Registry (GHCR).
• Long-term Preservation Strategy: Established a framework for preserving analysis pipelines independently of future OS updates or dependency availability, provided Apptainer remains supported.

Results

The training was evaluated through surveys of 360 registered participants (pre-event) and 82 participants (post-event):

• Baseline Knowledge: Pre-event data (Figure 1) revealed that the vast majority of participants had "Never heard of" Apptainer commands or concepts, confirming a significant skills gap in the community.
• Knowledge Acquisition: Post-event data (Figure 2) showed a marked increase in familiarity. Participants reported being "Familiar" or "Very familiar" with core commands such as apptainer pull, apptainer shell, and apptainer exec.
• Advanced Topics: While core objectives were met, a significant number of users remained unfamiliar with apptainer instance, indicating this is an advanced topic requiring future focused attention.
• Satisfaction: Participants rated the difficulty level as "About right" and the number of exercises as sufficient for interactive learning (Figure 3), indicating a successful balance between theory and practice.

Significance

This work addresses a critical bottleneck in modern scientific research: the preservation of scientific integrity in a changing technological landscape. By training physicists in Apptainer, the HEP community can:

1. Enhance Reproducibility: Ensure that analysis results can be exactly replicated by other researchers, regardless of their local computing environment.
2. Facilitate Collaboration: Provide a common language and platform for sharing analysis pipelines across institutions and globally.
3. Optimize Resources: Enable efficient scaling of analysis workflows on HPC systems and clouds without the overhead of complex dependency management.
4. Future-Proof Research: Guarantee that data analysis remains accessible and interpretable for years, securing the long-term value of expensive experimental data.

The successful deployment of this module across multiple training events demonstrates its viability as a standard practice for next-generation data-intensive physics research.