Parallel Version of CORSIKA Code with Cherenkov Option for SPHERE-3 Project

To overcome queue time limits on the Lomonosov-2 supercomputer that caused premature termination of extensive air shower simulations with Cherenkov light for the SPHERE-3 project, the authors developed and validated a multithreaded parallel version of the CORSIKA code.

The Gist

Here is an explanation of the paper, translated into simple language with some creative analogies.

The Big Picture: Chasing Cosmic Ghosts

Imagine the universe is constantly raining down invisible, super-fast particles called Cosmic Rays. When one of these particles hits the Earth's atmosphere, it doesn't just stop; it explodes into a massive, cascading shower of billions of other particles. Scientists call this an Extensive Air Shower (EAS).

The researchers in this paper are part of a project called SPHERE-3. Their goal is to build a special telescope that flies in the air (like a drone) to catch the faint blue glow (Cherenkov light) left behind by these particle showers bouncing off the snow on Lake Baikal. This glow helps them figure out what the original cosmic particle was made of (was it a proton? an iron nucleus?).

The Problem: The "Too Slow" Computer

To build this telescope, they need to simulate millions of these particle showers on a supercomputer (Lomonosov-2) to see how the light behaves.

However, they hit a wall:

• The Bottleneck: The original computer code used to simulate these showers was like a single-lane road. It could only process one particle at a time.
• The Traffic Jam: At very high energies, the simulations took so long (sometimes over 20 hours for just one event) that the computer's "waiting line" (queue) would time out and kill the job before it finished.
• The Result: They were stuck. They couldn't generate enough data to build their telescope because the computer was too slow.

The Solution: The "Conductor and the Orchestra"

The team decided to rewrite the code to run on multiple cores (processors) at the same time. Think of it like turning a solo violinist into a full orchestra.

Here is how their new "Parallel Version" works, using a Construction Site analogy:

1. The Foreman (The Master Thread)

When a simulation starts, a "Master" thread acts like the Construction Foreman.

• It starts the job alone.
• It tracks the "Leader" particle (the most energetic one) as it crashes through the atmosphere, creating smaller particles.
• The Trick: The Foreman doesn't do the whole job. It only works until the Leader's energy drops to about 2% of its original power. This is the "sweet spot" where the work is manageable.

2. The Crew (The Slave Threads)

Once the Foreman has built up a pile of secondary particles, it stops doing the heavy lifting.

• It takes the pile of work and splits it up among a team of "Slave" workers (other computer cores).
• Imagine the Foreman handing out buckets of bricks to 10 different workers. Each worker builds their own section of the wall simultaneously.
• This is where the speed comes from: instead of one person building the wall for 20 hours, 10 people build it in a fraction of the time.

3. The Handoff

Once all the workers finish their sections, they hand their results back to the Foreman, who assembles the final picture (the data file) and saves it.

The Challenges: Uneven Workloads

The paper admits that splitting the work isn't always perfect.

• The "Heavy Box" Problem: Sometimes, the pile of particles contains one giant, high-energy particle (like a heavy box). If the Foreman hands that heavy box to one worker, that worker gets stuck for a long time, while the others finish quickly and sit around doing nothing.
• The Fix: They are working on better algorithms to make sure the "bricks" are distributed more evenly so no worker is left idle.

The Results: Faster and Accurate

They tested this new system on a local server (a powerful computer in their lab).

• Speed: For the hardest simulations (100 PeV energy), they cut the time from 20 hours down to 7.5 hours. That's a 3x speedup.
• Accuracy: They compared the results of the "Old Single-Lane" version with the "New Multi-Lane" version. The results were almost identical. The blue glow patterns looked the same, proving that speeding up the process didn't break the physics.

Why This Matters

This new code is a game-changer for the SPHERE-3 project.

• Before: They were struggling to get enough data because the computer was too slow.
• Now: They can generate a massive database of simulated events quickly. This allows them to design their telescope better and eventually fly it to catch real cosmic rays, helping us solve the mystery of what these high-energy particles are made of.

In short: They took a slow, single-person process and turned it into a fast, team-based assembly line, allowing them to simulate the universe's most energetic explosions without waiting days for the computer to finish.

Technical Summary

Here is a detailed technical summary of the paper "Parallel Version of CORSIKA Code with Cherenkov Option for SPHERE-3 Project."

1. Problem Statement

The SPHERE-3 project aims to study the mass composition of primary cosmic rays (PCR) in the energy range of $10^{15}$to$10^{17}$ eV using a Cherenkov technique involving air-borne telescopes observing light reflected from snow surfaces. To optimize the detector design, the team requires a vast database of simulated Extensive Air Shower (EAS) events.

Key Challenges:

• Computational Bottleneck: The standard CORSIKA code (version 7) does not support parallel processing for sub-cascades involving Cherenkov light generation. Consequently, each event simulation must run on a single CPU core.
• Time Constraints: At primary energies slightly below 100 PeV, simulation times often exceed the job queue time limits of the Lomonosov-2 supercomputing facility.
• Inefficiency: Many high-energy events are terminated prematurely ("killed") before completion due to these time limits, resulting in a loss of critical data for the project.
• Data Requirements: The project requires specific output formats (Cherenkov light distributions at specific altitudes and angles) not provided by the standard CORSIKA output, necessitating code modification.

2. Methodology

The authors developed a custom multithreaded version of the CORSIKA code to overcome these limitations. The approach involves two main phases: code modification and a novel parallelization algorithm.

A. Code Modification

• Output Optimization: The standard particle output files were removed as they were unnecessary for the Cherenkov technique. Instead, the code was modified to generate compact binary files containing:
  • Spatial-temporal Cherenkov light distributions at the snow level (Lake Baikal).
  • Histograms of light distributions in space, angles, and arrival times at three altitudes (500m, 1000m, 1500m).
• Efficiency: These modifications reduced the file size of a single event from ~6 GB to <1 GB (compressed), while preserving essential physical information.

B. Parallelization Strategy (Two-Stage Algorithm)

The core innovation is a hybrid sequential-parallel approach designed to balance load distribution:

1. Stage 1: Sequential "Leader" Tracking (Master Thread)

   • The master thread performs standard CORSIKA initialization.
   • It tracks the primary particle and its most energetic secondary descendant (the "leader") through the atmosphere.
   • Stack Reordering: The code is modified so the leader is the last particle added to the stack, ensuring it is the first to be processed.
   • Termination Condition: The master thread stops tracking the leader when its energy drops to approximately 2% of the primary energy. This threshold was empirically chosen to ensure a sufficient pool of secondary particles has been generated while minimizing sequential overhead.
   • Exception: If a high-energy gamma quantum appears in the stack (which triggers computationally expensive electromagnetic cascades), the master thread stops immediately to offload the work.

2. Stage 2: Parallel Sub-cascade Processing (Slave Threads)

   • Once the leader tracking stops, the master thread distributes the remaining particle stack among $N$ slave threads.
   • Load Balancing Algorithm:
     • The total energy of the stack ($E_{tot}$) is calculated.
     • A target energy per process ($e_p$) is set (initially $E_{tot}/n_{slaves}$, later refined to $E_{tot}/(n_{slaves}+1)$ to handle indivisible high-energy particles).
     • Particles are sorted by energy (ascending) and assigned to sub-stacks until the cumulative energy of a sub-stack exceeds the target.
   • Aggregation: Slave threads track their assigned sub-cascades and generate Cherenkov photons. The master thread aggregates these results into multidimensional arrays, which are then saved to the final binary file.

3. Key Contributions

• Development of a Parallel CORSIKA Variant: Successfully adapted the serial CORSIKA code to utilize multi-core architectures specifically for Cherenkov light generation, a feature previously unsupported.
• Hybrid Tracking Algorithm: Introduced a "leader-first" sequential tracking phase followed by parallel distribution. This ensures that the most complex, high-energy interactions are handled sequentially to maintain stability, while the bulk of the cascade is processed in parallel.
• Optimized Data Format: Created a specialized, highly compressed binary output format tailored to the SPHERE-3 detector requirements, reducing storage needs by an order of magnitude.
• Robust Testing Framework: Validated the parallel code against the original sequential version using statistical comparisons of physical distributions.

4. Experimental Results

Tests were conducted on a local server (AMD Ryzen 9 5950X, 16 cores) and compared against the Lomonosov-2 environment.

• Performance (Speedup):

  • For $10^{17}$ eV proton events, the average processing time dropped from 20 hours (sequential) to 7.5 hours (parallel).
  • The speedup factor ($S = T_{seq}/T_{par}$) ranged from 2.2 to 3.6, depending on the primary particle type (Proton vs. Iron) and energy.
  • The parallel version allowed the system to process multiple events simultaneously without memory swapping issues.

• Physical Validation:

  • Lateral Distribution Functions (LDF): Comparisons of Cherenkov photon spatial distributions showed negligible differences between sequential and parallel versions, well within statistical fluctuations.
  • Photon Yields:
    • Iron Nuclei: Relative discrepancy of 1–4% across the energy range.
    • Protons: Relative discrepancy of 1–8%. The higher variance for protons is attributed to the intrinsic stochastic nature of proton-induced showers and differences in sample sizes, rather than algorithmic errors.
  • Conclusion: No systematic bias was introduced by the parallelization.

5. Significance and Future Work

• Project Enablement: The developed code is now fully operational and enables the generation of the massive EAS event database required for the SPHERE-3 detector design, solving the critical bottleneck of queue time limits on supercomputers.
• Scalability: The method allows for the simulation of ultra-high-energy events that were previously infeasible to complete.
• Limitations & Future Directions:
  • Load Imbalance: The current stack partitioning algorithm occasionally results in uneven energy distribution among threads (e.g., if a single high-energy gamma is assigned to one thread), causing some threads to remain idle.
  • Next Steps: Future work will focus on optimizing the stack partitioning algorithm for better granularity and exploring GPU acceleration for specific code fragments to further reduce computation time.

In summary, this work successfully bridges the gap between the computational demands of modern cosmic ray simulations and the limitations of legacy serial codes, providing a robust tool for the SPHERE-3 project and potentially other Cherenkov-based cosmic ray experiments.