GPU-Accelerated Simulation of 3D Diamond Sensors Using the TeRABIT Infrastructure

This paper presents a GPU-accelerated, distributed simulation framework leveraging the TeRABIT infrastructure to efficiently solve a novel third-order partial differential equation for modeling signal formation in 3D diamond sensors, thereby enabling rapid what-if studies on sensor geometry.

The Gist

Imagine you are trying to listen to a whisper in a very long, noisy hallway. The whisper is a tiny signal from a particle of radiation hitting a diamond sensor. The hallway is the diamond itself, and the walls are lined with special carbon "wires" (electrodes) that carry the sound to your ear (the computer).

The problem is that these carbon wires aren't perfect; they are a bit like old, rusty pipes. When the signal travels through them, it gets delayed and distorted, much like how a sound echoes and fades in a long tunnel. This makes it hard to know exactly when the whisper started, which is crucial for high-speed physics experiments.

Here is how the researchers in this paper solved the problem of figuring out exactly how that signal behaves, using a mix of super-smart math and super-fast computers.

1. The Old Way: Trying to Map a Maze with a Flashlight

Previously, scientists tried to simulate how these signals move through the diamond. It was like trying to map a giant, 3D maze by walking through it one step at a time with a flashlight.

• The Bottleneck: The math required to predict how the signal twists and turns through the "rusty pipes" was incredibly heavy. It took a supercomputer a whole week to simulate just one version of the sensor.
• The Limitation: Because it took so long, they couldn't test many different designs. They were stuck with one shape, unable to ask, "What if we made the wires thinner?" or "What if the diamond was shorter?"

2. The New Tool: The "TeRABIT" Super-Express

The authors built a new simulation engine called WeightingTide. Think of this as replacing the slow, step-by-step flashlight with a fleet of high-speed drones that can fly over the entire maze at once.

• The GPU Boost: They moved the heavy math onto GPUs (the powerful chips usually found in video game computers). Instead of one brain doing the math, they used thousands of tiny brains working simultaneously. This turned a one-week job into a few hours.
• The "TeRABIT" Network: To handle even more work, they didn't just use one computer. They connected computers in different cities (Florence, Bologna, and Padova) using a special internet protocol called InterLink. Imagine a relay race where runners in different cities pass the baton instantly. If one computer is busy, the work is instantly handed off to another one nearby. They stored the data in a central "cloud locker" (S3 storage) so everyone could grab what they needed without clogging the local roads.

3. The "What-If" Game: Designing the Perfect Sensor

With this new, fast system, the team could finally play a game of "What if?" They tested thousands of different shapes for the diamond sensor to see which one would give the clearest, fastest signal.

They focused on two main parts of the sensor:

• The "Bias" Wires (The Power Supply): They wondered if making these wires thinner would help.
  • The Surprise: They found that making these wires thinner didn't actually change the timing much. It was like realizing that tightening the handle on a door doesn't stop the squeak; the squeak comes from the hinges elsewhere.
• The "Readout" Wires (The Signal Path): They tested making the diamond thinner, which shortens the path the signal has to travel.
  • The Discovery: This did help! Shortening the path the signal travels reduced the delay. It's like shortening a long hallway; the whisper reaches your ear faster and clearer.

4. The Result: A Sharper Picture

By combining these findings, the team proposed a new design:

1. Make the "readout" wires shorter (by using a thinner diamond).
2. Make the "bias" wires as thin as possible (to save money and reduce the risk of breaking the diamond during manufacturing), since their size doesn't hurt the timing.

The Bottom Line:
This new simulation method is like upgrading from a slow, manual mapmaker to a real-time GPS system. It allowed the scientists to quickly test designs and found a way to improve the sensor's timing precision by about 10%. This brings them closer to the ultimate goal: detecting particles with a timing resolution so fast it's better than 100 picoseconds (that's 100 trillionths of a second!).

They didn't invent a new sensor today, but they built the "wind tunnel" that allows engineers to design the best possible sensor for the future.

Technical Summary

Technical Summary: GPU-Accelerated Simulation of 3D Diamond Sensors Using the TeRABIT Infrastructure

Problem Statement
Diamond detectors featuring electrodes orthogonal to the surface, created via laser-induced graphitization, offer significant potential for High Energy Physics, Nuclear Medicine, and dosimetry due to their radiation hardness and fast timing capabilities. Recent advancements in graphitization have enabled time resolutions better than 100 ps. However, accurately simulating signal formation in these devices remains a major computational challenge. The signal formation process involves a complex interplay of energy deposition fluctuations, carrier transport, signal propagation, and readout electronics.

Standard simulation methods based on the Ramo–Shockley theorem are insufficient because they do not account for the impedance effects of resistive electrodes, which dominate time resolution in these 3D structures. While an extension of the theorem by Riegler incorporates these effects via a four-dimensional correlation function, $\vec{H}(\vec{x}, t)$, the numerical computation of this function is computationally expensive. Previous attempts using pseudo-spectral methods in Python (with C extensions) and COMSOL MultiPhysics® were either too slow for systematic geometric optimization or constrained by memory limits that prevented distributed computing on High-Throughput Computing (HTC) grids.

Methodology
The authors developed a new simulation framework that addresses these bottlenecks through three primary technical innovations:

1. Orchestrated Micro-Step Workflow: The simulation is structured as a Directed-Acyclic-Graph (DAG) workflow orchestrated by Snakemake. This workflow breaks the process into six distinct steps:

   • Geometry compilation (OpenSCAD to STL).
   • Electrostatic field computation (solving the Laplace equation).
   • Correlation function computation (solving a third-order partial differential equation derived from Maxwell's laws in a quasi-static approximation).
   • Carrier trajectory simulation (using Garfield++).
   • Induced current calculation (convolution of velocity and correlation function).
   • Electronics and DAQ simulation.
     Intermediate data formats are optimized for performance, including the use of Apache Arrow to reduce file sizes by ~80% and deserialization overhead.

2. GPU Acceleration (WeightingTide): Performance-critical steps, specifically the computation of the electrostatic field and the correlation function $\vec{H}(\vec{x}, t)$, were ported to GPUs using the CuPy library.

   • Electrostatics: An iterative pseudo-spectral method solves the Laplace equation using 3D FFTs and custom CUDA kernels. Custom kernels were employed to multiply spectral representations by the squared momentum of the wave-vector, halving the required GPU memory compared to a pure Python implementation.
   • Correlation Function: A custom CUDA kernel computes the voltage set by point charges at the boundaries, enforcing boundary conditions by copying rescaled chunks of fundamental solutions. This step, which dominated CPU runtime, achieved a 100× speedup on GPUs.
   • Convolution: The convolution of carrier velocities with the correlation function was offloaded to the GPU. The correlation function was loaded into GPU texture memory to leverage hardware-accelerated linear interpolation along carrier trajectories.

3. Distributed Computing via TeRABIT and InterLink: To enable systematic parameter scans, the workflow was distributed across heterogeneous computing resources. The system utilizes the TeRABIT HPC "Bubbles" (specifically the Padova site with H200 GPUs) and the INFN-CNAF HTC resources (CPU-only nodes).

   • The Snakemake workflow interacts with a Kubernetes cluster managed by Kueue.
   • The InterLink project defines virtual nodes that execute containerized payloads (managed via Apptainer and CVMFS) on remote sites (CNAF and Padova) rather than local physical nodes.
   • Data exchange between geographically distant sites is handled via a centralized S3-compatible storage (Deuxfleurs Garage), though data transfer bandwidth is noted as the primary bottleneck in parallel execution.

Key Contributions

• Algorithmic Innovation: The development of a solver using fundamental solutions to impose boundary conditions and spectral methods to extend the solution to the bulk of the diamond detector, solving the third-order PDE required for time-dependent weighting potentials.
• Software Porting: The successful migration of performance-critical simulation steps to GPUs, reducing the time-to-insight for a single geometry from approximately one week to a few hours.
• Infrastructure Integration: The demonstration of a distributed computing model that seamlessly integrates HTC (CNAF) and HPC (TeRABIT Padova) resources via the InterLink protocol and Kubernetes, allowing for the orchestration of complex, multi-step scientific workflows across sites.
• Package Development: The creation of the WeightingTide Python package, which encapsulates the CuPy-based GPU-accelerated methods for electrostatic and correlation function calculations.

Results
Using the new simulation architecture, the authors performed systematic parametric studies on 3D diamond detectors with graphitized electrodes, focusing on two geometric variables: bias electrode diameter and diamond bulk thickness.

• Bias Electrode Diameter: Contrary to initial expectations that electrode resistance would linearly dictate time resolution, the study found that the time resolution is largely independent of the bias electrode diameter (varied from 6 to 14 $\mu$m). The authors attribute this to the 3D geometry, where the contribution of bias electrodes to the correlation function evolution is negligible compared to readout electrodes. This suggests that bias electrodes can be fabricated with very small diameters to minimize damage risks during graphitization without compromising timing performance.
• Bulk Thickness: The study confirmed that time resolution improves as the diamond bulk thickness decreases. This is consistent with the finding that signal propagation along the readout electrode is the dominant factor limiting timing. Shorter readout electrodes (thinner bulk) directly enhance timing performance.
• Optimization Proposal: Based on these findings, the authors propose a new sensor geometry (Fig. 7) featuring shortened readout electrodes and minimized bias electrode diameters. This configuration is projected to improve time resolution by approximately 10% while maintaining fabrication feasibility.

Significance and Outlook
The paper claims that this work enables "what-if" studies on sensor geometry that were previously computationally prohibitive. By reducing the time-to-insight from weeks to hours and enabling distributed execution, the framework allows for systematic optimization of detector parameters.

The authors conclude that while incremental gains (~10%) can be achieved through geometric optimization (specifically reducing readout electrode length), the most effective strategy for further performance enhancement remains improving the conductivity of the graphitized columns. The results indicate that 3D diamond detectors are rapidly approaching the fundamental limits imposed by readout electronics. The successful integration of GPU acceleration with distributed HPC/HTC resources via the TeRABIT infrastructure provides a robust pathway for future detector design and optimization.