End-to-end Differentiable Calibration and Reconstruction for Optical Particle Detectors

This paper introduces the first end-to-end differentiable optical particle detector simulator that unifies simulation, calibration, and reconstruction into a single gradient-based framework, demonstrating improved accuracy, speed, and flexibility for analyzing large-scale neutrino detectors compared to traditional methods.

The Gist

Imagine you are trying to solve a giant, 3D jigsaw puzzle in a pitch-black room. You can't see the pieces, but you have a thousand tiny cameras on the walls that can hear the faint "click" when a piece lands. Your goal is to figure out exactly what the picture looks like, where the pieces came from, and how fast they were moving, just by listening to those clicks.

This is essentially what physicists do with optical particle detectors (like the massive Super-Kamiokande tank in Japan). They watch for flashes of light (Cherenkov radiation) created when invisible particles zoom through water.

For decades, solving this puzzle has been a messy, three-step process that didn't talk to each other well. This paper introduces a revolutionary new tool called LUCiD that turns the whole process into a single, smooth, self-correcting machine.

Here is the breakdown of the problem and the solution, using everyday analogies.

The Old Way: The "Guess, Check, and Fix" Loop

Traditionally, physicists treated three tasks as separate jobs:

1. Simulation (The Map): They built a digital model of the detector to predict what the cameras should see.
2. Calibration (The Tuning): They compared the model to real data and manually tweaked knobs (like "how clear is the water?" or "how sensitive are the cameras?") until the model matched reality.
3. Reconstruction (The Detective Work): Once the model was tuned, they used it to guess the properties of the actual particles that caused the flashes.

The Problem: These steps were like three people passing a note down a line. If the first person made a small mistake, the second person had to guess how to fix it, and the third person had to guess again. It was slow, inefficient, and often led to errors because the "knobs" were all connected. If you turned up the "water clarity" knob, you might accidentally break the "camera sensitivity" setting, but the system didn't know that.

The New Way: The "Self-Correcting GPS"

The authors created LUCiD, a system that combines all three steps into one End-to-End Differentiable framework.

What does "Differentiable" mean?
Think of a standard video game. If you press "jump," the character jumps. If you press "jump" harder, the character jumps higher. But if you ask the game engine, "How much harder do I need to press to jump exactly 10 inches higher?" the game usually says, "I don't know, I just do what you tell me."

LUCiD is different. It's like a GPS that knows exactly how every single button on your dashboard affects your destination. If you are 5 miles off course, the GPS doesn't just tell you to "turn left." It calculates the exact angle and speed you need to adjust to get back on track instantly.

In physics terms, this means the computer can calculate the gradient (the direction of improvement) for every single variable at once.

How LUCiD Works: The "Smooth Light" Trick

One of the biggest headaches in physics is that light behaves in "jumpy" ways. A photon either hits a sensor or it doesn't. It's a binary switch: ON or OFF.

If you try to use a smooth GPS on a bumpy road, the car gets stuck. Similarly, if a simulation tries to calculate how to adjust a sensor based on a "hit or miss" switch, the math breaks down because there is no "in-between" to guide the adjustment.

The Solution: Photon Relaxation
The authors invented a clever trick called Photon Relaxation.

• Old Way: Imagine a laser pointer hitting a wall. It's either on the wall (1) or off the wall (0).
• LUCiD Way: They pretend the laser beam is slightly fuzzy, like a soft glow. Even if the beam misses the wall by a tiny bit, it still casts a faint shadow on the wall.

This "fuzziness" turns the harsh "ON/OFF" switch into a smooth, sliding dimmer switch. Now, the computer can see, "Oh, the beam is almost hitting the sensor. If I move the particle just a tiny bit to the left, the signal gets stronger." This allows the system to use gradient descent (sliding down a hill to find the lowest point) to find the perfect answer instantly.

What Can LUCiD Do?

The paper shows that this new system is a game-changer in three areas:

1. Calibrating the Detector (Tuning the Radio):
   Imagine you have a radio with 10,000 knobs, and you need to tune them all so the music sounds perfect. The old way was to turn one knob, listen, turn another, listen, and hope you didn't mess up the first one.
   LUCiD turns all 10,000 knobs simultaneously. It listens to the "static" and adjusts every single knob at the exact same time to find the perfect harmony. It successfully calibrated thousands of sensors in seconds, something that used to take days.

2. Reconstructing Tracks (Finding the Culprit):
   When a particle zips through the detector, it leaves a trail of light. LUCiD can look at that trail and instantly calculate: "The particle started here, was moving this direction, and had this much energy."
   It does this by "sliding" the particle's path in the simulation until the predicted light pattern matches the real camera data perfectly. It's as accurate as the old methods but much faster.

3. Designing Future Detectors (The Architect):
   Because the system is so flexible, scientists can now ask: "What if we made the detector 20% bigger?" or "What if we used cheaper, less sensitive cameras?"
   LUCiD can instantly simulate the result and tell you how it affects the physics. It allows scientists to "test drive" different detector designs on a computer before spending billions of dollars to build them.

The Big Picture

This paper is about unification. For decades, particle physics has been like a relay race where runners (simulation, calibration, reconstruction) drop the baton and run their own separate laps.

LUCiD turns it into a synchronized swim team. Everyone moves together, reacting to the same music (the data) at the same time. By making the entire process "differentiable" (smooth and mathematically connected), the authors have created a tool that is not only faster and more accurate but also opens the door to a new era of physics where we can optimize our experiments from the very first design sketch all the way to the final discovery.

In short: They took a clunky, manual process and replaced it with a self-driving car that can tune its own engine, navigate the road, and design a better car for the future, all while you sit back and watch.

Technical Summary

Here is a detailed technical summary of the paper "End-to-end Differentiable Calibration and Reconstruction for Optical Particle Detectors."

1. Problem Statement

Large-scale homogeneous optical particle detectors (e.g., Super-Kamiokande, IceCube, JUNO) rely on complex workflows involving three traditionally disjoint steps: Simulation, Calibration, and Reconstruction.

• The Bottleneck: Standard simulators (like GEANT4) are "prediction-only" black boxes. They are non-differentiable, meaning they cannot compute gradients with respect to input parameters.
• Consequences:
  • Inefficient Calibration: Parameters (e.g., optical properties, sensor efficiencies) must be tuned sequentially using discrete grid searches or derivative-free algorithms. This ignores correlations between parameters and suffers from the "curse of dimensionality."
  • Suboptimal Reconstruction: Track reconstruction relies on maximizing likelihoods using non-differentiable forward simulations, often requiring heuristic approximations or separate tuning.
  • Workflow Fragmentation: The separation of these tasks prevents a unified optimization pipeline, leading to systematic uncertainties and inefficient detector design processes.

2. Methodology: LUCiD Framework

The authors introduce LUCiD (Light-based Unified Calibration and tracking Differentiable simulation), a JAX-based, end-to-end differentiable simulator. It unifies light generation, propagation, and detection into a single gradient-based optimization pipeline.

Core Components

• Differentiable Ray Tracing (Expectation-Based): Instead of sampling individual stochastic photons (which breaks differentiability), LUCiD propagates "rays" representing the expected value (ensemble average) of photon contributions. This allows for smooth, continuous gradients.
• Ray Generation:
  • Uses a Sinusoidal Representation Network (SIREN) as a surrogate model for Cherenkov light emission, trained on GEANT4-generated lookup tables. This replaces complex microscopic physics with a differentiable function mapping particle energy and track position to emission distributions.
  • Calibration sources (lasers, isotropic points) are modeled deterministically using Fibonacci lattices to minimize sampling variance.
• Ray Propagation:
  • Analytic Intersection: Uses analytic formulas for ray-surface intersections rather than step-wise ray marching, ensuring precision and speed.
  • Grid Acceleration: Implements a spatially hashed grid system to reduce sensor lookup complexity from $O(N_s)$ to $O(1)$.
  • Implicit Capture: Instead of probabilistically terminating rays (which creates noise), rays are attenuated continuously, and expected weights are deposited into sensors. This ensures smooth gradients.
  • Photon Relaxation: To handle the binary nature of geometric hits (hit/miss), the authors model photons as having a spatial extent (Gaussian width). This creates a smooth overlap weight ($w_{geo}$) between the ray and the sensor, ensuring non-zero gradients even when a ray misses a sensor.
  • Differentiable Stochasticity: Uses the Reparameterization Trick and Straight-Through Estimators (STE) to make stochastic processes (scattering, reflection, discrete path selection) differentiable.
• Hit Making: Converts ray intensities into expected photoelectron counts (charge) and soft-minimum arrival times, providing continuous observables for loss calculation.

Optimization Strategy

The framework minimizes a composite loss function using the Adam optimizer:

• Calibration Loss: Minimizes the negative log-likelihood of observed photoelectron counts against the simulation's expected values.
• Tracking Loss: A composite loss combining vertex timing, charge distribution (Poisson NLL), and cone timing (quantile loss) to reconstruct particle tracks (vertex, direction, time, energy).

3. Key Contributions

1. First End-to-End Differentiable Optical Simulator: LUCiD is the first framework to jointly handle calibration and reconstruction for large-scale optical detectors within a single differentiable pipeline.
2. Unified Optimization: It eliminates artificial boundaries between simulation, calibration, and reconstruction, allowing for the simultaneous optimization of thousands of parameters (e.g., individual sensor efficiencies and medium optical properties).
3. Variance Reduction Techniques: Introduces specific methods like Fibonacci lattice sampling for calibration sources and photon relaxation to ensure gradients are stable and physically meaningful.
4. Modularity: The framework is geometry-agnostic and can be adapted to various detector shapes (cylinders, spheres, boxes) and materials without retraining reconstruction algorithms.

4. Results

The authors validated LUCiD using a Super-Kamiokande (SK)-like geometry (water Cherenkov detector).

• Calibration Performance:
  • Successfully recovered correlated medium parameters (scattering length, absorption length, wall/sensor reflection rates) from a central laser source, converging to true values from multiple random initializations.
  • Calibrated ~10,000 individual sensor quantum efficiencies simultaneously using 15 isotropic sources, achieving a residual RMS of 1.0% with no bias.
• Tracking Performance:
  • Reconstructed 500 muon events ($E \approx 1$ GeV).
  • Vertex Resolution: ~15.6 cm.
  • Angular Resolution: ~1.1°.
  • Momentum Resolution: ~1.5%.
  • Comparison: These metrics match or slightly surpass the state-of-the-art non-differentiable method FitQun (15.8 cm, 1.0°, 2.3%), despite LUCiD using a simplified expectation-based approach.
• Computational Efficiency:
  • Gradient evaluation adds only a ~10–15% overhead compared to forward prediction.
  • Reconstruction of a single event takes ~15 seconds (converging in ~100 iterations), comparable to existing methods but with the added benefit of gradient-based optimization.
• Scalability: The framework allows for rapid "what-if" analysis of detector designs (e.g., varying sensor counts) without retraining, demonstrating immediate utility for future experiment optimization.

5. Significance

This work establishes a new paradigm for experimental particle physics:

• From Discrete to Continuous: It transforms the analysis pipeline from a sequence of discrete, independent steps into a continuous, differentiable flow.
• Uncertainty Propagation: By maintaining differentiability, calibration uncertainties can be naturally propagated through reconstruction to final physics results, preserving correlations that are typically lost in traditional pipelines.
• Detector Design: It enables the direct optimization of detector geometry and sensor placement based on physics performance metrics, rather than relying on heuristic tuning.
• Future Readiness: The authors argue that this technique is ready for adoption in current and future experiments (e.g., Hyper-Kamiokande, DUNE Phase II), potentially superseding traditional simulation-based workflows with a more robust, efficient, and unified approach.

In summary, LUCiD demonstrates that differentiable programming can solve the "curse of dimensionality" in detector calibration and reconstruction, offering a path toward fully integrated, gradient-driven particle physics analysis.