Binary Classification of Light and Dark Time Traces of a Transition Edge Sensor Using Convolutional Neural Networks

This paper investigates the use of convolutional neural networks (CNNs) as binary classifiers to distinguish photon-triggered pulses from background noise in transition edge sensors for the ALPS II experiment, finding that the CNN approach failed to outperform traditional cut-based analysis and suggesting that regression or unsupervised models may be more effective for future signal processing.

The Gist

The Tale of the Cosmic Needle in a Haystack

Imagine you are a detective trying to find a single, specific type of needle in a massive, messy haystack. This needle is special: it’s a "Light Needle" (a photon from a laser), and it’s incredibly rare. You are looking for it because it might prove the existence of "Axions"—mysterious, ghostly particles that could explain how the universe works.

To find these needles, you use a super-sensitive machine called a TES (Transition Edge Sensor). Think of the TES as a high-tech, microscopic scale that can feel the weight of a single grain of dust.

The Problem: The "Fake" Needles

The problem is that the haystack is full of "Dark Needles" (background noise). Most of these are easy to spot—they are big, heavy, or shaped weirdly. But there is one type of fake needle that is a nightmare: The Black-Body Impostor.

These impostors are caused by heat leaking into the system through a fiber optic cable. They look, feel, and weigh almost exactly like the real Light Needles. They aren't just similar; they are nearly identical twins.

The Plan: Hiring a Robot Detective

The researchers decided to hire a "Robot Detective"—a Convolutional Neural Network (CNN).

In the world of AI, a CNN is like a detective with a magnifying glass that is incredibly good at spotting tiny patterns in shapes. Instead of a human looking at the "weight" of the needles, the robot looks at the "wiggle" of the signal (the time trace) to see if it’s a real photon or a heat-induced fake.

The researchers spent a lot of time "training" the robot. They showed it thousands of examples: "This is a real needle, this is a fake one, this is a real one..." They even tweaked the robot's settings (called "hyperparameters") over and over to make it the smartest detective possible.

The Twist: The Robot Gets Confused

Here is where the story takes a turn. Even after all that training, the Robot Detective wasn't as good as the old-fashioned method.

The old method was just a human using a simple checklist (a "cut-based analysis"). The human said: "If it weighs between X and Y, it's a needle." Surprisingly, the human's simple checklist actually worked better than the high-tech robot.

Why did the robot fail?
The researchers discovered a phenomenon called "Training Confusion."

Imagine you are teaching a child to distinguish between a real apple and a very realistic plastic apple. But, in your pile of "plastic apples," you accidentally mixed in a few real apples. Every time the child picks up a real apple from the "plastic" pile, you yell, "No! That's plastic!"

Eventually, the child gets confused. They stop trusting their eyes because your instructions contradict reality.

This is exactly what happened to the CNN. Because the "Dark" pile (the background) actually contained some real "Light" photons (the black-body impostors), the robot was being told that real signals were actually noise. This "label noise" broke the robot's brain.

The Lesson Learned

The researchers didn't view this as a failure, but as a breakthrough in understanding. They learned two big things:

1. Better Data > Better Robots: You can have the smartest AI in the world, but if you feed it "dirty" data (confusing labels), it will never be smarter than a simple checklist.
2. The Hardware Fix: To truly win this game, we shouldn't just try to build better detectives; we need to clean up the haystack. They suggest using physical filters (like a pair of specialized sunglasses) to block out the heat-induced impostors before they ever reach the sensor.

In short: The researchers tried to use AI to find a needle in a haystack, but the haystack was so full of "impostor needles" that it confused the AI. The solution isn't just a smarter robot—it's a cleaner haystack.

Technical Summary

Technical Summary: Binary Classification of Signal and Background Triggers of a Transition Edge Sensor Using Convolutional Neural Networks

1. Problem Statement

The Any Light Particle Search II (ALPS II) experiment aims to detect axions and axion-like particles (ALPs) using a "light shining through a wall" technique. A critical component of this experiment is the detection of reconverted 1064 nm photons using Transition Edge Sensors (TES). To achieve the required sensitivity (on the order of $10^{-24}$ W or $10^{-5}$ Hz), the detector must distinguish extremely rare signal pulses from a background of "dark" pulses.

The primary challenge is the extrinsic background, specifically near-1064 nm black-body radiation coupled into the system via optical fibers. These black-body photons produce pulse shapes that are physically nearly indistinguishable from the 1064 nm signal pulses due to the energy resolution limits of the TES. The researchers sought to determine if Convolutional Neural Networks (CNNs) could outperform traditional "cut-based" analysis (manual filtering based on pulse parameters) in suppressing this background.

2. Methodology

The study employed a supervised machine learning approach to perform binary classification on univariate time-series data.

• Data Acquisition: The researchers collected 3,898 "light" pulses (triggered by a 1064 nm laser) and 8,872 "dark" pulses (measured with the laser disconnected). Data was sampled at 50 MHz.
• Preprocessing: Pulses were truncated to a 24 $\mu$s window, voltage offsets were removed, and a minor filter was applied to remove outliers.
• CNN Architecture: The model utilized a series of convolutional layers (using tanh activation) and average pooling layers, followed by dropout and dense layers (using ReLU activation).
• Optimization: The researchers performed an extensive hyperparameter optimization using 2,000 iterations of random search, tuning the number of layers, filters, kernel size, dropout rate, learning rate, and batch size.
• Evaluation Metrics: Performance was evaluated using:
  • Detection Significance ($S$): A metric tailored to the ALPS II physics goals, balancing background rate ($n_b$) and analysis efficiency ($\epsilon_a$).
  • F1 Score: To measure the balance between precision and recall.
• Comparative Analysis: The CNN performance was benchmarked against a cut-based analysis that used fitting parameters (rise/decay times, $\chi^2$ errors, and peak height in the frequency domain) to define signal regions.

3. Key Contributions

• Investigation of ML in TES Physics: The paper provides a rigorous test of whether deep learning can surpass traditional statistical cuts in superconducting microcalorimeter data.
• Identification of "Training Confusion": The study identifies that the presence of near-1064 nm black-body photons in the "dark" dataset acts as label noise. Because these background photons are physically similar to the signal, training a classifier to label them as "dark" confuses the model.
• Relabeling Experiment: To prove the impact of label noise, the authors performed a "relabeling" experiment, where misclassified dark pulses (presumed to be black-body photons) were re-categorized as "light" pulses and used for retraining.

4. Results

• CNN vs. Cut-Based Analysis: The optimized CNN did not outperform the traditional cut-based analysis. The cut-based method achieved a detection significance of $S = 1.29 \pm 0.03$, while the CNN ensemble achieved $S = 0.95 \pm 0.23$ (approximately 36% lower).
• Impact of Label Noise: After relabeling the misclassified dark pulses as "light" and retraining, the CNN's significance improved to $S = 1.61 \pm 0.58$. While this showed the CNN had potential, it still did not consistently beat the cut-based method in the presence of the actual experimental dataset.
• Background Characterization: The study confirmed that the majority of misclassified dark pulses were indeed near-1064 nm black-body photons, as evidenced by Principal Component Analysis (PCA) and comparison with Planck’s spectrum theoretical rates.

5. Significance and Outlook

The paper concludes that for the current TES setup, hyperparameter optimization is not the bottleneck for machine learning performance; rather, the quality and purity of the training data are the limiting factors.

Recommendations for future work include:

1. Moving from Classification to Regression: Instead of binary classification (light vs. dark), future models should use regression-based CNNs to predict the specific wavelength of a photon, which would allow for better discrimination.
2. Data Structuring: Emphasis should be placed on creating standardized, "clean" training datasets using simulation frameworks.
3. Hardware Solutions: Ultimately, the authors argue that reaching the $10^{-5}$ Hz background goal requires hardware-level improvements, such as installing cryogenic narrow bandpass optical filters to physically block the black-body radiation before it reaches the sensor.