🔭 astrophysics

Hermes - Towards an Optimal High-Performance Algorithm for Cosmic Statistics of Large Data Sets

The paper introduces Hermes, an open-source, in situ multiresolution framework that efficiently computes diverse cosmic large-scale-structure statistics from discrete catalogues by reconstructing continuous density fields and replacing explicit particle counting with algebraic operations, thereby offering a scalable and flexible solution for current and future large-scale galaxy surveys.

Original authors: Long-long Feng, Tengpeng Xu, Tian-Cheng Luan, Jiawei Li, Xin Sun, Wenjie Ju, Zhuoyang Li, Shiyu Yue, Weishan Zhu, Yan-Chuan Cai

Published 2026-07-28

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Long-long Feng, Tengpeng Xu, Tian-Cheng Luan, Jiawei Li, Xin Sun, Wenjie Ju, Zhuoyang Li, Shiyu Yue, Weishan Zhu, Yan-Chuan Cai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the universe as a giant, invisible ocean made of dark matter and galaxies. For decades, astronomers have tried to understand how this ocean moves and swirls by counting how many "islands" (galaxies) sit next to each other. This is called studying "clustering." If you count how many islands are a short distance apart versus a long distance apart, you get a map of the universe's structure. But here's the catch: as the universe gets older, gravity pulls these islands together in messy, complicated ways that aren't just simple pairs. To understand the full story, scientists need to count groups of three, four, or even more islands at once. The problem is, counting these groups is like trying to find every possible trio of people in a stadium of a billion people. It takes so much computer power that it's almost impossible to do for the massive surveys coming soon.

This is where a new tool called Hermes (and its Python version, PyHermes) comes in. Think of the old way of doing things as a librarian trying to count every single book in a library by walking up to each shelf and checking the spine one by one. It's slow, tedious, and if you want to count books in a different way (like by color or author), you have to walk the aisles all over again. Hermes changes the game by turning the entire library into a smooth, glowing fog of light. Instead of counting individual books, the computer looks at the brightness of the fog in different spots. If you want to know how many books are in a specific section, you just measure the light in that area. If you want to count trios, you don't walk around looking for three books; you just mix the light from three different areas together mathematically. This paper shows that by using this "fog" method, scientists can measure the complex patterns of the universe much faster and more flexibly than before, opening the door to analyzing the billions of galaxies that future telescopes will discover.

The Old Way: Counting Stars One by One

Imagine you are at a massive concert, and you want to know how the crowd is moving. The traditional way to study this is to pick a person, look around, and count how many other people are standing exactly 5 feet away, then 10 feet away, then 15 feet away. Then you pick a different person and do it all over again. If you have a million people, you have to do this millions of times. This is what astronomers call "pair counting." It works fine for small crowds, but when you have billions of galaxies, it's like trying to count every grain of sand on a beach by picking them up one by one. It takes forever.

Even worse, if you want to study how groups of three people interact (maybe they are dancing in a triangle), the math gets crazy complicated. You have to find every possible trio. For a billion galaxies, the number of trios is so huge that even the world's fastest supercomputers would struggle to finish the job in a reasonable time.

The New Way: Turning the Crowd into a Smooth Fog

The authors of this paper, led by Long-long Feng and his team, propose a clever shortcut. Instead of counting individual galaxies, they turn the whole crowd into a smooth, continuous "fog" or "density field."

Think of it like this: Imagine you have a bucket of sand. If you want to know how much sand is in a specific spot, you could count every single grain. Or, you could just look at how deep the sand is in that spot. The "Hermes" method does the latter. It takes the discrete points (the galaxies) and reconstructs them as a smooth, continuous map, kind of like turning a pixelated image into a high-definition photo.

Once the galaxies are turned into this smooth fog, the computer doesn't need to count pairs or trios anymore. Instead, it uses a set of mathematical "filters" (called window functions) to measure the fog.

The Filter: Imagine holding a cookie cutter over the fog. If you press it down, you measure how much fog is inside the cutter.
The Magic: In the old method, changing the shape of the cookie cutter (maybe from a circle to a square) meant you had to start counting from scratch. In the Hermes method, you just change the shape of the filter in the math, and the computer instantly recalculates the result using the same smooth fog.

What They Found: Speed and Flexibility

The team built a software tool called PyHermes to test this idea. They used it on simulated data from the "Quijote" and "Kun" projects, which are like massive video games of the universe containing millions of fake galaxies.

Here is what they discovered:

It's Fast: Because the computer is doing algebra on the smooth fog instead of counting billions of pairs, it is incredibly fast. They showed that they could measure complex patterns (like how three galaxies form a triangle) in a fraction of the time it would take with old methods.
It's Flexible: The best part is that you can ask different questions without rebuilding the whole map. If you want to measure how galaxies cluster in a sphere, a ring, or a weirdly shaped blob, you just change the "window" (the cookie cutter) in the code. The underlying fog stays the same.
It's Accurate: They compared their results with the old, trusted methods and found that as long as the "fog" is detailed enough (high resolution), the results are identical. They even showed that they could measure things like the "velocity divergence" (how fast the fog is flowing) and "gravitational potential" (how deep the gravity wells are) just by applying different math filters to the same map.

Why This Matters for the Future

We are about to enter a new era of astronomy. Telescopes like the DESI, Euclid, and the Nancy Grace Roman Space Telescope are about to map out billions of galaxies. These datasets are so huge that the old "counting" methods will simply be too slow to handle them.

Hermes offers a way to keep up. By treating the universe as a smooth, mathematical field rather than a bag of discrete dots, it allows scientists to ask complex questions about the "dark energy" pushing the universe apart, the "neutrinos" zipping through space, and the mysterious "dark matter" holding it all together. It's not just a faster calculator; it's a new way of seeing the universe that turns a mountain of counting into a single, elegant calculation.

In short, the paper suggests that by changing how we look at the data—from counting individual stars to measuring the smooth glow of the cosmos—we can unlock secrets of the universe that were previously hidden by the sheer weight of the numbers.

Technical Summary: Hermes – Towards an Optimal High-Performance Algorithm for Cosmic Statistics of Large Data Sets

Problem Statement
The analysis of cosmic large-scale structure (LSS) is entering an era of unprecedented volume and statistical precision driven by surveys such as DESI, Euclid, and the Vera C. Rubin Observatory. A primary bottleneck in this regime is the computational cost of measuring clustering statistics. Conventional methods rely on explicit "ex situ" counting of particle tuples (pairs, triplets, etc.), which scales as $\mathcal{O}(N_g^N)$ for an $N$ -point correlation function (NPCF) with $N_g$ objects. This brute-force approach becomes impractical for modern catalogs containing millions to billions of tracers. Furthermore, existing software packages are often designed for specific statistics or estimators, lacking a unified framework that treats different cosmic statistics as variations of a single underlying measurement. Additionally, the need for repeated evaluations of clustering statistics and reliable covariance estimation from large ensembles of mock catalogues imposes severe computational demands.

Methodology
The paper introduces Hermes (HypER-speed MultirEsolution cosmic Statistics), a framework that replaces discrete particle counting with algebraic operations on continuous, window-filtered density fields. The core methodology rests on three pillars:

Multiresolution Analysis (MRA) Field Reconstruction: Discrete point distributions (e.g., galaxy or halo catalogs) are reconstructed as continuous density fields using a compactly supported basis of scaling functions (specifically Daubechies D4 wavelets). This transforms the singular density field into a set of expansion coefficients on a grid. The resolution level $J$ controls the trade-off between spatial localization and smoothness.
Window-Operator Formalism: Statistical estimators are defined not by counting algorithms, but by window functions ( $W$ ). A statistical measurement is formulated as a spatial product of locally filtered fields. For example, a two-point correlation function (2PCF) is the inner product of a density field and its window-filtered counterpart. This unifies conventional binning schemes (e.g., spherical shells, rings) and generalized statistics (e.g., marked correlations, differential operators) under a single mathematical framework where changing the window kernel modifies the statistic without redesigning the estimator.
In Situ Algebraic Evaluation: Instead of iterating over particle pairs, the framework computes statistics via operations on the MRA coefficients.
- 2PCF: Evaluated as a scalar product of the original field coefficients and the window-convolved field coefficients.
- NPCF (Monopoles): Higher-order statistics are computed by constructing "folded" density fields (e.g., a two-fold field for 3PCF) through local stencil contractions of scaling coefficients using precomputed "connection coefficients." This reduces the complexity from enumerating $N$ -tuples to $\mathcal{O}(N_{grid})$ operations, independent of the number of particles.
- Multipole Decomposition: Anisotropic statistics (e.g., in redshift space) and higher-order multipoles are handled by decomposing the window kernels into spherical harmonics and performing the necessary contractions in the coefficient space.

The framework is implemented in PyHermes, an open-source Python toolkit utilizing FFT-based window convolutions, MPI/thread parallelism, and CUDA acceleration for high-order contractions.

Key Contributions

Unified Framework: The paper formulates a broad class of statistics—including counts-in-cells (CIC), 2PCF, 3PCF, multipole statistics, filtered variances, and operator-based measurements (e.g., gravitational potential, velocity divergence)—as window operations on multiresolution density fields.
Generalized Estimators: It demonstrates that conventional 2PCF and 3PCF estimators correspond to specific window choices within this framework. The formalism naturally extends to weighted, marked, and higher-order correlation functions by modifying the window kernel.
PyHermes Implementation: A high-performance, open-source Python library is released, featuring:
- Reusable reconstructed fields (SFCField) that decouple field reconstruction from statistical measurement.
- Support for arbitrary window functions (built-in and user-defined).
- Hybrid parallelization (MPI + Numba threads) and GPU acceleration for computationally intensive contractions.
Validation and Benchmarks: The framework is validated using halo catalogues from the Quijote and Kun simulation suites.
- Convergence: Measurements of 2PCF and 3PCF monopoles show convergence with increasing MRA resolution ( $J$ ), agreeing with conventional pair-counting codes (e.g., Corrfunc, TreeCorr) at sufficient resolution.
- Flexibility: Demonstrations include marked correlations (power-law and inverse-density marks), redshift-space anisotropic statistics using various window geometries (rings, shells), and the construction of derived physical fields (velocity divergence, Newtonian potential).
- Performance: Benchmarks indicate that the computational cost scales with the number of grid cells and window operations rather than the number of particles. For 3PCF multipole estimators, GPU acceleration of the final contraction stage yields a speedup of $\sim 4.1\text{--}4.7\times$ compared to CPU-only execution for that stage, with an overall workflow speedup of $\sim 1.8\text{--}1.9\times$ .

Results

Accuracy: The PyHermes measurements of the isotropic 2PCF and 3PCF monopoles match theoretical emulator predictions and conventional pair-counting results when the MRA grid resolution is fine enough to resolve the binning scales.
Scalability: The algorithm successfully handles large datasets (e.g., $\sim 4 \times 10^5$ haloes) with computational costs dominated by the grid size ( $N_{grid}$ ) and the number of window applications, rather than the combinatorial explosion of particle pairs.
Versatility: The framework successfully constructs complex statistics, such as band-averaged power spectra using wavelet filters and multipole-decomposed 3PCFs, without requiring separate counting algorithms for each statistic.

Significance and Claims
The paper claims that Hermes provides a unified computational framework where a single reconstructed field can be reused for a wide range of standard and customized cosmic statistics. By separating the field representation from the statistical windows, the framework enables scalable analysis of the increasingly large datasets expected from current and future galaxy surveys.

The authors position PyHermes not merely as a faster pair-counter for specific statistics, but as a flexible statistical infrastructure for precision cosmology. It allows for the efficient extraction of non-Gaussian information (via higher-order correlations, marked statistics, and density-split statistics) and the construction of derived physical fields (potential, acceleration) within a single pipeline. The paper asserts that this approach is particularly suited for the regime where many statistics, complex window geometries, or high-order correlations are required from large data sets, addressing the computational bottlenecks that hinder the full exploitation of next-generation surveys like DESI, Euclid, and CSST.

The Old Way: Counting Stars One by One

The New Way: Turning the Crowd into a Smooth Fog

What They Found: Speed and Flexibility

Why This Matters for the Future

More like this