Almanac: MCMC-based signal extraction of power spectra and maps on the sphere

Almanac is a Hamiltonian Monte Carlo-based framework that extracts noiseless all-sky maps and their corresponding power spectra from noisy cosmological observations across multiple redshift bins, providing model-independent posterior data products that avoid issues like $EB$-leakage and enable robust diagnostics of systematic errors or new physics.

The Gist

Imagine the universe as a giant, three-dimensional canvas covered in a chaotic, swirling fog of matter and energy. Astronomers try to take a picture of this fog, but their cameras are imperfect: the images are grainy (noisy), and sometimes parts of the sky are blocked by clouds or the camera's own blind spots (masks).

The paper introduces a new tool called Almanac (which stands for MCMC-Based Signal Extraction of Power Spectra and Maps on the Sphere). Think of Almanac not as a camera, but as a super-smart detective that can look at these grainy, incomplete photos and reconstruct the entire original, clear picture of the universe, along with a detailed statistical report on how the fog is organized.

Here is how it works, broken down into everyday concepts:

1. The Problem: The Grainy, Patchy Photo

When we look at the Cosmic Microwave Background (the afterglow of the Big Bang) or map the distribution of galaxies, we get data that is:

• Noisy: Like static on an old TV.
• Incomplete: We can't see the whole sky at once; some parts are hidden.
• Complex: The data isn't just a simple picture; it's a mix of different "types" of waves (like how sound has pitch and volume). In physics, these are called "spin-weight 0" (like temperature) and "spin-weight 2" (like polarization or the twisting of light).

Traditional methods often try to take a single "best guess" (a point estimate) of what the universe looks like. The paper argues this is like trying to guess the weather by looking at a single snapshot; you miss the full story and the uncertainty.

2. The Solution: The "All-Seeing" Detective

Almanac uses a technique called Hamiltonian Monte Carlo (HMC).

• The Analogy: Imagine you are in a dark, foggy room trying to find the shape of a giant, invisible sculpture. You can only feel small parts of it.
  • Old methods might feel one spot, guess the shape, and stop.
  • Almanac is like a detective who doesn't just guess one shape. Instead, it explores thousands of possible shapes that fit the clues you have. It creates a "cloud" of possibilities, showing you not just what the sculpture likely looks like, but exactly how sure (or unsure) it is about every curve and corner.

3. How It Handles the "Messy" Data

The paper highlights two major tricks Almanac uses to solve the puzzle:

• The "Cholesky" Trick (Untangling the Knots):
  The math behind the universe involves complex relationships between different parts of the sky. If you try to solve this directly, the math gets tangled like a knot of headphones. The authors found that using a specific mathematical "untying" method (called Cholesky decomposition) makes the knot fall apart, allowing the detective to move much faster and more accurately through the possibilities.
• The "No Preconceptions" Rule:
  Many tools assume a specific theory about how the universe works (e.g., "The universe is made of 5% normal matter"). Almanac refuses to make these assumptions. It only assumes the universe looks roughly the same in all directions (isotropy). It says, "Show me the data, and I will tell you what the patterns are, without forcing them into a pre-made box." This means the results are "model-independent"—they are pure facts derived from the data itself.

4. The "Leakage" Problem (E and B Modes)

In cosmology, there are two types of patterns: E-modes (like electric fields, which are "curl-free") and B-modes (like magnetic fields, which are "divergence-free").

• The Issue: Because our view of the sky is blocked (masked), traditional tools often get confused. They might mistake a little bit of an E-mode for a B-mode. This is called "leakage." It's like hearing a siren and thinking it's a car horn because the wind is blowing.
• The Almanac Fix: Because Almanac looks at the entire probability cloud rather than a single guess, it understands that E and B are linked in the masked areas. It doesn't "leak" the confusion into the final result. If it sees a B-mode signal where there shouldn't be one, it flags it as a potential error or a sign of new physics, rather than just a calculation mistake.

5. The Results: What Did They Find?

The team tested Almanac on simulated data that looks like the Cosmic Microwave Background (CMB).

• Temperature (Spin-0): They successfully reconstructed the temperature map of the universe, even in the "noisy" and "masked" parts.
• Polarization (Spin-2): They reconstructed the twisting patterns of light. They showed that Almanac can accurately find the strong signals (E-modes) while correctly identifying that the weak signals (B-modes) are consistent with zero (or noise), without creating fake signals.

6. Why It Matters (Without Overpromising)

The paper claims that Almanac is a powerful tool for characterizing the statistical properties of the universe.

• It produces "science-ready" data products.
• It handles millions of parameters at once (a task that would crash older computers).
• It is designed to work with future, massive surveys (like the Euclid mission) that will map huge chunks of the sky.

In short: Almanac is a new, highly efficient mathematical engine that takes noisy, incomplete pictures of the universe and reconstructs the most likely "true" maps and patterns, while rigorously accounting for uncertainty and avoiding common calculation errors. It does this without forcing the data to fit a specific theory, letting the universe speak for itself.

Technical Summary

Problem Statement
Cosmological inference frequently relies on analyzing noisy observations of random fields on the celestial sphere, such as Cosmic Microwave Background (CMB) maps, continuous maps of cosmic structure, or point tracers. Traditional methods often rely on quadratic estimators of two-point statistics (e.g., angular power spectra). While these estimators are unbiased under specific conditions, they suffer from well-known drawbacks: they fail to capture the full information content of non-Gaussian fields or non-optimal statistics, require complex modeling of survey geometry effects on covariance matrices, and struggle with leakage between E- and B-modes (particularly in CMB polarization and cosmic shear). Furthermore, point estimates of two-point statistics do not fully utilize the available information in modern, high-resolution surveys.

Methodology
The paper introduces Almanac, a software package designed for field-level inference (FLI) using Hamiltonian Monte Carlo (HMC) sampling. Unlike traditional approaches that marginalize over field values analytically or rely on point estimates, Almanac performs a Bayesian hierarchical inference to simultaneously sample the underlying all-sky, noiseless maps of cosmic structures and their auto- and cross-power spectra across multiple redshift bins.

Key methodological components include:

• Bayesian Hierarchical Model (BHM): The model assumes a statistically isotropic and Gaussian random field on the sky (a maximum-entropy assumption). It treats the spherical harmonic coefficients ($a$) and the power spectra ($C$) as free parameters. The posterior distribution combines a likelihood of the data given the maps and noise, a Gaussian prior on the coefficients given the power spectra, and a prior on the power spectra themselves.
• Spin-Weight Handling: The algorithm handles both spin-weight 0 fields (e.g., CMB temperature, galaxy number density) and spin-weight 2 fields (e.g., CMB polarization, cosmic shear). For spin-weight 2 fields, it infers E- and B-mode power spectra and parity-violating EB power, avoiding the EB-leakage problem inherent in point estimators by sampling the full posterior.
• Hamiltonian Monte Carlo (HMC): To navigate the high-dimensional parameter space (involving millions of parameters, e.g., $\sim 10^7$ for CMB and $\sim 10^8$ for Euclid-like weak lensing), Almanac utilizes HMC. This requires the computation of gradients of the potential energy (negative log-posterior), which are derived analytically.
• Parameterization and Tuning: To address the non-linear constraints of positive-definiteness for the covariance matrix $C$ and the "funnel" geometry of the posterior, the authors implement a Cholesky decomposition parameterization ($C = LL^T$) with logarithmic scaling of diagonal elements. This decorrelates modes and normalizes them, significantly improving sampling efficiency compared to naive parameterizations. The sampler employs a four-stage tuning process: burn-in, step-size tuning based on Hessian-derived scales, leapfrog tuning to optimize acceptance rates, and a main sampling stage.
• Convergence Diagnostics: The paper employs several diagnostics to monitor convergence in high dimensions, including the Gelman-Rubin statistic, the Fraction of Missing Information (FMI), Hanson's statistic, and Effective Sample Size (ESS). FMI is highlighted as a particularly sensitive early-warning indicator of poor sampling efficiency.

Key Contributions

1. Development of Almanac: A generic, all-sky signal extraction tool capable of handling multiple fields (spin-0 and spin-2) simultaneously.
2. Full Posterior Sampling: By sampling the full posterior distribution of maps and spectra rather than point estimates, Almanac provides science-ready data products that are independent of specific cosmological models (beyond statistical isotropy) and naturally handles EB-leakage.
3. Scalability: The algorithm is demonstrated to handle extremely high-dimensional posteriors (up to hundreds of millions of parameters) efficiently, scaling nearly quadratically with the maximum multipole $\ell_{max}$ ($t \propto \ell_{max}^{2.03}$) on multi-core architectures.
4. Robustness to Noise and Masks: The method infers field values in masked (unobserved) regions by synthesizing information from observed pixels, providing a Bayesian solution to missing data without introducing boundary artifacts common in flat-sky approximations.

Results
The authors demonstrate Almanac's performance on simulated CMB-like experiments:

• Temperature (Spin-0): In a low signal-to-noise scenario, Almanac successfully recovered the fiducial angular power spectra within 1- and 2-sigma credible intervals, despite starting from dispersed pseudo-spectra containing significant noise.
• Polarization (Spin-2): The algorithm handled a mix of high signal-to-noise (E-modes) and very low signal-to-noise (B-modes). The inferred E-modes were accurate, while the B-modes were correctly inferred as consistent with zero.
• Convergence: The chains demonstrated convergence across multiple diagnostics. The Fraction of Missing Information (FMI) values were approximately 0.80 for temperature and 0.77 for polarization, indicating efficient exploration of the energy landscape. Effective sample sizes reached $10^4$–$10^5$ for well-detected modes.
• Map Reconstruction: The method successfully reconstructed mean posterior maps and typical realizations, showing expected Wiener-filter-like smoothing and larger variances in masked regions.

Significance
The paper positions Almanac as a crucial tool for the next generation of cosmological surveys (e.g., Euclid, CMB-S4) that cover large fractions of the sky and require curved-sky approaches to avoid biases. Its primary significance lies in its ability to extract maximum statistical information from field data without relying on a specific cosmological model, thereby providing model-independent data products. This allows for a separation of signal extraction from cosmological parameter inference. The authors note that while the computational cost is higher than traditional summary statistic analysis, the improved inference accuracy and the ability to diagnose systematic errors (via non-zero B-modes or EB leakage) justify the expense. The work serves as a foundation for more complex analyses involving correlated fields, which are detailed in a companion paper.