📊 statistics

Estimating Complex Densities using Two-Stage Normalizing Flows

This paper proposes a Two-Stage Normalizing Flows framework that approximates and samples from complex, intractable target distributions by first learning densities from sample-only components and then integrating them with analytically specified terms, enabling robust inference in scenarios where the full target density is unavailable.

Original authors: Roxana Darvishi, David C. Stenning, Ted von Hippel, Owen G. Ward

Published 2026-03-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Roxana Darvishi, David C. Stenning, Ted von Hippel, Owen G. Ward

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a giant, complex jigsaw puzzle. But here's the catch: you don't have the picture on the box, and you don't have all the pieces in one pile.

Some pieces are scattered in a box labeled "Simulator." You can shake the box and pull out pieces, but you can't see the picture on them, and you can't write down a formula for what they look like.
Other pieces are sitting on your desk. You can see them perfectly, and you know exactly what they look like because you have a math formula for them.
The Goal: You need to assemble the entire picture to understand the system you are studying.

For a long time, statisticians had a hard time doing this. If they tried to use the "Simulator" pieces, they often had to throw away the "Math Formula" pieces, or vice versa. They had to guess the whole picture based on just one type of piece, which often led to a distorted or blurry final image.

This paper introduces a clever new method called Two-Stage Normalizing Flows. Think of it as a two-step assembly line that can mix and match these different types of puzzle pieces to build the perfect picture.

The Problem: The "Black Box" and the "Blueprint"

In many scientific fields (like astronomy or medicine), the "target" (the thing we want to understand) is a mix of two things:

The Black Box: A computer program that spits out data (samples) but won't tell you how it made them. It's like a magic machine that gives you apples, but you don't know the recipe.
The Blueprint: A known mathematical rule (like a law of physics) that describes part of the system perfectly.

Old methods were like trying to build a house using only the bricks from the magic machine, ignoring the blueprints. Or, they tried to force the blueprints to fit the bricks, which often resulted in a crooked house.

The Solution: The Two-Stage Assembly Line

The authors propose a two-stage process using a tool called Normalizing Flows. Imagine Normalizing Flows as a magical, stretchy clay mold. It can take a simple shape (like a ball of clay) and stretch, twist, and warp it until it looks exactly like a complex shape (like a dragon).

Stage 1: Learning the "Magic Machine" (The Forward Step)

First, the method looks at the pieces coming out of the "Black Box" (the simulator).

The Analogy: Imagine you have a bucket of mystery clay. You don't know the recipe, but you can feel the texture. You use a machine to learn how to stretch a simple ball of clay so that it mimics the shape of the mystery clay.
What happens: The computer learns to mimic the "Black Box" data perfectly. It creates a mathematical "shadow" of the simulator's output. Now, instead of a black box, we have a clear, stretchy model of those specific pieces.

Stage 2: Mixing the Shadow with the Blueprint (The Reverse Step)

Now, we have our "Shadow" of the Black Box pieces, and we still have our "Blueprint" pieces (the known math formulas).

The Analogy: Imagine you take your "Shadow" clay and your "Blueprint" clay and mix them together on a table. You now have a complete, messy pile of the entire puzzle.
What happens: The computer takes this complete mix and uses a second machine to learn how to stretch a simple ball of clay to match this entire new, complex shape.

Why is this a Big Deal?

Once this two-stage process is done, you have a superpower. You can now do two things that were previously impossible or very hard:

Generate New Samples: You can ask the model, "Give me a new, realistic example of this system," and it will instantly generate one. It's like having a 3D printer that can print any part of the puzzle you need.
Evaluate the Density: You can ask, "How likely is this specific scenario?" and the model can give you a precise number. It's like being able to look at any spot on the puzzle and say exactly how important that piece is.

Real-World Example: The Stars

The authors tested this on a real astronomy problem involving 30 white dwarf stars.

The Challenge: They had data from a complex computer simulation (the Black Box) about the stars' ages and masses, but they also had known physical laws (the Blueprint) about how stars behave.
The Old Way: The old method (called Two-Stage Fully Bayesian) was like trying to guess the shape of the stars by only looking at the simulation data. It worked, but the results were "jagged" and "noisy," like a low-resolution photo. It also got confused if the simulation started with a slightly different assumption.
The New Way: The Two-Stage Normalizing Flow method blended the simulation data with the physical laws smoothly. The result was a crisp, high-definition picture of the stars' properties. It didn't get confused by the simulation's quirks and revealed subtle details about the stars that the old method missed.

The Takeaway

This paper is about building a bridge between what we can simulate (but don't fully understand) and what we know mathematically (but can't easily simulate).

By using this two-stage "stretchy mold" approach, scientists can finally combine all their scattered pieces of information to build a complete, accurate, and flexible picture of complex systems, whether they are studying the birth of stars, the spread of diseases, or the behavior of financial markets. It turns a messy, incomplete puzzle into a clear, solvable image.

1. Problem Statement

In many scientific domains (e.g., astrophysics, biology), the target probability distribution (posterior or joint density) cannot be evaluated in closed form or sampled from directly. Instead, information about the distribution is often heterogeneous and fragmented:

Sample-based components: Some parts of the distribution are only accessible via samples generated by black-box simulators or external datasets (e.g., Monte Carlo draws from a specific likelihood-prior combination).
Analytic components: Other parts are known through tractable mathematical expressions or structural assumptions (e.g., specific priors, conditional independence structures).
Partial Observations: Data may exist for sub-vectors of variables from different sources without matched joint observations.

Limitations of Existing Methods:

MCMC/VI: Require full access to the target density (or unnormalized density) for evaluation, which is unavailable when likelihoods are intractable.
Non-parametric Estimators: Fail in high dimensions and cannot incorporate known analytic structures.
Approximate Bayesian Computation (ABC): Requires generating joint samples from the full model, which is impossible if only sub-vector samples are available.
Two-Stage Fully Bayesian (TSFB): An existing method for hierarchical models with inaccessible likelihoods. However, it relies on Metropolis-Hastings steps using software-generated samples as proposals. This makes it highly sensitive to the "software prior"; if the software samples do not overlap well with the true posterior, the method fails to capture the correct shape and range, leading to bias and truncation.

2. Methodology: Two-Stage Normalizing Flows (TSNF)

The authors propose a Two-Stage Normalizing Flows framework that integrates heterogeneous information sources to reconstruct a tractable approximation of the full target distribution. The method utilizes Normalizing Flows (NF), which define complex distributions via invertible transformations of a simple base distribution (e.g., standard normal).

Stage 1: Learning Sample-Based Components (Forward KL)

Goal: Approximate the unknown densities of components for which only samples are available.
Mechanism: For each sub-vector $y_i$ $y_{i}$ with available samples, a Forward KL Normalizing Flow is trained.
- The loss function minimizes $D_{KL}(f_{true} \parallel f_{flow})$ , which corresponds to maximizing the likelihood of the observed samples under the flow model.
- Output: A set of fitted density estimators $\hat{g}_i(y_i)$ that can be evaluated point-wise.
Stability Constraint: To ensure numerical stability for the next stage, the flow architecture in Stage 1 prioritizes stable Jacobian determinants (using simple layers) over extreme flexibility to prevent extreme values or discontinuities.

Stage 2: Reconstructing the Full Target (Reverse KL)

Goal: Combine the learned sample-based densities with the known analytic terms to form the full target distribution.
Mechanism:
- Construct an unnormalized target density: $\tilde{f}_X(x) \propto h(y_0) \prod \hat{g}_i(y_i)$ , where $h(y_0)$ represents the known analytic terms (e.g., priors, hyperpriors) and $\hat{g}_i$ are the Stage 1 outputs.
- Train a Reverse KL Normalizing Flow on this constructed target.
- The loss function minimizes $D_{KL}(f_{flow} \parallel \tilde{f}_X)$ , which requires evaluating the target density $\tilde{f}_X$ at points generated by the flow. Since $\tilde{f}_X$ is now computable (via the product of Stage 1 outputs and analytic terms), this is feasible.
Output: A single flow model $\hat{f}(x)$ that approximates the full joint distribution.

Capabilities

The resulting model supports both:

Efficient Sampling: Drawing samples from the base distribution and mapping them through the learned transformation.
Density Evaluation: Computing the probability density at any arbitrary point using the change-of-variables formula.

3. Key Contributions

Framework for Heterogeneous Data: A novel method to coherently integrate sample-based information and analytic expressions without requiring joint samples from the full model or direct evaluation of the intractable target density.
Robustness to Software Priors: Unlike the TSFB algorithm, the TSNF method explicitly cancels out the "software prior" in the Stage 2 objective function. This makes the inference robust even when the software-generated samples are shifted away from the true high-density regions of the posterior.
Dual Functionality: The framework provides a unified approach for both sampling and density evaluation, overcoming the limitation of methods that typically support only one of these tasks.
Scalability: Demonstrated effectiveness in high-dimensional settings (e.g., 151-dimensional astrophysical models) where standard non-parametric methods fail.

4. Results

The paper validates the method through simulation studies and a real-world case study:

Joint Density Inference (Simulation):
- The method successfully reconstructed a complex, highly non-linear 3D joint density using only samples from 2D sub-vectors and one known marginal.
- It accurately captured non-linear dependence structures and multimodality.
Hierarchical Models with Inaccessible Likelihoods (Simulation):
- Comparison: TSNF was compared against the Two-Stage Fully Bayesian (TSFB) algorithm.
- Findings: When the software prior was misspecified (shifting samples away from the true posterior), TSFB failed to recover the correct posterior shape, exhibiting truncation and bias. TSNF remained robust, accurately recovering marginal distributions, means, variances, and covariance structures.
- Metrics: TSNF showed significantly lower estimation errors (Frobenius norm of covariance error: 0.016 vs. 1.846 for TSFB).
Case Study: Astrophysics (White Dwarf Stars):
- Context: Inference on 30 white dwarf stars using the BASE-9 software suite, where the likelihood is intractable.
- Results: TSNF produced smoother and more distinct marginal posterior estimates compared to TSFB. TSFB estimates showed "jagged" artifacts due to the discrete nature of the Metropolis-Hastings proposals, whereas TSNF provided continuous density estimates.
- Observation: TSNF revealed meaningful variability between star-specific parameters that TSFB obscured due to its reliance on the software prior's proposal distribution.

5. Significance and Conclusion

The Two-Stage Normalizing Flows framework addresses a critical gap in statistical inference where data is fragmented and likelihoods are intractable. By decoupling the learning of sample-based components from the integration of analytic knowledge, the method offers a robust, flexible, and computationally efficient alternative to existing simulation-based inference techniques.

Its primary significance lies in its ability to:

Perform inference without direct access to the full target density.
Mitigate the bias introduced by "software priors" in black-box simulation scenarios.
Provide both density evaluation and sampling capabilities, enabling a wider range of inferential tasks (e.g., hypothesis testing, uncertainty quantification) in complex scientific applications.

The authors conclude that this approach is particularly valuable for modern scientific problems involving hierarchical structures and black-box simulators, offering a path forward where classical assumptions of tractability do not hold. Future work will focus on optimizing flow architectures for specific structures and developing diagnostic tools for model checking in Stage 2.