⚛️ lattice

Symbolic Density Estimators for Unnormalized Distributions

This paper presents a framework that combines deep generative models with symbolic regression to estimate interpretable, symbolic forms of unnormalized probability distributions from observed samples, demonstrating its effectiveness on multivariate toy data and complex computational physics systems like the XY model and $\phi^4$ theory.

Original authors: Vikas Kanaujia, Riyansha Singh, Shashank Sharma, Vipul Arora

Published 2026-10-09

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Vikas Kanaujia, Riyansha Singh, Shashank Sharma, Vipul Arora

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the physical sciences, understanding how the world works often comes down to describing how things are likely to behave. Imagine a system made of countless tiny parts, like atoms in a solid or spins in a magnetic material. These parts do not move randomly; they follow rules that make some arrangements of the system much more likely than others. Scientists describe these likelihoods using a mathematical map called a probability distribution. Within this map, there is a hidden landscape of energy, where the system naturally settles into the lowest valleys. This energy landscape, often called a Hamiltonian, is the key to predicting how the system will react to heat, pressure, or time. For decades, physicists have relied on their own intuition and deep theoretical knowledge to write down the equations that describe these energy landscapes. They would guess the form of the equation and then tweak the numbers to fit the data they observed. While this approach has worked well, it is slow, labor-intensive, and entirely dependent on human expertise. If the underlying rules are too complex or the system behaves in ways that defy simple guesses, the traditional method hits a wall.

A team of researchers at IIT Kanpur and KU Leuven has developed a new way to bypass this human bottleneck. They created a framework that allows a computer to look at raw data from a complex system and automatically write down the mathematical equation that describes its energy. This is not just about finding a curve that fits the points; it is about discovering the symbolic language of physics itself. The researchers combined two powerful types of artificial intelligence. First, they used deep generative models, which are excellent at learning the shape of complex data clouds and telling us how likely any specific arrangement of particles is. Second, they used symbolic regression, a technique that searches for simple, readable mathematical formulas rather than just black-box predictions. By feeding the insights from the first AI into the second, the system can translate the messy, high-dimensional data of a physical system into a clean, interpretable equation.

The researchers tested this method on several challenging scenarios, ranging from simple toy distributions to complex models used in computational physics. In one set of experiments, they looked at systems with many "wells," or deep energy valleys, which are notoriously difficult to map because the data is scattered across many different states. They also examined the XY model, a classic problem in physics that describes how spins on a grid interact, and the $\phi^4$ theory, which is used to study how fields behave in particle physics. In every case, the framework successfully recovered the underlying energy equations directly from the data. For the simpler systems, the computer found symbolic approximations of the formulas that human experts would have written down, but without needing any prior knowledge of what those formulas should look like. It correctly identified the relationships between variables and the specific numbers that govern them, even when the data was high-dimensional and noisy, though the results are best understood as compact symbolic approximations rather than exact identities.

What makes this work particularly significant is how it handles the most difficult part of the problem: the sheer number of variables. In systems with thousands of interacting parts, the search space for a mathematical formula becomes impossibly large. To solve this, the researchers introduced a strategy of breaking the system down into small, local patches. Instead of trying to find one giant equation for the whole system, the AI looks at small neighborhoods of interacting parts, finds the rule for that small piece, and then stitches those rules together. This approach, combined with a technique that adds controlled noise to the data to ensure the AI sees rare but important high-energy states, allowed the system to work effectively even in high-dimensional settings where other methods fail. The results showed that the proposed framework could estimate compact, symbolic approximations of the energy function that were not only accurate but also physically meaningful.

The team also applied their method to a problem known as renormalization, which is akin to zooming out on a map to see the big picture while ignoring the tiny details. In physics, this involves taking a fine-grained system and creating a simpler, effective version that behaves the same way at a larger scale. Traditionally, deriving the equation for this larger-scale version is a massive analytical challenge, often impossible to solve by hand in complex, non-perturbative regimes. The researchers fed their framework data from a coarse-grained lattice and asked it to find the new energy equation. The system successfully produced symbolic expressions that matched the behavior of the original system across different scales. These expressions were consistent with the known physics and reproduced the correct statistical properties of the system, demonstrating that the framework can uncover effective laws of nature directly from data, even when the underlying derivation is too complex for traditional analytical methods.

The study explicitly ruled out the idea that this is merely a curve-fitting exercise. The researchers compared their approach to existing methods that try to guess the form of an equation based on a fixed set of building blocks. Those older methods often failed to converge or produced equations with incorrect coefficients and extra, spurious terms that did not belong. In contrast, the new framework, by integrating deep learning with symbolic search and using local factorization, consistently found the correct structure. The researchers noted that while the method relies on some prior knowledge, such as knowing which types of mathematical functions are physically relevant, it does not need to know the final answer in advance. The results were robust across multiple random trials, showing that the framework reliably discovers the correct interactions and suppresses irrelevant ones.

Ultimately, this work offers a new path for scientific discovery. It suggests that we can move beyond the limitations of human intuition when it comes to formulating the laws that govern complex systems. By automating the process of finding the mathematical language of nature, the framework allows scientists to harness the power of machine learning to analyze vast datasets while still retaining the clarity and insight of a human-readable equation. The researchers demonstrated that this approach works for a wide variety of systems, from simple statistical distributions to the intricate lattices of particle physics, providing a tool that can estimate the energy functions of systems that were previously too difficult to model analytically. The findings indicate that with the right combination of deep learning and symbolic reasoning, we can uncover the hidden rules of the universe directly from the data it leaves behind.

Technical Summary: Symbolic Density Estimators for Unnormalized Distributions

Problem Statement
Estimating the symbolic or analytical form of probability density functions (PDFs) from observed samples is a fundamental challenge in statistical and computational modeling. While deep generative models (such as normalizing flows and score-based models) excel at sampling from complex distributions, they function as "black box" estimators that do not provide interpretable analytical expressions for the underlying energy functions or Hamiltonians ( $H(x)$ ). Conversely, traditional Symbolic Regression (SR) methods struggle to scale to high-dimensional data and often fail when the distribution is unnormalized or when the search space is too large. The core problem addressed is the automatic inference of an analytical, symbolic representation of an unnormalized distribution directly from observed data, without relying heavily on manual domain expertise to select functional forms.

Methodology
The authors propose a two-stage framework that integrates deep generative modeling with symbolic regression to convert an unsupervised density estimation problem into a supervised symbolic regression problem.

Generative Modeling Stage: The framework first trains a deep generative model on observed samples. The authors utilize two types of models:
- Likelihood-based models (Normalizing Flows): These provide the negative log-likelihood ( $-\log p_\theta(x)$ ), which serves as the target for the energy function up to a constant.
- Score-based models: These estimate the score function $s(x) = \nabla_x \log p(x)$ . To address integration challenges with complex symbolic scores, the authors propose modeling the energy function $H_\theta(x)$ directly via a neural network and obtaining the score via automatic differentiation ( $s_\theta(x) = -\nabla_x H_\theta(x)$ ). Denoising score matching is employed to improve estimation in low-density regions.
Data Augmentation: To ensure robust symbolic estimation, the framework augments the original high-density samples with noisy perturbations ( $x' = x + \epsilon$ ). This ensures the SR model sees samples from both high and low-density regions, which is critical for stable estimation.
Symbolic Regression (SR) Stage: The outputs from the generative model (log-likelihoods or score fields) serve as labels for an SR algorithm. The SR task is to find a concise closed-form expression $\hat{H}(x)$ that approximates these labels. The framework employs two SR backends:
- PySR: A genetic programming-based algorithm.
- EQL (Equation Learner): A deep neural network-based approach with sparse regularization.
Key Inductive Biases: To make the problem tractable for high-dimensional systems (e.g., lattice models), the framework incorporates three specific strategies:
- Local Factorization: Leveraging the spatial locality of physical systems, the global Hamiltonian is factorized into local patches (e.g., $2\times2$ or $3\times3$ ). SR estimates the local energy function $\hat{h}(x_p)$ over a small number of variables, which is then summed to reconstruct the global energy.
- Task-Specific Primitives: The search space is constrained by including domain-specific primitive functions (e.g., trigonometric functions for the XY model, polynomials for $\phi^4$ theory).
- Multi-scale Perturbation: As noted, adding noise to samples improves coverage of the state space.

Key Contributions

Framework Integration: The paper introduces a modular framework that bridges deep generative models and SR, allowing for the discovery of interpretable energy functions from unnormalized distributions without assuming a parametric form a priori.
Scalability via Factorization: By employing local factorization of the Hamiltonian, the method effectively reduces the dimensionality of the symbolic search space, enabling the estimation of energy functions for high-dimensional lattice systems (up to $d=64$ and beyond in renormalization contexts) where standard SR fails.
Handling Non-Perturbative Regimes: The framework successfully estimates effective Hamiltonians for strongly coupled systems (specifically the renormalization of $\phi^4$ theory) where traditional perturbative or analytic derivations are challenging or intractable.
Robustness to Misspecification: Experiments demonstrate that even when the primitive library is misspecified (e.g., removing trigonometric operators for the XY model), the framework can still recover meaningful polynomial approximations of the underlying interactions.

Experimental Results
The framework was evaluated on several datasets ranging from low-dimensional toy distributions to high-dimensional lattice physics models:

Toy Distributions: For multivariate Gaussians and many-well distributions, the framework achieved near-perfect recovery of the true Hamiltonian ( $R^2 \approx 1.0$ , MSE $\approx 0$ ) across dimensions $d=2$ to $d=64$ . PySR generally required fewer samples ( $10^3$ ) than EQL ( $10^5$ ) to achieve comparable results.
XY Model: On an $8\times8$ lattice ( $d=64$ ), the framework successfully recovered the cosine-based interaction terms. Flow-based models combined with PySR outperformed score-based methods and EQL, achieving high precision and recall with no spurious terms.
Scalar $\phi^4$ Theory: The method accurately estimated the local Hamiltonian for both weakly and strongly coupled regimes. Notably, PySR consistently identified the correct nearest-neighbor interactions even when larger patch sizes (introducing more variables) were used, effectively suppressing irrelevant long-range dependencies.
Renormalization: In a non-perturbative setting, the framework estimated effective Hamiltonians for coarse-grained lattices ( $32\times32$ down to $8\times8$ ). The estimated Hamiltonians reproduced physical observables (magnetization, susceptibility) with high fidelity (histogram overlap $>98\%$ ), validating the method's ability to capture coarse-grained dynamics without analytical derivation.
Comparison: The proposed method significantly outperformed the baseline MeSSY, which struggled to converge in dimensions $d>2$ and often produced expressions with spurious terms.

Significance and Claims
The authors position this work not as a novel SR algorithm, but as a practical framework for scientific discovery. The primary significance lies in the ability to automate the derivation of interpretable energy functions for complex, high-dimensional systems directly from data. The paper claims that this approach offers a viable alternative to labor-intensive, expert-driven model formulation, particularly in non-perturbative regimes where analytic solutions are unavailable.

The authors modestly note that the framework relies on physically motivated inductive biases (patch size, primitive libraries) and that the estimated expressions are "compact and interpretable symbolic approximations" rather than exact representations of the underlying dynamics. They acknowledge that scalability is currently constrained by the SR component's ability to handle very large numbers of variables, limiting the discovery of long-range or non-local interactions in systems requiring very large patches. Future work is suggested to address mode-covering behaviors in generative models and to extend the framework to non-equilibrium systems.

More like this