🤖 machine learning

A Compositional Theory of Curvature in Probabilistic Circuits

This paper demonstrates that global sharpness regularization is suboptimal for Probabilistic Circuits due to their compositional curvature structure, and proposes an adaptive, locally-targeted regularizer that preserves closed-form EM updates while recovering generalization performance.

Original authors: Hrithik Suresh, Sahil Sidheekh, Shelar Parth Vijay, Yasir Z, Sriraam Natarajan, Narayanan Chatapuram Krishnan

Published 2026-08-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Hrithik Suresh, Sahil Sidheekh, Shelar Parth Vijay, Yasir Z, Sriraam Natarajan, Narayanan Chatapuram Krishnan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Shape of Learning: Why Flat isn't Always Better

Imagine you are trying to teach a computer to understand the world, like recognizing a cat in a photo or predicting the weather. In the world of artificial intelligence, there is a special family of models called Probabilistic Circuits. Think of these not as the giant, messy neural networks that power today's chatbots, but as highly organized, logical flowcharts. They are built with strict rules that allow them to do something magical: they can calculate the exact probability of an event happening without needing to guess or approximate. This makes them incredibly reliable for tasks where being wrong isn't an option, like medical diagnosis or autonomous driving.

However, just like a student cramming for a test, these circuits can sometimes "overfit." This happens when the model memorizes the training data too perfectly, including all the random noise and quirks, and fails to understand the general rules. To fix this, scientists often look at the "shape" of the learning process. They want the model to settle into a "flat" valley in the landscape of possibilities, because flat spots are usually more stable and generalize better to new data. The standard tool for this is a "global sharpness" check, which measures how bumpy the entire landscape is and tries to smooth everything out equally. But what if smoothing everything out actually makes the model worse? What if the problem isn't the whole landscape, but just a few specific, jagged rocks hidden inside it? This is the puzzle a team of researchers set out to solve.

The Puzzle of the "Flat" Mistake

The researchers, led by Hrithik Suresh and Sahil Sidheekh, discovered that the standard way of smoothing out Probabilistic Circuits was actually missing the mark. They found that treating the entire model as one big, uniform object to be flattened was a mistake. In fact, when they tried to flatten the whole thing, the models often started to "underfit," meaning they became too simple and stopped learning the data properly, even though they were technically sitting in a flatter spot.

To understand why, the team looked inside the circuit and found that the "sharpness" (or bumpiness) of the model isn't a single, global thing. Instead, it is compositional, meaning it is built from two very different ingredients mixed together. They proved mathematically that the contribution of any single part of the circuit to the overall bumpiness is the product of two factors:

Contextual Usage: How much traffic flows through that part of the circuit. Imagine a busy highway intersection; even if the road is smooth, if millions of cars pass through it every second, it feels like a major part of the traffic system.
Local Curvature: How bumpy that specific road is on its own, regardless of traffic.

The authors showed that the standard "global" method was like a city planner who sees a traffic jam and decides to pave over the entire city to make it smoother. But in reality, the jam was caused by a few specific, pothole-ridden side streets that just happened to be on the main route. By trying to smooth the whole city, the planner ruined the perfectly good, smooth main roads, making the whole system less efficient.

The "Gatekeeper" Solution

The paper argues that the global method fails because it confuses "busy" with "bumpy." A part of the circuit might contribute a lot to the total sharpness just because it is used constantly (high traffic), not because it is actually jagged. Conversely, a part of the circuit might be incredibly bumpy (a deep pothole) but sit on a quiet side street with no traffic, so the global method ignores it.

To fix this, the researchers proposed a new, smarter way to smooth the model. Instead of applying a uniform "flatness" penalty to every part of the circuit, they introduced an adaptive regularizer. Think of this as a smart gatekeeper. Before the model tries to smooth out a specific part of the circuit, the gatekeeper checks: "Is this part actually bumpy on its own?"

If the part is intrinsically bumpy (high local curvature), the gatekeeper opens wide and applies a strong smoothing penalty.
If the part is just busy but already smooth, the gatekeeper lets it be, preserving its ability to learn complex patterns.

This approach allows the model to keep its "muscle" (its ability to fit the data) while only softening the specific spots that are actually causing trouble.

What They Found

The team tested this idea on 20 different datasets, ranging from simple binary data to more complex real-world scenarios like traffic accidents and audio files. Their results were clear:

The Global Method Fails: When they used the old, uniform smoothing method, the models often got worse. They became flatter, yes, but they also became less accurate, failing to capture the nuances of the data. In many cases, the models ended up "underfitting," essentially becoming too simple to learn the details.
The Adaptive Method Wins: When they used their new "gatekeeper" approach, the models maintained their accuracy. They managed to reduce the dangerous sharpness without sacrificing the ability to fit the data. In fact, on several datasets, the new method performed better than both the unregularized model and the old global method.

The researchers also visualized the data to show exactly why the old method failed. They found that a tiny fraction of the circuit nodes (less than 10%) were responsible for almost all of the "global sharpness" signal. However, when they tried to fix only those top contributors, the model got even worse. This proved that the "busiest" nodes were not necessarily the "bumpiest" ones. The global signal was being hijacked by the traffic flow, not the actual geometry of the bumps.

The Takeaway

This paper doesn't just offer a new trick; it offers a new way of thinking about how these models learn. It shows that in Probabilistic Circuits, you cannot treat the whole system as a single block. You have to understand that the "sharpness" you see from the outside is a mix of how much a part is used and how bumpy it actually is. By separating these two factors, the authors created a method that knows exactly where to apply pressure and where to let the model breathe.

The work suggests that the future of making these models robust isn't about making everything flatter, but about being precise about where the bumps are. It's a shift from a "one-size-fits-all" approach to a tailored, surgical fix. While the paper focuses on these specific circuits, the idea that "busy" doesn't always mean "broken" could be a valuable lesson for understanding other complex learning systems, too. The authors conclude that this decomposition opens the door to smarter ways of designing, compressing, and protecting these intelligent systems, ensuring they remain both sharp and reliable.

Technical Summary: A Compositional Theory of Curvature in Probabilistic Circuits

Problem Statement

Probabilistic Circuits (PCs) are generative models that support exact inference and offer a tractable measure of loss-surface curvature: the trace of the Hessian of the negative log-likelihood (NLL). Recent work has utilized this trace as a global regularizer to bias learning toward flatter optima, aiming to improve generalization in low-data regimes. However, this paper identifies a critical flaw in treating sharpness as a single global measure. Empirical observations reveal that global trace regularization often leads to underfitting in high-data regimes, lowering both training and test log-likelihoods. Furthermore, the distribution of curvature is highly non-uniform; a small fraction of sum nodes accounts for nearly all global sharpness. Crucially, restricting regularization to these high-contributing nodes based on global trace further degrades performance, suggesting that global contribution is a poor signal for allocating regularization.

Methodology

The authors propose a compositional theory of sharpness that decomposes a sum node's contribution to the global Hessian trace into two distinct factors.

Exact Decomposition:
The paper proves that the global trace contribution $T_n(x)$ of a sum node $n$ factorizes exactly as:
$T_n(x) = F_n(x)^2 \cdot t_n(x)$
- Contextual Usage ( $F_n(x)^2$ ): This term represents the squared circuit flow through node $n$ , measuring how heavily the node is utilized in explaining the input $x$ . It is derived from the standard circuit-flow recursion.
- Local Curvature ( $t_n(x)$ ): This term is a local measure determined solely by the node's output distribution. The authors prove that the local Hessian of a sum node is rank-one, and its unique non-zero eigenvalue is exactly $t_n(x)$ . Thus, $t_n(x)$ fully captures the node's intrinsic second-order geometry.
Theoretical Insights:
- Depth Bias: The decomposition reveals a systematic depth bias in global trace regularization. Because circuit flow is attenuated as it passes through sum nodes, shallow, high-flow nodes accumulate larger $F_n(x)$ values. Consequently, global trace contributions ( $T_n$ ) are biased toward shallow nodes, regardless of their intrinsic local curvature ( $t_n$ ).
- Ranking Reversal: Ranking nodes by global contribution ( $T_n$ ) does not necessarily identify nodes with high intrinsic local curvature ( $t_n$ ). A node may dominate the global trace simply because it lies on a high-flow path, even if its local mixture is flat. Conversely, a locally sharp node deep in the circuit may contribute negligibly to the global trace due to flow attenuation.
Proposed Solution: Adaptive Sharpness-Aware Regularization:
To address the misallocation of regularization, the authors introduce an adaptive regularizer that uses local curvature to gate the global trace penalty.
- Gating Mechanism: Instead of applying a uniform penalty strength $\mu$ to all nodes, the method assigns a gate $\omega_n$ to each sum node based on its empirical local trace $\hat{t}_n$ . The gate is defined as $\omega_n = \hat{t}_n / \max_m \hat{t}_m$ , ensuring nodes with higher intrinsic curvature receive stronger regularization.
- Tractable Updates: The authors derive a closed-form Expectation-Maximization (EM) update for the sum-node weights that incorporates these gates. The update preserves the linear-time complexity and closed-form nature of the original global trace regularizer, merely scaling the effective regularization strength per node ( $\mu \to \mu \omega_n$ ).

Key Results

The paper validates its theory and method across 20 binary density-estimation benchmarks (DEBD) using Hidden Chow-Liu Trees (HCLTs):

Anatomy of Global Trace: Experiments confirm that global trace contributions are highly concentrated in early circuit partitions (shallow nodes), whereas local curvature is distributed much more broadly across the circuit. This confirms that the concentration of global sharpness is driven primarily by contextual usage rather than intrinsic geometry.
Selection Criteria: When regularization is restricted to a subset of nodes, selecting based on local curvature preserves model fit significantly better than selecting based on global contribution. Restricting by global contribution leads to performance degradation, while restricting by local curvature maintains or slightly improves upon the unregularized baseline.
Underfitting and Recovery: Global trace regularization frequently causes underfitting, degrading performance on 18 out of 20 datasets compared to the unregularized baseline. In contrast, the proposed gated method mitigates this underfitting, matching or improving upon the unregularized model on the majority of datasets while retaining the robustness benefits of sharpness-aware learning.

Significance and Claims

The paper claims that treating sharpness as a global, uniform property is misspecified for Probabilistic Circuits due to their compositional nature. The primary contribution is the theoretical demonstration that global curvature is a product of local geometry and contextual usage.

The significance of this work lies in:

Resolving the Underfitting Puzzle: It explains why global sharpness regularization fails in high-data regimes: it penalizes high-usage components (which may be flat) rather than intrinsically sharp components, thereby suppressing model capacity unnecessarily.
Algorithmic Improvement: It provides a theoretically grounded, computationally efficient method to regularize PCs that targets intrinsic curvature rather than structural usage.
Future Directions: The decomposition opens avenues for curvature-aware model compression, targeted robustness interventions, and adaptive circuit design, where local sensitivity and contextual influence can jointly guide structural modifications.

The authors maintain a modest stance, noting that while the current gating function is a proof of concept, the framework supports richer gate functions and does not claim to solve all generalization issues in PCs, but rather corrects a specific misallocation of regularization inherent in global trace methods.