🤖 machine learning

Probabilistic Block Term Decomposition for the Modelling of Higher-Order Arrays

This paper proposes an efficient variational Bayesian Block-Term Decomposition (pBTD) that utilizes the von-Mises Fisher matrix distribution to enforce orthogonality, demonstrating its effectiveness in robustly inferring patterns and quantifying model order for noisy higher-order tensor data.

Original authors: Jesper Løve Hinrich, Morten Mørup

Published 2026-08-10

📖 6 min read🧠 Deep dive

CC0 1.0

Original authors: Jesper Løve Hinrich, Morten Mørup

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but instead of a single crime scene, you have a massive, multi-layered library of clues. Some clues are just simple lists (like a grocery list), some are spreadsheets (like a budget), but the most interesting ones are 3D cubes or even higher-dimensional "hyper-cubes" of data. In the world of science, these are called tensors. You might find them everywhere: in chemistry, where they track how chemicals glow under different lights; in biology, where they map how genes change over time and across different conditions; or even in psychology, tracking how people answer questions about different objects.

To make sense of these giant, messy data cubes, scientists use a technique called tensor decomposition. Think of it like taking apart a complex Lego castle to see the individual bricks and how they were snapped together. The goal is to break the data down into simpler, understandable patterns. For a long time, scientists have used two main ways to do this: one that treats every pattern as a completely separate, independent line (like a stack of single Lego bricks), and another that treats the patterns as a big, interconnected web where everything touches everything else. But what if the truth is somewhere in the middle? What if your data is made of several distinct "blocks," where each block is a small, interconnected web, but the blocks themselves don't talk to each other? This is the "Block-Term Decomposition" (BTD), a method that tries to find that perfect middle ground.

However, there's a catch. Traditional ways of solving these puzzles rely on finding just one "best guess" answer. If the data is noisy or messy (like a photo taken in the dark), that single guess can easily get tricked, leading to a wrong conclusion. This is where the new paper comes in. Instead of just guessing one answer, the authors propose a smarter, "probabilistic" approach. Imagine instead of asking, "What is the one true shape of this Lego castle?" you ask, "What are all the possible shapes this castle could be, and how likely is each one?" By using a method called Bayesian inference, they don't just find a single answer; they map out a landscape of possibilities, allowing them to see how uncertain they are and to automatically ignore the parts of the data that are just random noise.

The Paper's New Tool: The "Smart" Lego Builder

In this paper, Jesper Løve Hinrich and Morten Mørup introduce a new, highly efficient version of this probabilistic method specifically for the Block-Term Decomposition (BTD). They call it pBTD (probabilistic Block Term Decomposition). Their big idea is to build a model that can handle the messy reality of real-world data by treating the unknown parts of the puzzle as distributions (ranges of possibilities) rather than fixed numbers.

To make this work, they had to solve a tricky math problem: how to keep the different "blocks" of the data distinct and not let them blur into each other. They did this by using a special mathematical rule (the von Mises-Fisher distribution) that forces the building blocks to stay "orthogonal." In everyday terms, think of orthogonality as making sure your Lego bricks are perfectly perpendicular to each other, like the corner of a room. This keeps the different patterns from getting tangled up, which is a common problem in other methods.

What They Found: Smarter Guessing and Noise Filtering

The authors tested their new pBTD tool in two ways: first, with computer-generated "fake" data where they knew the exact answer, and second, with two real-world datasets: one from an industrial chemical process and another from brainwave (EEG) recordings.

When they tested the tool on the fake data, they found something fascinating. When the data was very noisy (like trying to hear a whisper in a hurricane), the traditional methods (called Maximum Likelihood Estimation or MLE) kept trying to fit the noise into their patterns, essentially "overfitting" and creating a messy, wrong model. In contrast, the new pBTD tool was smart enough to realize, "Hey, this part is just noise," and it effectively turned off the parts of the model that didn't make sense. It didn't just find the answer; it knew when not to find an answer.

They also used the tool to figure out the "right" structure of the data. Imagine you have a box of Legos and you don't know if you should build one big tower, a few small towers, or a flat wall. The pBTD tool uses a score called the Evidence Lower Bound (ELBO) to act like a judge, telling them which structure fits the data best. In their simulations, the tool successfully identified the correct number of blocks and their sizes in most cases, even when they started with a model that had too many pieces. It was able to "prune" the extra, unnecessary pieces, shrinking them down to near-zero size, much like a gardener trimming away dead branches to reveal the healthy tree.

Real-World Results: From Chemicals to Brains

When they applied pBTD to real data, the results were equally promising.

The Chemical Data: In a dataset from an industrial injection molding process, the tool suggested that a full "Tucker" model (where everything is interconnected) was the best fit. However, it also showed that many of the connections in that model were very weak or uncertain, effectively pruning them away to show the core structure.
The Brain Data: In the EEG dataset, which measured brain activity during hand stimulation, the tool again favored the full interconnected model. But here, the authors noted a trade-off. While the full model was statistically the "best" fit, the simpler, more separated models (like the CPD model) were actually easier for humans to understand. For instance, the simpler model clearly showed that one specific pattern of brain activity happened when the left hand was stimulated and a different one for the right hand. The complex, fully connected model was harder to interpret, even though it was mathematically robust.

The Bottom Line

The paper concludes that this new probabilistic approach offers a unified way to handle all types of tensor decompositions, from the simplest to the most complex. It provides a "safety net" against overfitting, allowing scientists to be more confident in their results even when the data is noisy. While the math behind it is heavy, the result is a tool that doesn't just crunch numbers; it understands the difference between a signal and a glitch. The authors suggest that this method is just as fast as the old ways of doing things but gives you a much richer, more reliable picture of what your data is actually telling you. They also note that while they used a specific math trick called "variational inference" to make it fast, the framework is flexible enough to be used with other, even more detailed methods in the future. Ultimately, they have given scientists a better way to look at the multi-dimensional world, helping them see the patterns without getting lost in the noise.

Technical Summary: Probabilistic Block Term Decomposition for the Modeling of Higher-Order Arrays

Problem Statement
Tensor decompositions are essential for characterizing higher-order structures in scientific data, with Canonical Polyadic Decomposition (CPD) and Tucker decomposition being the most prevalent. The Block-Term Decomposition (BTD) serves as a structured intermediate framework, aggregating multiple small Tucker models to bridge the gap between the rank-1 outer products of CPD and the fully connected multi-linear structures of Tucker. Traditionally, these methods rely on Maximum Likelihood Estimation (MLE), which provides point estimates but lacks mechanisms for uncertainty quantification, robustness to noise, and principled model order selection. While Bayesian approaches have been developed for CPD and Tucker, existing probabilistic BTD models (e.g., [8]) often fail to preserve critical structural properties, such as the orthogonality of factor matrices central to standard MLE-based BTD, or are restricted to specific block configurations (e.g., $(L_r, L_r, 1)$ ). Furthermore, determining the optimal number of cores and their sizes in BTD remains a challenge, typically addressed via heuristic criteria.

Methodology
The authors propose an efficient Variational Bayesian Probabilistic Block Term Decomposition (pBTD). The core of the methodology involves:

Generative Model: The data is modeled as a sum of $T$ Tucker-like blocks. The factor matrices $U^{(n)}_t$ for each block are constrained to be orthogonal, drawn from a von Mises-Fisher (vMF) matrix distribution with a uniform prior on the Stiefel manifold. The core arrays $G_t$ are modeled as zero-mean normal distributions with a precision matrix $\Psi_t$ .
Orthogonality Enforcement: Unlike previous probabilistic BTD attempts that used Gaussian factor distributions, this model explicitly imposes orthogonality via the vMF distribution, aligning with the properties of conventional BTD and ensuring the decoupling of core elements.
Automatic Relevance Determination (ARD): To address model order selection and prevent overfitting, the authors employ ARD priors on the core arrays. The precision matrix $\Psi$ is structured to allow for the pruning of irrelevant slices or elements within the cores. Three prior types are considered: scale, sparsity, and ARD, with the implementation focusing on the sparsity prior.
Variational Inference: Due to the intractability of the exact posterior, a mean-field variational approximation is used. The posterior is approximated by a set of independent distributions $Q(\theta)$ $Q (θ)$ . The optimization maximizes the Evidence Lower Bound (ELBO).
- Factor Matrices: The variational distribution for the factor matrices remains a vMF distribution, updated using a concentration matrix derived from the residual data and expected core values.
- Core Arrays: The variational distribution for the vectorized core arrays follows a multivariate normal distribution. The orthogonality of the factor matrices simplifies the covariance calculation, resulting in independent univariate distributions for core elements.
- Hyperparameters: The noise precision and the precision parameters of the core priors are updated using Gamma distributions derived from the variational updates.

Key Contributions

Unified Framework: The paper provides a unified Bayesian framework that naturally encompasses CPD (as a special case with rank-1 blocks) and Tucker (as a single block) within the BTD structure.
Orthogonal Probabilistic BTD: It introduces a probabilistic BTD model that successfully incorporates the orthogonality constraints of factor matrices via the vMF distribution, a feature missing in prior probabilistic BTD attempts.
Model Order Quantification: The framework utilizes the ELBO and ARD priors to automatically prune irrelevant core elements and determine suitable multi-linear structures, offering a principled alternative to heuristic model selection.
Efficiency: The proposed variational inference scheme achieves computational complexity similar to conventional MLE procedures, making it feasible for practical application despite the added benefits of Bayesian inference.

Results
The authors evaluated the pBTD on synthetic data and two real-world datasets:

Synthetic Data:
- Robustness: Compared to MLE-based BTD methods (BTD-NLS and BTD-minf), pBTD demonstrated superior robustness to noise. At high noise levels, pBTD effectively "turned off" modeling to avoid overfitting, whereas MLE methods overfit the noise.
- Model Selection: When fitting various BTD structures to data generated from a known ground truth, pBTD successfully identified the correct structure via the ELBO in most cases. The ARD prior allowed the model to prune elements in over-specified models, mitigating model misspecification.
- Performance: pBTD achieved lower reconstruction errors than MLE methods on incorrect model orders and was significantly faster than BTD-NLS (10-15x), though slightly slower than BTD-minf.
Real Data:
- Flow Injection Data: The model supported a full Tucker representation, with ARD pruning many core elements to near zero. The inferred Tucker core showed components with higher uncertainty compared to more constrained CPD/BTD representations.
- EEG Data: The ELBO favored the full Tucker model, indicating a preference for full multi-linear representation. However, the CPD representation yielded more interpretable results, specifically distinguishing between left and right hand stimulation in the first and eleventh components, a clarity lost in the more complex block interactions of the Tucker and intermediate BTD models.

Significance and Claims
The paper claims that the proposed pBTD offers a robust, principled approach to tensor decomposition that extends the benefits of Bayesian inference—specifically uncertainty quantification, noise modeling, and automatic model order selection—to the BTD framework. The authors emphasize that this is achieved without sacrificing computational efficiency, maintaining complexity comparable to standard MLE.

The significance lies in the ability to:

Quantify uncertainty in decomposition parameters, moving beyond point estimates.
Perform automatic relevance determination to prune core elements, aiding in the selection of the appropriate model complexity (CPD vs. Tucker vs. intermediate BTD).
Provide a unified probabilistic treatment for CPD, Tucker, and BTD, facilitating a more comprehensive analysis of multi-linear data structures.

The authors modestly note that while the current implementation uses variational inference, the framework is generalizable to MCMC methods (e.g., Gibbs sampling). They also highlight that while the full Tucker model often maximizes the ELBO on real data, the interpretability of simpler structures like CPD may sometimes be superior, suggesting that model selection should balance statistical evidence with domain-specific interpretability.

Technical Summary: Probabilistic Block Term Decomposition for the Modeling of Higher-Order Arrays

More like this