TenExp: Mixture-of-Experts-Based Tensor Decomposition Structure Search Framework

The paper proposes TenExp, a novel unsupervised framework that leverages a mixture-of-experts approach to dynamically search for and activate optimal single or mixed tensor decompositions, thereby overcoming the limitations of existing methods confined to fixed factor-interaction families.

The Gist

Imagine you have a massive, messy pile of Lego bricks. These bricks represent your data (like a 3D video, a medical scan, or a colorful image). Your goal is to figure out the simplest, most efficient way to rebuild this structure so you can store it, compress it, or fix missing pieces.

In the world of data science, this process is called Tensor Decomposition. Think of it as trying to find the "blueprint" that explains how the data is built.

The Problem: The "One-Size-Fits-None" Dilemma

For a long time, scientists had a few specific blueprints they could use:

• The "Stack" method: Good for some things, bad for others.
• The "Chain" method: Great for connecting things in a line, but fails if the data is a web.
• The "Slice" method: Works well for flat layers, but struggles with 3D depth.

The big problem was that researchers had to guess which blueprint to use before they started. If they guessed wrong, the reconstruction was messy, or they couldn't compress the data well. It was like trying to fix a car engine using only a hammer, even though you might need a wrench or a screwdriver.

Furthermore, existing methods were stuck in a rut. They could only pick one blueprint. But what if your data is a hybrid? What if part of it looks like a stack and another part looks like a chain? Previous tools couldn't mix and match.

The Solution: TenExp (The "Master Chef" of Data)

The authors of this paper created TenExp. Think of TenExp as a Master Chef in a kitchen with a huge pantry of different cooking techniques (decompositions).

Instead of forcing you to pick one technique before you start cooking, TenExp works like a Mixture of Experts:

1. The Pantry (Candidate Search Set): TenExp has access to all the major blueprints (Stacks, Chains, Slices, and even some fancy new ones). It doesn't limit itself to just one style.
2. The Tasting Spoon (Rank Estimation): Before cooking, TenExp takes a tiny sample of your data (the "ingredients") and tastes it. It asks: "Does this data feel more like a stack? Or a chain? Or a mix?" It estimates how complex the data is without needing a pre-made recipe.
3. The Smart Gatekeeper (Gating Mechanism): This is the magic part. TenExp has a smart manager (the "Gatekeeper") who decides which chefs to call in.
   • Scenario A (Single Expert): If the data is simple, the Gatekeeper says, "Just use the 'Stack' chef."
   • Scenario B (The Mixture): If the data is complex, the Gatekeeper says, "We need the 'Stack' chef for the background, the 'Chain' chef for the edges, and the 'Slice' chef for the colors." It mixes them all together dynamically.

Why is this a Big Deal?

The paper highlights two superpowers of TenExp:

• It breaks the rules: Old methods were stuck in a "family" of techniques (like only using chains). TenExp can jump between different families to find the perfect fit.
• It's a team player: It doesn't just pick one winner; it can create a team of decompositions working together. This is like realizing that a complex puzzle needs a mix of straight-edge pieces and curved pieces to fit perfectly.

The Results: Fixing the Broken Data

The authors tested TenExp on real-world problems, like:

• Multispectral Images: Fixing blurry or missing parts of medical or satellite images.
• Videos: Reconstructing a video where 90% of the frames are missing (like a glitchy stream).
• Light Fields: Rebuilding 3D light data that is incredibly hard to capture.

The Analogy of the Result:
Imagine you have a shattered vase.

• Old methods tried to glue it back together using only one type of glue. Sometimes it worked, but often the vase was still cracked or looked weird.
• TenExp analyzed the cracks, realized some needed super-strong epoxy, others needed flexible putty, and some needed a specific type of tape. It mixed these materials perfectly. The result? A vase that looks brand new, with no visible cracks, and it's much lighter to carry (compressed).

In a Nutshell

TenExp is a smart, unsupervised system that doesn't force data into a box. Instead, it looks at the data, figures out exactly what kind of "mathematical recipe" it needs, and dynamically mixes different recipes together to get the best possible result. It's the difference between having a Swiss Army knife and having a whole toolbox where the right tool automatically appears when you need it.

Technical Summary

1. Problem Statement

Tensor decomposition is a fundamental tool for analyzing multi-dimensional data (e.g., images, videos, light fields) by capturing intrinsic low-rank structures. However, a critical challenge remains: selecting the most suitable tensor decomposition structure for a specific dataset.

• Limitations of Current Methods: Existing structure search methods are typically confined to a fixed factor-interaction family (e.g., only tensor contraction-based methods like TT or TR). They rely on "sampling-evaluation" frameworks that are computationally expensive and heuristic.
• Lack of Flexibility: Current approaches generally search for a single decomposition structure. They cannot adaptively combine different types of decompositions (e.g., mixing Tucker and T-SVD) to better capture complex data correlations.
• The Gap: There is no unified framework that can dynamically select, activate, and potentially mix different tensor decomposition families (Outer Product, Mode-n Product, Tensor Contraction, and T-Product) in an unsupervised manner.

2. Methodology: TenExp Framework

The authors propose TenExp, a Mixture-of-Experts (MoE) based framework designed to search for optimal tensor decomposition structures without supervision. The framework consists of three core components:

A. Cross-Factor-Interaction Candidate Search Set

TenExp systematically reviews and unifies various tensor decompositions into a general candidate set based on their factor interactions:

1. Mode-n Product Family: Includes CP and Tucker decompositions.
2. Tensor Contraction Family: Includes Tensor Train (TT), Tensor Ring (TR), and Fully-Connected Tensor Network (FCTN).
3. T-Product Family: Includes T-SVD induced Tensor Factorization (TF).

• Theoretical Insight: The paper establishes that CP is a special case of Tucker, and TT/TR are special cases of FCTN. This allows TenExp to treat Tucker, FCTN, and TF as the primary candidates for 3rd-order tensors, and Tucker/FCTN for higher-order tensors.

B. Cross-Factor-Interaction Energy-Based Rank Estimation

Since different decompositions define "rank" differently, TenExp introduces a unified energy-based rank estimation scheme:

• It calculates the accumulation energy ratio of top-$k$ singular values ($ \sum \sigma_i^2 / \sum \sigma_j^2 $).
• A threshold $\tau$ determines the boundary between "dominant components" (signal) and "secondary components" (noise/redundancy).
• This allows the framework to automatically estimate the appropriate rank for any candidate decomposition type without manual tuning.

C. Top-$k$ Gating Mechanism

This is the core innovation enabling the "Mixture of Experts" capability:

• Learnable Gating: The model uses learnable parameters to generate gating values ($g_{\theta_i}$) for each candidate decomposition.
• Top-$k$ Selection: A top-k function selects the $k$ candidates with the highest gating values.
• Softmax Normalization: The selected candidates are normalized via Softmax to produce weights ($\lambda_{\theta_i}$).
• Dynamic Output:
  • If $k=1$: The framework outputs a single, optimal decomposition.
  • If $k>1$: The framework outputs a mixture of decompositions, where the final reconstruction is a weighted sum of multiple decomposition types ($\hat{X} = \sum \lambda_{\theta_i} E_{\phi_i}$).

D. Unsupervised Training

TenExp is trained in a fully unsupervised fashion using observational data (even with missing entries). It optimizes the factor parameters ($\phi$) and gating parameters ($\theta$) simultaneously using the Adam optimizer to minimize the reconstruction error on observed data points.

3. Key Contributions

1. Unified Search Framework: TenExp is the first framework to search across multiple factor-interaction families (Mode-n, Contraction, T-Product) simultaneously, breaking the constraint of fixed families.
2. Mixture of Decompositions: It introduces the capability to deliver a mixture of decompositions (not just a single one), allowing the model to leverage the strengths of different structural priors for complex data.
3. Theoretical Guarantees: The authors derive an approximation error bound for TenExp, proving that the framework can approximate arbitrary tensors within a guaranteed error range based on the singular values of the underlying data.
4. Unsupervised Adaptability: The method requires no pre-constructed datasets and handles tensors of varying orders and sizes automatically.

4. Experimental Results

Extensive experiments were conducted on synthetic data and real-world datasets (Multispectral Images, Color Videos, Light Field Data).

• Metrics: Performance was measured using Compression Rate (CR), Relative Error (RE), PSNR, and SSIM.
• Comparisons: TenExp was compared against state-of-the-art methods including Tucker, TF, TNGreedy, SVDinsTN, HaLRTC, HTNN, and LTNN.
• Key Findings:
  • Single Decomposition: When $k=1$, TenExp dynamically selects the best single structure (e.g., T-Product for some MSIs, Tensor Contraction for others), outperforming methods fixed to one family.
  • Mixture Performance: When $k>1$, TenExp significantly outperforms all baselines. For example, in Light Field data completion at a 10% sampling rate (SR=0.1), TenExp achieved a PSNR of 46.14 dB compared to the next best (LTNN) at 44.46 dB.
  • Visual Quality: TenExp preserved fine textures and structural details better than competitors, which often suffered from blurring or artifacts.
  • Ablation Studies:
    • Search Capability: TenExp consistently found better structures than SVDinsTN (which is limited to tensor contraction).
    • Candidate Selection: Max sampling (selecting the highest gating value) outperformed random sampling.
    • Update Order: Alternating updates between decompositions and gating values (Order II) yielded the best results.

5. Significance

• Paradigm Shift: TenExp moves tensor decomposition from a "model selection" problem (choosing one fixed model) to a "model synthesis" problem (dynamically constructing the best model).
• Robustness: By allowing a mixture of experts, the framework is highly robust to diverse data types that may not fit neatly into a single low-rank definition (e.g., data with both spatial and spectral correlations that require different factor interactions).
• Practical Impact: The unsupervised nature and automatic rank estimation make TenExp highly applicable to real-world scenarios where ground truth is unavailable and data characteristics are unknown, such as medical imaging, remote sensing, and video restoration.

In conclusion, TenExp provides a flexible, theoretically grounded, and empirically superior framework for multi-dimensional data recovery by unifying diverse tensor decomposition strategies under a Mixture-of-Experts architecture.