🤖 machine learning

Dual-Attention Convolution Experts for Sparse Tensor Completion

This paper proposes DCGC, a novel neural tensor factorization method that combines a multi-channel convolution network with a gated dual-attention mechanism and group-level contrastive learning to effectively capture complex cross-mode interactions and mitigate extreme data sparsity in tensor completion tasks.

Original authors: Yanlei Liu, Zhenyu Liao

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yanlei Liu, Zhenyu Liao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to finish a giant, multi-dimensional puzzle, but most of the pieces are missing. This is the problem of Tensor Completion. In the real world, this happens all the time:

Recommendation Systems: You have a list of users, movies, and times, but most users haven't rated most movies. The "puzzle" of what you would like is full of holes.
Traffic Sensors: You have sensors on roads, but some are broken or offline, leaving gaps in the traffic flow data.

The paper introduces a new AI tool called DCGC (Dual-Attention Convolution Expert Networks with Group-Level Contrastive Learning) to solve this puzzle, especially when the data is extremely sparse (full of holes).

Here is how DCGC works, explained through simple analogies:

1. The Problem: The "Empty Room" and the "Overwhelmed Expert"

Traditional methods try to guess the missing pieces by looking at simple patterns. But modern data is complex and messy.

The Sparsity Issue: Imagine trying to guess a movie rating when you have almost no data about that user. It's like trying to guess the ending of a book when you've only read the first page.
The "Expert" Problem: Previous AI models tried to use a "team of experts" (neural networks) to solve this. However, often one or two "experts" would do all the work while the others sat idle. This is like a restaurant where only one chef cooks everything, leading to burnout and poor quality, while the other chefs' unique skills are wasted.

2. The Solution: DCGC's Three Superpowers

A. The "Specialized Kitchen" (Mixture of Convolution Experts)

Instead of one big brain, DCGC sets up a kitchen with many different chefs (called Convolution Experts).

Each chef specializes in a specific type of flavor or pattern.
The Innovation: Instead of letting one chef take over, DCGC uses a Dual-Attention Mechanism. Think of this as a smart manager who looks at the order and says, "For this specific dish, Chef A is the best, but we also need Chef B's special sauce."
The manager dynamically decides which chefs to listen to and which features of the data are most important. This ensures the model doesn't get stuck on just one pattern and can handle complex, non-linear relationships.

B. The "Fairness Manager" (Personalized Gating)

In many AI teams, the "loudest" experts dominate the conversation, ignoring the quiet ones who might have valuable insights for rare or difficult cases (like a user with very few ratings).

DCGC uses a Personalized Gating system. This is like a manager who actively forces the team to listen to the quiet experts when the data is sparse.
If a user has very little history, this mechanism ensures the model doesn't ignore them but instead carefully balances the input from all experts to make a fair guess. It prevents the "rich get richer" problem in AI training.

C. The "Study Group" (Group-Level Contrastive Learning)

This is the trickiest part, but also the most clever.

The Problem: When data is sparse, the AI doesn't have enough examples to learn from.
The Solution: DCGC organizes the data into "study groups" based on how strong the feedback is.
- Group 1 (The Enthusiasts): Users who rate things 5 stars.
- Group 2 (The Neutral): Users who rate things 3 stars.
- Group 3 (The Critics): Users who rate things 1 or 2 stars.
How it helps: The AI learns that "Enthusiasts" are similar to other "Enthusiasts," and "Critics" are similar to other "Critics." It pulls similar groups closer together and pushes different groups apart.
The Benefit: Even if a specific user has very few ratings (a "cold start" user), the AI can borrow knowledge from the "study group" they belong to. It's like a shy student learning from the confident students in their same grade level, rather than trying to learn from the whole school at once. This provides high-quality "self-supervised" signals to fill in the gaps.

3. The Results: Does it Work?

The authors tested DCGC on five different real-world datasets, including:

Traffic Data: Predicting traffic speeds on highways.
Movie Recommendations: Predicting ratings for movies (using datasets like MovieLens and Amazon Beauty).

The Outcome:

DCGC beat the current "state-of-the-art" methods (the best existing tools) in almost every test.
It performed particularly well when the data was very sparse (very few ratings or sensor readings).
It achieved this without needing massive amounts of computing power, thanks to its efficient design.

Summary

Think of DCGC as a highly organized, fair, and smart team of puzzle solvers.

It uses a team of specialists (Experts) instead of one generalist.
It has a smart manager (Dual-Attention) who knows exactly which specialist to call for each specific piece of the puzzle.
It has a fairness policy (Gating) to ensure the quiet specialists get a chance to speak, especially when the puzzle is hard.
It organizes solvers into study groups (Contrastive Learning) so that even those with little experience can learn from the experts in their own group.

The result is a system that can fill in the missing pieces of a giant, messy puzzle much better than previous methods, even when there are very few pieces to start with.

Technical Summary: Dual-Attention Convolution Experts for Sparse Tensor Completion

Problem Statement
Tensor factorization (TF) is a fundamental tool for recovering missing entries in high-dimensional, sparse data found in applications like recommendation systems and spatiotemporal forecasting. However, existing approaches face two primary challenges:

Modeling Complex Interactions: Traditional linear or multi-linear decomposition methods struggle to capture highly non-linear cross-mode interactions among latent factors. While neural TF methods (MLP-based or CNN-based) offer non-linear modeling capabilities, CNN-based architectures often fail to capture fine-grained interactions between latent factors and may still suffer from overfitting or rigid representation entanglement.
Data Sparsity: Severe data sparsity limits the training signals available for deep neural networks. Existing solutions often rely on auxiliary information (e.g., similarity matrices, social networks), which can introduce noise if the auxiliary data is unreliable or is tailored to specific scenarios.

Methodology: DCGC
The authors propose Dual-Attention Convolution Expert Networks with Group-Level Contrastive Learning (DCGC), a neural TF framework designed to model non-linear interactions robustly under sparse conditions. The architecture consists of three core components:

Embedding and Second-Order Interactions:
- The model maps discrete indices to latent factor matrices ( $U, P, W$ ).
- Unlike standard linear approaches, DCGC explicitly constructs second-order cross features by computing pairwise Hadamard products between latent factors (e.g., $u_i \odot p_j$ ). These interaction vectors are concatenated with the original embeddings to form a unified 2D representation ( $H_{emb}$ ).
Mixture-of-Convolution-Experts with Dual-Attention and Gating:
- Mixture-of-Experts (MoE): Inspired by MMoE, the model employs multiple convolutional experts ( $C$ ) to perform weighted non-linear alignments of the latent factors. This allows different experts to specialize in heterogeneous underlying patterns while sharing parameters to mitigate overfitting.
- Dual-Attention Mechanism: To dynamically prioritize critical information, the model computes two types of attention scores from the expert outputs:
  - Expert Attention ( $A_{exp}$ ): Weights the importance of individual convolution channels (experts).
  - Feature Attention ( $A_{fea}$ ): Weights the importance of aligned feature elements within the channels.
  - These are fused additively to modulate the value embeddings, allowing the model to focus on both critical experts and critical features.
- Personalized Gating: To address attention polarization (where a few experts dominate) and the neglect of sparsely observed factors, a personalized gating mechanism generates input-dependent routing weights. This reallocates modeling capacity toward sparse interactions, outperforming standard dropout strategies in load balancing.
Group-Level Contrastive Learning (GLCL):
- To alleviate data sparsity without relying on noisy auxiliary data, DCGC introduces a self-supervised contrastive strategy.
- Grouping: Observed triplets are categorized into distinct feedback levels (e.g., High, Medium, Low) based on their original tensor values.
- Contrastive Objective: The model aggregates positive samples within the same feedback level while separating negative samples from lower feedback levels.
- Loss Function: An InfoNCE loss is applied to the auxiliary task, encouraging the representation space to cluster samples with identical feedback levels and separate those with different levels. This provides high-quality self-supervised signals to stabilize the learning of latent factors for sparse sensors or users.

Key Contributions

Novel Architecture: The proposal of a mixture-of-convolution-experts network enhanced with a dual-attention mechanism and personalized gating, enabling fine-grained modeling of non-linear cross-mode interactions.
Self-Supervised Sparsity Mitigation: The development of an efficient in-batch Group-Level Contrastive Learning (GLCL) strategy that exploits feedback-level consistency to generate self-supervised signals, reducing reliance on unreliable auxiliary data.
Empirical Validation: Extensive experiments on five real-world datasets (traffic prediction and recommendation) demonstrating consistent superiority over state-of-the-art TF methods (e.g., CoATF, HOCTC) and competitive performance against specialized GNN-based methods.

Experimental Results
The authors evaluated DCGC on five datasets: GZspeed and PEMS (traffic), and Movielens-100k, Movielens-1M, and AmazonBeauty (recommendation).

Performance: DCGC outperformed the best baselines by 3.34%–5.06% in RMSE/MAE on traffic datasets and achieved improvements of 1.44%–3.56% (RMSE) and 1.32%–7.43% (MAE) on recommendation datasets.
Comparison: The method significantly outperformed traditional linear methods (P-Tucker) and recent neural approaches (CoSTCo, MDMTF, HOCTC). It also maintained competitive performance against GNN-based methods designed for cold-start scenarios (e.g., LightGCN, MetaHIN).
Ablation Studies:
- Removing the dual-attention mechanism or replacing it with single-attention branches resulted in performance degradation, confirming the necessity of modeling both expert and feature importance.
- Replacing the personalized gating mechanism with standard dropout led to higher error rates, particularly in highly sparse scenarios (e.g., AmazonBeauty), and caused expert load polarization.
- Replacing GLCL with vanilla contrastive learning or removing it entirely degraded performance, validating the efficacy of grouping by feedback levels.
Efficiency: The model maintains competitive performance even with a low number of convolution channels and a reduced rank, indicating that the attention mechanisms allow for efficient parameter usage.

Significance and Claims
The paper claims that DCGC addresses the dual challenges of capturing complex non-linear interactions and handling extreme data sparsity in tensor completion. By integrating a fine-grained mixture of experts with a dual-attention mechanism, the model achieves robust non-linear modeling without excessive parameter growth. Furthermore, the Group-Level Contrastive Learning strategy offers a novel way to inject self-supervised signals directly from the data distribution, mitigating sparsity without the noise associated with external auxiliary information. The authors position DCGC as a robust, general-purpose framework for sparse tensor completion in both spatiotemporal and recommendation domains.

Limitations
The authors acknowledge two limitations for future work:

The current dual-attention module uses a simple additive fusion for expert and feature scores, which may not fully capture complementary information; more adaptive fusion mechanisms are suggested.
The GLCL strategy relies on predefined feedback intervals; future work could explore adaptive interval construction based on data distribution or interaction density.