💻 computer science

TGSE-DFN: Text-guided Generative Semantic Enhancement and Disentangled Adaptive Fusion Network for Multimodal Sentiment Analysis

The paper proposes TGSE-DFN, a novel multimodal sentiment analysis framework that leverages text-guided generative semantic enhancement and disentangled adaptive fusion to bridge the semantic gap between modalities and effectively capture both global sentiment trends and fine-grained temporal dynamics.

Original authors: Kejian Dong, Zhenzhen He, Tiquan Gu, Yaling Wan, Xiaochao Fan, Bowen Gu

Published 2026-10-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kejian Dong, Zhenzhen He, Tiquan Gu, Yaling Wan, Xiaochao Fan, Bowen Gu

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Human communication is rarely just about the words we speak. A single sentence can carry a completely different meaning depending on the tone of voice used, the pause before it is spoken, or the expression on a face. To truly understand how a person feels, we must listen to the whole picture: the text, the sound, and the sight. This is the challenge of multimodal sentiment analysis, a field of computer science dedicated to teaching machines to read these complex emotional cues. While computers have become quite good at reading text, they often struggle to connect the explicit meaning of words with the subtle, continuous signals of a voice or a facial expression. The gap between what is said and how it is said often leads to misunderstandings, leaving machines unable to grasp the full depth of human emotion.

Researchers at Xinjiang Normal University and Xinjiang University have developed a new approach to bridge this gap, aiming to help computers understand the hidden emotional layers in audio and video. Their work, titled TGSE-DFN, addresses a specific problem: existing systems often treat audio and visual data as secondary to text, or they try to force these continuous signals into rigid, discrete categories that lose important details. Instead, the team created a system that uses the rich meaning of language as a guide to help the computer "see" and "hear" emotions more clearly, without losing the fine-grained details of the original signals.

The core idea behind this new system is to use text as a map for the other senses. In many videos, a person might say something neutral while their voice trembles or their face tightens, revealing true feelings that the words alone do not show. The researchers found that previous methods often failed to extract these high-level emotional meanings from audio and video because they lacked a strong semantic anchor. To fix this, they built a module that acts like a translator, guiding the raw audio and visual data into a continuous space where they can be understood alongside language. This process does not convert a voice or a face into a simple written description, which would strip away the timing and nuance of the emotion. Instead, it generates a continuous, latent representation—a kind of emotional blueprint—that preserves the flow of the original signal while infusing it with the deep semantic knowledge found in language models.

Once the audio and visual data have been enriched with this linguistic guidance, the system faces another challenge: deciding which information to trust. In any given moment, the text might be the most reliable indicator of sentiment, while in another, a sudden change in tone or a fleeting facial expression might be the key. The researchers designed a mechanism to separate the shared emotional information from the unique details of each sense. This "disentanglement" allows the computer to identify what is common across all three modes—such as a general feeling of sadness—while keeping the specific, private cues of the voice or the face separate. This prevents the different types of data from conflicting with one another and ensures that the unique strengths of each modality are preserved.

The final step involves bringing all these pieces together in a way that adapts to the specific situation. The system uses a fusion strategy that looks at both the big picture and the small details. It captures the overall sentiment trend of a video segment while simultaneously tracking fine-grained changes, such as a pause in speech or a micro-expression. Crucially, the system learns to weigh the importance of text, audio, and vision dynamically. If a sample is ambiguous in its words but clear in its tone, the system automatically places more trust in the audio. If the text is explicit but the voice is flat, it leans more heavily on the words. This adaptive weighting allows the model to make more stable and accurate predictions across a wide variety of emotional expressions.

The researchers tested their system on two large, well-known datasets containing thousands of video clips with human annotations of sentiment intensity, ranging from strongly negative to strongly positive. The results showed that their method outperformed many existing techniques, achieving higher accuracy in predicting both the direction of the sentiment and its intensity. The team conducted detailed experiments to confirm that each part of their system contributed to this success. When they removed the text-guided enhancement, the system's ability to understand audio and video dropped. When they removed the separation of shared and private information, the model became less precise. These tests confirmed that the combination of semantic guidance, representation separation, and adaptive fusion is what drives the improvement.

Despite these successes, the researchers acknowledge that the task remains difficult. Their analysis of errors revealed that the system still struggles with samples where the sentiment is mixed or where the intensity is extreme. For instance, the model sometimes underestimated the intensity of very strong negative emotions or overestimated the positivity of weakly positive statements. In cases where a speaker used sarcasm or contradictory cues, the system occasionally misjudged the overall polarity. These findings suggest that while the system has made significant progress in understanding the interplay between text, sound, and sight, the complexity of human emotion still presents challenges that require further refinement. The work stands as a significant step toward machines that can truly understand the full spectrum of human feeling, not just the words we choose to say.

Technical Summary: TGSE-DFN for Multimodal Sentiment Analysis

1. Problem Definition

Multimodal Sentiment Analysis (MSA) aims to recognize human affective states by integrating textual, acoustic, and visual information. Despite advances in cross-modal fusion and representation learning, the paper identifies two persistent challenges:

Semantic Gap and Underutilization: There is a significant semantic gap between explicit textual semantics and implicit nonverbal cues (prosody, facial expressions). Existing methods often treat audio and vision as low-level auxiliary signals or compress them into discrete textual descriptions, failing to extract high-level affective cues (polarity, intensity, context) from nonverbal modalities without losing fine-grained temporal dynamics.
Inadequate Fusion Strategies: Current fusion methods struggle to jointly capture global sentiment trends and fine-grained temporal dynamics (e.g., pauses, micro-expressions). Furthermore, they often fail to simultaneously address modality redundancy, modality conflict, and sample-specific differences in modality reliability within a unified framework.

2. Methodology: TGSE-DFN

The authors propose the Text-Guided Generative Semantic Enhancement and Disentangled Adaptive Fusion Network (TGSE-DFN). The architecture consists of five sequential stages:

A. Text-Guided Generative Semantic Expert

Instead of generating discrete natural language descriptions, this module uses a T5 encoder to produce continuous latent semantic representations for audio and vision.

Mechanism: Textual semantics act as a cross-modal anchor. A multi-scale convolutional projection encodes nonverbal temporal structures, which are then filtered via a text-guided cross-attention gating mechanism to align with textual affective semantics.
Prompt-Driven Enhancement: The module employs hard prompts (task descriptions, modality instructions) and learnable soft prompts to guide the frozen T5 encoder. Low-Rank Adaptation (LoRA) is applied to the Query, Key, Value, and Output projections of the T5 encoder to adapt it to the sentiment task with minimal trainable parameters.
Output: Continuous semantic features ( $G_a, G_v$ ) that inherit affective knowledge from the text space while preserving the sequential structure of nonverbal modalities.

B. Generated-Semantics-Driven Tri-Modal Collaborative Enhancement

The generated semantic features are injected back into all three modalities (text, audio, vision) via cross-modal attention.

Tri-Modal Injection: Unlike language-centric approaches that only inject nonverbal data into text, TGSE-DFN enhances the textual representation with generated audio/visual semantics and simultaneously enhances audio/visual representations with their respective generated semantics.
Adaptive Gating: A gating mechanism balances the contributions of generated acoustic and visual semantics when enhancing the text modality.

C. Common-Specific Disentanglement

Following semantic enhancement, the model separates shared affective factors from modality-private cues to reduce redundancy and conflict.

Architecture: An independent Transformer encoder extracts modality-specific representations ( $S_m$ ), while a shared Transformer encoder extracts common representations ( $C_m$ ).
Modality Adapters: To prevent the erasure of modality-specific characteristics in the shared space, lightweight modality adapters apply conditional corrections to the common representations.
Constraints: A reconstruction decoder and a Disentanglement Constraint Loss ( $L_{dec}$ ) are used. $L_{dec}$ enforces feature reconstruction, consistency of common representations across modalities, and statistical independence (via HSIC) between common and specific branches.

D. Text-Centered Bidirectional Specific Cross-Modal Interaction

The disentangled modality-specific representations undergo bidirectional interaction (text-to-modality and modality-to-text) using Transformer encoders and memory encoders to capture complementary relationships.

E. Global-Local Pairwise Communication Adaptive Fusion (GLPC-AF)

This final stage fuses the enhanced and disentangled features using a multi-granular strategy:

Global Branch: Uses a bottleneck-attention network on common representations to capture overall sentiment trends.
Local Branch: Uses CAGE (BiGRU + convolutional gating + attention pooling) on specific representations to model local temporal dynamics (prosody, pauses).
Pairwise Communication: An MLP-Communicator models lightweight interactions between text and nonverbal modalities.
Adaptive Weights: A modality-adaptive weighting module dynamically adjusts the contribution of text, audio, and vision based on sample-specific cue reliability.
Gated Fusion: All components are concatenated and fused via a gated mechanism to produce the final representation for sentiment prediction.

3. Key Contributions

Text-Guided Generative Semantic Enhancement: A mechanism that uses textual semantics as an anchor to guide audio and vision into a continuous latent semantic space, enhancing their high-level affective expressiveness without compressing temporal details into discrete text.
Unified Disentanglement and Fusion Framework: A framework that performs common-specific disentanglement after semantic enhancement, followed by global-local adaptive fusion. This jointly models shared information, private cues, global trends, and local dynamics while adapting to sample-level modality reliability.
Empirical Validation: Comprehensive experiments on CMU-MOSI and CMU-MOSEI datasets demonstrating competitive performance, supported by ablation studies and visualization analyses.

4. Experimental Results

The model was evaluated on CMU-MOSI (2,199 samples) and CMU-MOSEI (22,856 samples).

Performance: TGSE-DFN achieved state-of-the-art or competitive results across multiple metrics (Accuracy-7, Accuracy-5, Accuracy-2, F1, Correlation, MAE).
- On MOSI, it achieved Acc-7: 48.83, Acc-2: 86.74, and F1: 86.69.
- On MOSEI, it achieved Acc-7: 55.55, Acc-2: 86.32, and F1: 86.33.
Comparisons: It outperformed baseline methods including MulT, MAG-BERT, MMIM, and recent disentanglement/fusion models (DLF, MFMB-Net, DGFN).
Ablation Studies: Removing any key component (T5 enhancement, disentanglement structure, disentanglement loss, cross-modal interaction, or adaptive fusion) resulted in performance degradation, confirming the necessity of the unified framework.
Visualization: t-SNE plots showed clear clustering of sentiment classes. CKA similarity analysis confirmed effective separation of common and specific representations. Attention maps demonstrated the model's ability to selectively focus on relevant semantic positions and adaptively balance modality contributions.

5. Significance and Limitations

Significance:
The paper argues that TGSE-DFN addresses the critical limitation of existing MSA methods: the lack of high-level semantic guidance for nonverbal modalities. By using text to guide the generation of continuous latent semantics for audio and vision, the model bridges the semantic gap while preserving temporal dynamics. The integration of disentanglement and adaptive fusion allows the model to handle modality heterogeneity and sample-specific reliability effectively.

Limitations (as stated by the authors):

Computational Overhead: The use of a T5 encoder introduces additional computational costs.
Prompt Design: The current prompt design relies on manual experience.
Generalization: The model's generalization to cross-lingual and complex real-world scenarios requires further validation.

The authors conclude that future work will focus on lightweight semantic enhancement architectures, automated prompt construction, and extending the framework to cross-lingual and cross-domain tasks.