Technical Summary: S2Dialog

Problem Statement

Multimodal Dialogue Retrieval (MDR) aims to retrieve dialogues from a Multimodal Dialogue Bank (MDB) that match a target dialogue in both textual semantics and acoustic conversational styles. This capability is critical for downstream tasks such as Emotion Recognition in Conversation, Spoken Dialogue Systems, and Conversational Speech Synthesis, where external dialogue examples serve as semantic and stylistic references.

However, existing retrieval methods face two primary limitations:

1. Granularity: Most approaches operate at the utterance level rather than the dialogue level, failing to capture the global semantic coherence and stylistic consistency of an entire conversation.
2. Modality: Many methods are unimodal or rely on simple aggregation (e.g., average pooling) of pre-trained features. These approaches often lack dedicated mechanisms to model complex interactions between textual and acoustic modalities across a full dialogue sequence, leading to suboptimal alignment of semantics and styles.

Methodology

The authors propose S2Dialog, a unified framework designed for dialogue-level semantic-style retrieval. The architecture consists of three core components:

1. Dialogue-level Textual Retriever (DTR)

The DTR encodes the textual modality of an entire dialogue into a fixed-dimensional semantic representation through a two-stage process:

• Utterance-level Extraction: A frozen pre-trained Sentence-BERT model extracts semantic embeddings for each utterance.
• Temporal Aggregation: A Gated Recurrent Unit (GRU) processes the sequence of utterance embeddings to capture long-range dependencies and evolving semantic flow. The final hidden state is passed through a non-linear projection head (MLP with ReLU) to generate the final dialogue-level semantic representation ($\mathcal{F}_t$).

2. Dialogue-level Acoustic Retriever (DAR)

Parallel to the textual branch, the DAR encodes the acoustic modality to capture conversational style:

• Utterance-level Extraction: A frozen pre-trained Wav2Vec2-IEMOCAP model extracts acoustic embeddings for each speech segment, capturing phonetic and prosodic cues.
• Temporal Aggregation: A GRU aggregates these embeddings to model stylistic consistency and acoustic transitions. The final hidden state is projected via an MLP to produce the dialogue-level style representation ($\mathcal{F}_a$).

3. Dialogue-level Textual-Acoustic Contrastive Learning (DTACL)

To align the textual and acoustic modalities within a unified retrieval space, the authors introduce DTACL. Unlike standard contrastive learning that focuses on utterance-level or single-modality pairs, DTACL operates on complete dialogues.

• Positive Sets: For a target dialogue, the positive set includes its own representation in the other modality (e.g., $\mathcal{F}_a$ is positive for $\mathcal{F}_t$) and the representations of Top-K semantically/stylistically similar candidate dialogues.
• Negative Sets: The negative set consists of Bottom-K dissimilar dialogues.
• Objective: The framework minimizes a contrastive loss ($\mathcal{L}_{cl}$) that pulls semantically and stylistically similar dialogues closer while pushing unrelated ones apart. This is achieved through separate losses for the textual anchor ($\mathcal{L}_{cl}^t$) and acoustic anchor ($\mathcal{L}_{cl}^a$), which are summed for the final objective.

Key Contributions

1. S2Dialog Framework: The introduction of a dialogue-level semantic-style retrieval framework specifically designed for multimodal dialogue banks, moving beyond utterance-level aggregation.
2. Dedicated Retrievers and Contrastive Learning: The design of specialized textual and acoustic dialogue-level retrievers, coupled with a Dialogue-level Textual-Acoustic Contrastive Learning module. This jointly models dialogue semantics and conversational styles, enabling more discriminative retrieval from complete multimodal dialogues.
3. Empirical Validation: Extensive experiments on the DailyTalk dataset demonstrating that S2Dialog outperforms representative baselines, including text pooling, multimodal pooling, text summarization, and multimodal summarization approaches.

Experimental Results

The model was evaluated on the DailyTalk dataset, a high-quality multimodal corpus containing 2,541 conversations.

• Performance vs. Baselines: S2Dialog achieved a Recall@10 of 50.68% and Recall@50 of 83.56%. This significantly outperformed the best baseline (Multimodal Summarization, which achieved 25.60% Recall@10) and other categories like Text Pooling (11.60% Recall@10).
• Ablation Studies:
  • Removing the Textual Retriever caused the largest drop in performance (Recall@10 dropped to 39.60%), indicating that semantic matching is the primary signal.
  • Removing the Acoustic Retriever also reduced performance (Recall@10 to 47.60%), confirming the value of style modeling.
  • Removing Bottom-K negative samples resulted in the most severe degradation (Recall@10 dropped to 21.12%), highlighting the critical role of hard negative mining in shaping the retrieval space.
  • Removing either contrastive loss ($\mathcal{L}_{cl}^t$ or $\mathcal{L}_{cl}^a$) weakened the discriminative power of the respective representation spaces.
• Comparison with Large Models: S2Dialog substantially outperformed general-purpose multimodal large models (e.g., Qwen3-Omni, Kimi-Audio, Step-Audio) when used for retrieval, suggesting that general encoders are not optimized for dialogue-level semantic-style tasks.
• Similarity Trends: Analysis of similarity scores across retrieval ranks showed that the full model produces a clear descending trend (high similarity for Top-K, low for Bottom-K), whereas ablated variants showed flatter, less discriminative curves.

Significance and Claims

The paper claims that S2Dialog addresses a critical gap in dialogue retrieval by shifting the paradigm from utterance-level aggregation to holistic dialogue-level multimodal modeling. The authors assert that:

• Modeling entire dialogue sequences is necessary to capture the long-range dependencies and stylistic consistency inherent in human communication.
• The synergy between dedicated retrievers and contrastive learning significantly enhances the discriminative power between heterogeneous conversational contexts.
• The proposed framework provides a more robust foundation for tasks requiring external dialogue references, such as improving the generation capabilities of dialogue systems and enhancing conversational speech synthesis.

The authors acknowledge limitations, noting that the current evaluation is restricted to the DailyTalk dataset and that the system relies on frozen feature extractors. Future work is planned to extend the evaluation to larger, more diverse corpora and to explore finer-grained retrieval approaches.