Hierarchical Semantic Correlation-Aware Masked Autoencoder for Unsupervised Audio-Visual Representation Learning

1. Problem Statement

The paper addresses the challenge of learning aligned multimodal embeddings from weakly paired, label-free corpora. Current unsupervised audio-visual (AV) learning methods face three primary practical hurdles:

• Weak Alignment & Noise: Real-world clips often contain multiple concurrent events, temporal misalignments, and spurious co-occurrences, violating the "hard-positive" assumption (one-to-one pairing) common in standard contrastive learning.
• Pre-extracted Features: Many modern pipelines only provide compact, pre-extracted feature vectors (e.g., VGGish for audio, InceptionV3 for video) rather than raw data. This limits the ability to use patch-based masking and requires methods to build strong semantics from condensed descriptors.
• Ambiguity: In multi-event clips, a single audio clip may have multiple valid visual matches (and vice versa), making the standard "single-positive-per-anchor" assumption brittle and prone to confirmation bias.

2. Methodology: HSC-MAE

The authors propose HSC-MAE, a dual-path Teacher-Student framework designed to enforce semantic consistency across three hierarchical levels of representation (from coarse to fine).

A. Architecture Overview

The model utilizes shared encoder parameters for both audio and visual modalities but operates in two coordinated forward modes:

1. CCA Path (Teacher/Global Geometry):
   • Input: Clean, unmasked inputs.
   • Mechanism: Uses Deep Canonical Correlation Analysis (DCCA) to align audio and visual embeddings into a shared, low-dimensional modality-invariant subspace.
   • Role: An Exponential Moving Average (EMA) of the encoder acts as a Teacher. It produces stable canonical embeddings and affinity weights without receiving gradients, serving as a source of geometric targets and soft positive mining.
2. MAE Path (Student/Robust Semantics):
   • Input: Value-masked inputs (a fraction of feature dimensions are zeroed out).
   • Mechanism: A student network reconstructs the missing features and learns local semantics.
   • Role: Enforces robustness to partial observations and learns discriminative intra-modal features.

B. Three Hierarchical Objectives

The framework optimizes three complementary loss functions:

1. Global-Level: Canonical-Geometry Correlation (DCCA)

   • Goal: Align global distributions of audio and visual embeddings.
   • Implementation: Maximizes the top-$r$ canonical correlations between clean-view embeddings ($Z^{cca}_a, Z^{cca}_v$). This establishes a structured manifold for finer constraints to build upon.
   • Loss: $L_{cca} = -\sum \rho_m(Z^{cca}_a, Z^{cca}_v)$.

2. Local-Level: Neighborhood-Semantics Correlation (Soft Top-$k$ InfoNCE)

   • Goal: Preserve multi-positive relational structures among semantically similar instances, relaxing the brittle single-positive assumption.
   • Implementation: The Teacher mines the top-$k$ neighbors for each anchor and assigns soft affinity weights ($W_{ij}$) based on semantic similarity. The Student uses these weights in an InfoNCE loss.
   • Loss: An affinity-weighted multi-positive InfoNCE that tolerates multiple plausible positives per anchor.

3. Sample-Level: Conditional-Sufficiency Correlation (Masked Autoencoding)

   • Goal: Ensure individual embeddings retain sufficient discriminative content to predict missing feature dimensions.
   • Implementation: The student reconstructs masked feature dimensions from the corrupted input.
   • Loss: Mean Squared Error (MSE) between the original input and the reconstructed output ($L_{rec}$).

C. Optimization Strategy

• Multi-Task Weighting: Learnable log-variance weights ($\sigma$) are used to dynamically balance the competing objectives ($L_{rec}, L_{infoNCE}, L_{cca}, L_{dis}$), preventing optimization conflicts.
• Distillation: An optional consistency loss ($L_{dis}$) aligns student embeddings with the stable teacher geometry to reduce confirmation bias.
• Linear CCA Projection: A linear projection layer is appended to the encoder to further reduce dimensionality and emphasize correlated directions during inference.

3. Key Contributions

1. HSC-MAE Framework: A novel dual-path masked autoencoder that enforces hierarchical semantic correlations (Global Geometry $\to$ Local Neighborhood $\to$ Sample Sufficiency) specifically tailored for pre-extracted feature vectors.
2. Teacher-Guided Soft Mining: Introduction of an EMA teacher that generates stable affinity weights for soft top-$k$ neighborhood mining. This mitigates the "one-positive" bias and handles the ambiguity of multi-event clips.
3. Principled Multi-Task Learning: A scheme using learnable weights and optional distillation to reconcile reconstruction, correlation, and contrastive objectives, stabilizing optimization across hierarchical levels.
4. State-of-the-Art Performance: Demonstrated significant improvements over strong unsupervised baselines on standard benchmarks.

4. Experimental Results

The method was evaluated on two datasets: AVE (1,955 clips) and VEGAS (28,103 clips), using Mean Average Precision (mAP) for Audio-to-Visual (A2V) and Visual-to-Audio (V2A) retrieval.

• Performance Gains:
  • AVE Dataset: HSC-MAE achieved an average mAP of 0.7737, surpassing the previous best (CAV-MAE) by +15.72% (0.6165 $\to$ 0.7737).
  • VEGAS Dataset: Achieved an average mAP of 0.8026, improving over CAV-MAE by +4.91%.
• Ablation Studies:
  • Removing any single component (Global DCCA, Soft InfoNCE, or Reconstruction) resulted in consistent performance drops.
  • Reconstruction and Soft InfoNCE were identified as the most critical components for robust representation.
  • Mask Ratio: Performance peaked at moderate mask ratios (0.1–0.3); excessive masking disrupted cross-modal semantics, while low masking failed to regularize the model.
• Qualitative Analysis: Visualizations showed that the model successfully retrieves semantically matching clips (e.g., truck audio $\to$ truck video), with errors primarily occurring between acoustically/visually similar classes (e.g., bus vs. truck), indicating high semantic alignment.

5. Significance

• Robustness to Weak Supervision: HSC-MAE effectively handles the noise and ambiguity inherent in real-world, unlabeled audio-visual data without requiring manual labels.
• Feature-Level Adaptability: Unlike many MAE variants that require raw pixel/spectrogram patches, this approach is specifically designed to work with compact, pre-extracted features, making it highly applicable to existing large-scale multimedia pipelines.
• Theoretical Grounding: By explicitly modeling hierarchical correlations, the method provides a principled way to reduce retrieval risk and avoid the confirmation bias common in standard contrastive learning.
• Practical Application: The framework offers a scalable solution for embodied agents and multimodal systems operating in label-scarce environments.