Towards Expressive and Faithful Audio-to-Image Generation: A Unified Multimodal Dataset and Synthesis Framework

Technical Summary: Towards Expressive and Faithful Audio-to-Image Generation

Problem Statement
Audio-to-Image (A2I) generation, a subfield of cross-modal generation, aims to synthesize static visual content from audio inputs. Despite the remarkable progress in Text-to-Image (T2I) models, A2I performance remains fundamentally limited. The authors attribute this to the lack of high-quality, fine-grained datasets. Existing datasets (e.g., AudioSet, VGGSound) often prioritize audio quality over visual fidelity or suffer from misalignment between audio and randomly sampled video frames. Consequently, current methods struggle to achieve high-quality generation or precise cross-modal alignment, often relying on coarse category alignment or indirect text-token conversion which neglects the description gap between audio and image.

Methodology
To address these limitations, the paper proposes a unified framework comprising a new dataset, a synthesis pipeline, and a novel generation model.

1. Dataset Synthesis (A2I-Set):
   The authors constructed A2I-Set, a large-scale, high-quality tri-modal dataset containing 323,446 images and 260,060 audio clips paired with detailed text captions, where a single audio clip may correspond to multiple valid images. The data synthesis pipeline involves four stages:

   • Vision-Enhanced Audio Captioning: Leveraging existing datasets (AudioSet, VGGSound), the authors use visual language models (PLLaVA) and audio models (Qwen-audio-2) to generate captions. These are fused via GPT-4o-mini to create reliable audio captions with visual clues, reducing hallucinations common in single-modality captioning.
   • Real Image Extraction: High-resolution, stable video frames are extracted from source videos using optical flow jitter filtering and scene segmentation (TransNet V2). Image quality is assessed using ARNIQA.
   • Synthetic Image Generation: Recognizing that single video frames often fail to align perfectly with complex audio, the authors use the FLUX.1-Krea-dev model to generate realistic images. Audio captions are refined by GPT-4o-mini into professional visual generation instructions (including subject, atmosphere, lighting, and composition) before being fed to the generator.
   • Alignment Filtering: A rigorous filtering process ensures alignment. "Sam Audio" performs object-level audio separation, while visual grounding models (Qwen3-VL-Flash, Ground DINO) verify the presence of sound-emitting objects in the images. A VLM-based "Cross-Check" filters out synthetic images that do not resemble real photographs.

2. Model Architecture (AudioCanvas):
   The authors propose AudioCanvas, a model fine-tuned on A2I-Set using a Stable Diffusion (SD 1.4) backbone.

   • Pre-Alignment Audio Projector: Before fine-tuning the diffusion model, an audio projector aligns CLAP audio embeddings with the CLIP text space using a combination of Mean Squared Error (MSE) and InfoNCE contrastive losses.
   • FiLM-weighted Audio-Text Fusion (FAT Fusion): Instead of simple concatenation, the model employs a decoupled cross-attention mechanism. Audio and text features are injected separately into the UNet. A Feature-wise Linear Modulation (FiLM) strategy is then applied to dynamically modulate the intermediate features based on both audio and text representations, allowing for flexible feature fusion.
   • Training Strategy: The model is trained with empty text inputs to force reliance on audio, utilizing classifier-free guidance. A mix of real and synthetic data is used to balance aesthetic quality and realism.

Key Contributions

• A2I-Set: A unified dataset comprising 323,446 images and 260,060 audio clips covering 241 hierarchical audio classes (music, speech, natural sounds), featuring detailed multimodal captions and high-fidelity images. It includes a manually curated evaluation set (A2I-eval) and a balanced subset (A2I-balanced).
• Synthesis Pipeline: A complete workflow for generating high-quality audio-image-text tri-paired data, addressing the scarcity of aligned data through vision-enhanced captioning and controlled synthetic generation.
• AudioCanvas: A new A2I baseline model that introduces a Pre-Alignment projector and a FiLM-weighted fusion module, demonstrating that high-quality data can yield superior results even with a standard backbone.

Results
Experiments on the A2I-eval (mixed-source) and Landscape (single-source) datasets show that AudioCanvas outperforms existing state-of-the-art methods (including AudioToken, MACS, CoDi, and SoundAdapter) in several metrics:

• Visual Quality: AudioCanvas achieves competitive Fréchet Inception Distance (FID) and Inception Score (IS), with particularly strong performance in Human Preference Scores (HPSv3) and Aesthetic scores (Aes), indicating high visual appeal.
• Cross-Modal Alignment: The model demonstrates superior Audio-Image Similarity (AIS) and Text-Image Similarity (TIS), suggesting better semantic alignment between the input audio and the generated image.
• Data Efficiency: Despite being trained on a dataset of 323K images (compared to CoDi's ~100M), AudioCanvas achieves performance that surpasses models trained on significantly larger or noisier datasets, highlighting the efficacy of the A2I-Set quality.

Significance and Claims
The paper claims that the primary bottleneck in A2I generation is data quality rather than model architecture alone. By introducing a rigorous data synthesis pipeline and a high-quality dataset, the authors demonstrate that it is possible to achieve expressive and faithful audio-to-image generation. They assert that their approach opens new avenues for the field by proving that high-fidelity, fine-grained cross-modal alignment can be achieved through careful data curation and targeted model adaptation, even without massive-scale training data or novel backbone architectures. The authors acknowledge limitations regarding potential style overfitting to synthetic data and the use of an older diffusion backbone, but emphasize the foundational value of their dataset and framework for future research.