A Survey on Music Generation from Single-Modal, Cross-Modal, and Multi-Modal Perspectives

This paper provides a comprehensive survey of music generation research by categorizing systems across single-modal, cross-modal, and multi-modal perspectives, while examining key aspects such as representation, data alignment, datasets, evaluation methods, current challenges, and future directions.

The Gist

Imagine you are a chef trying to cook a perfect meal. In the past, chefs only had one recipe book (text) or one set of raw ingredients (audio) to work with. They could make a decent dish, but it was hard to capture the exact mood, the visual beauty, or the specific story they wanted to tell.

This paper is like a comprehensive cookbook review for a new generation of "AI Chefs." These AI chefs are learning to cook music not just from a recipe, but by looking at a picture of the dish, watching a video of the cooking process, reading a story about the flavor, and listening to a sample of the taste all at the same time.

Here is a breakdown of the paper's journey, explained simply:

1. The Evolution: From Solo to Symphony

• The Solo Act (Single-Modal): Imagine a musician playing a piano. If they only listen to a recording of a piano and try to play a new song based on it, that's Single-Modal. It's like trying to paint a picture using only a single color. It works, but it's limited.
• The Duet (Cross-Modal): Now, imagine the musician is given a poem and asked to write a song that fits the poem's mood. Or they are shown a picture of a storm and asked to compose thunderous music. This is Cross-Modal. The AI is translating one language (words or images) into another (music).
• The Full Orchestra (Multi-Modal): This is the paper's main focus. Imagine the AI is given a video of a dancing couple, a text description saying "romantic but sad," and a sketch of a sunset. It has to combine all these clues to create a song that fits the dance, matches the words, and captures the sunset's colors. This is Multi-Modal Music Generation.

2. The Ingredients (Representations)

To cook this music, the AI needs to understand different "languages":

• Audio (The Sound): This is the raw sound waves. It's like the actual sizzling of food. It's rich but messy and hard for computers to digest directly, so they compress it into "tokens" (like digital LEGO bricks).
• Symbolic Music (The Sheet Music): This is the structured notes (MIDI, piano rolls). It's like a recipe card with exact measurements. It's precise but lacks the "soul" or the raw sound.
• Text (The Story): This is the description. "A happy jazz song" or "Lyrics about heartbreak." The AI uses models (like the brains behind chatbots) to understand these words.
• Images & Video (The Visuals): This is the mood board. A picture of a rainy street or a video of a dancer. The AI has to figure out how a visual "sadness" translates into a musical "minor key."

3. The Kitchen Tools (Models & Methods)

The paper reviews the tools these AI chefs use:

• The Bridge Builders: Since "text" and "sound" speak different languages, the AI uses special bridges (like CLAP or MuLan) to translate them into a shared space where they can understand each other.
• The Generators:
  • Autoregressive Models: Like writing a sentence one word at a time, predicting the next note based on the previous one.
  • Diffusion Models: Imagine starting with a cloud of static noise and slowly sculpting it into a clear melody, refining it step-by-step until it sounds perfect.
  • Transformers: The "super-brains" that look at the whole picture (or video) at once to understand the context.

4. The Pantry (Datasets)

You can't cook without ingredients. The paper points out a major problem: We don't have enough high-quality, multi-ingredient recipes.

• Most existing data is just "Song + Lyrics" or "Video + Music."
• We are missing massive libraries where a video, a text description, a sketch, and the music are all perfectly synced.
• The authors suggest we need to build bigger, better "pantries" (datasets) and find clever ways to use existing single-ingredient data to train these complex models.

5. The Taste Test (Evaluation)

How do we know if the AI's music is good?

• The Robot Judge (Objective Metrics): Computers measure things like "Does the rhythm match the video?" or "Is the pitch distribution similar to real music?" It's like a machine measuring the temperature and weight of a cake.
• The Human Judge (Subjective Metrics): Ultimately, music is art. Humans have to listen and say, "Does this make me feel sad?" or "Does this sound like a real band?" The paper notes that we need better ways to combine these human feelings with robot measurements.

6. The Future Menu (Challenges & Directions)

Even though these AI chefs are getting good, they still have a long way to go:

• Creativity: Currently, the AI is a bit of a copycat. It tends to remix what it's already heard. We need it to be truly creative and invent new sounds.
• Efficiency: Cooking a symphony takes a long time and a lot of computer power. We need faster ways to generate music so it can happen in real-time (like for a video game).
• The "Uncanny Valley": The music often sounds "almost" human but has a slight glitch. We need to fix the quality so it sounds professional.
• Alignment: Sometimes the AI gets the rhythm right but the emotion wrong. We need it to understand the whole picture, not just the parts.

The Bottom Line

This paper is a map for the future of music. It tells us that while we have amazing tools to turn text, images, and videos into music, we are still in the early stages. The goal is to create an AI that doesn't just play notes, but understands the feeling of a sunset, the energy of a dance, and the story in a poem, and weaves them all into a perfect, original song.

Technical Summary

Based on the provided survey paper, here is a detailed technical summary covering the problem, methodology, key contributions, results, and significance.

1. Problem Statement

While Artificial Intelligence has made significant strides in single-modal tasks (e.g., text-to-text or image-to-image), multi-modal music generation remains a complex and emerging challenge. Humans naturally integrate visual, auditory, and textual cues, but AI systems struggle to synthesize these diverse information sources into coherent musical compositions.

Current literature lacks a comprehensive review focused specifically on modalities in music generation. Existing surveys often focus on foundational models (e.g., Transformers, Diffusion) or specific domains (e.g., symbolic music or emotional generation) without adequately addressing:

• The distinct representations and gaps between modalities (e.g., the semantic gap between audio waveforms and symbolic scores, or between text and music).
• The evolution from single-modal (audio-to-audio) to cross-modal (text-to-music) and finally multi-modal (text+image+video-to-music) generation.
• The scarcity of high-quality, large-scale, and well-aligned multi-modal datasets.
• The lack of systematic evaluation metrics for cross-modal consistency and creativity.

2. Methodology and Framework

The paper adopts a modality-oriented perspective to categorize and analyze music generation systems. It structures the review around three core pillars:

A. Modality Representations

The authors analyze how different data types are encoded for AI processing:

• Audio: Utilizes compression techniques like VQ-VAE, RVQ, SoundStream, and EnCodec to convert waveforms into discrete tokens. Self-supervised models like AudioMAE and MERT are highlighted for learning semantic audio representations.
• Symbolic Music: Covers representations like MIDI, REMI, Piano Rolls, and CP (Compound Words). Pre-training methods (e.g., BERT, MASS) are used to encode structural and melodic information.
• Text: Leverages NLP encoders (e.g., BERT, T5, FLAN-T5) and joint embedding models (e.g., CLAP, MuLan) to map natural language descriptions to musical latent spaces.
• Image & Video: Uses CNNs, ViT, and ViViT for spatial and spatiotemporal feature extraction. Latent Diffusion Models (LDMs) are noted for their ability to bridge visual and auditory latent spaces.

B. Generation Paradigms

The paper categorizes generation methods based on the number and type of input modalities:

1. Single-Modal:
   • Audio-to-Audio: Tasks like inpainting, continuation, and style transfer using models like VampNet, AudioLM, and MusicGen.
   • Symbolic-to-Symbolic: Tasks like melody completion and accompaniment generation using MusicVAE and Theme Transformer.
2. Cross-Modal:
   • Score-to-Audio: Converting MIDI to audio (e.g., MIDI-DDSP, Deep Performer).
   • Text-to-Music: Generating audio/symbolic music from text descriptions or lyrics (e.g., MusicLM, MeLoDy, SongGLM).
   • Visual-to-Music: Generating music from images or videos (e.g., SDMuse, RhythmicNet, LORIS).
3. Multi-Modal:
   • Systems integrating multiple external modalities (e.g., Text + Image + Video) to guide generation.
   • Key architectures include Seed-Music (Text+Audio+Symbolic), MelFusion (Text+Image), MuMu-LLaMA (Text+Image+Video), and XMusic (Text+Image+Video+Audio).
   • Fusion Strategies: The paper details fusion mechanisms such as Cross-Attention, Concatenation, Joint Embeddings, and Bridge Models (e.g., using an LLM to bridge visual and audio tokens).

C. Data and Evaluation

• Datasets: The survey catalogs existing datasets (e.g., LMD, MusicCaps, AIST, HIMV-200K) and highlights their limitations (small scale, lack of fine-grained alignment). It also discusses solutions like using pre-trained models to synthesize paired data or crawling web data.
• Evaluation:
  • Objective Metrics: Includes FID/FAD for audio quality, KLD for distribution similarity, CLAP/MuLan scores for text-audio alignment, and specific controllability metrics (Tempo, Key, Chord matching).
  • Subjective Metrics: Includes MOS (Mean Opinion Score), Turing Tests, and Forced Choice comparisons to assess human perception of quality, relevance, and creativity.

3. Key Contributions

1. First Modality-Centric Survey: Unlike previous reviews focusing on model architectures, this paper uniquely categorizes music generation based on input modalities (Single, Cross, Multi), providing a clear roadmap for how different data sources interact.
2. Comprehensive Technical Taxonomy: It systematically maps the landscape of:
   • Representation learning for diverse modalities (Audio, Symbolic, Text, Vision).
   • Fusion techniques (Cross-attention, Bridge models, Multi-stage conditioning).
   • Specific tasks (Lyrics-to-melody, Video-to-music, Score-to-audio).
3. Critical Analysis of Datasets: It provides a detailed inventory of current datasets, identifying the "data scarcity" bottleneck and proposing strategies for data augmentation and synthetic alignment.
4. Evaluation Framework: It consolidates fragmented evaluation metrics into a structured framework, distinguishing between intrinsic quality (audio fidelity, structural coherence) and extrinsic consistency (alignment with text, image, or video).
5. Identification of Future Challenges: The paper explicitly outlines gaps in creativity (avoiding overfitting), efficiency (autoregressive bottlenecks), modal fusion (handling conflicting signals), and professional applicability.

4. Results and Findings

• Evolution of Capability: The field has progressed from simple single-modal adaptation to complex multi-modal fusion. However, true multi-modal generation (simultaneously processing text, video, and audio) is still in an exploratory stage.
• Architecture Trends: Diffusion Models (LDMs) and Transformers (including DiT and LLMs) are the dominant architectures. The trend is moving toward unified frameworks (e.g., MuMu-LLaMA) that use a large language model as a central "bridge" to integrate diverse modalities before passing them to a generative decoder.
• Data Limitations: High-quality, large-scale, multi-modal datasets are the primary bottleneck. Most existing datasets are small (hundreds to thousands of samples) or lack precise temporal alignment (e.g., lyrics-to-melody or video-to-beat).
• Evaluation Gaps: Current objective metrics (like FAD or CLAP scores) correlate poorly with human perception of "musicality" and "creativity." Subjective evaluations remain the gold standard but are expensive and non-scalable.

5. Significance

• Research Roadmap: This survey serves as a foundational guide for researchers entering the field, clarifying the distinct challenges of different modalities and the state-of-the-art techniques for fusing them.
• Bridging Theory and Application: By highlighting the gap between current research (often limited to short clips or simple prompts) and professional needs (long-form, editable, high-fidelity music), it directs future work toward practical applications in gaming, film, and therapy.
• Standardization: The paper advocates for standardized evaluation metrics and large-scale dataset construction, which are crucial for the reproducibility and advancement of the field.
• Future Outlook: It sets the stage for the next generation of AI music systems that are not just "generative" but creative, efficient, and deeply aligned with human multi-modal intent, moving beyond mere imitation of training data.