🤖 machine learning

MUNI: Multimodal Unified Latent Diffusion for Coherent Any-to-Any Generation

The paper introduces MUNI, an end-to-end multimodal latent diffusion framework that unifies subset-conditioned generation and unconditional joint sampling through a shared stochastic latent space trained via a novel routed objective, thereby overcoming the limitations of existing LLM-based or text-aligned models to achieve superior coherence in any-to-any generation.

Original authors: Kyeongmin Yeo, Yunhong Min, Minhyuk Sung

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kyeongmin Yeo, Yunhong Min, Minhyuk Sung

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a team of artists: a painter, a musician, and a poet. In the past, if you wanted them to create a story together, you had to force them to speak a single, common language (like text) to understand each other. If the painter wanted to show a picture, they had to describe it in words first, and the musician had to turn those words back into sound. This often meant losing the unique "flavor" of the original picture or the specific emotion of the sound.

MUNI is a new system that lets these artists work together without forcing them to translate everything into words first. Instead, it gives them a shared "dream space" where they can all meet, understand each other's vibes, and create something new together.

Here is how MUNI works, broken down into simple concepts:

1. The Problem: The "Translator" Bottleneck

Most current AI systems act like a strict translator. If you give them a picture and ask for music, the AI first has to turn the picture into a description (text), and then turn that text into music.

The Flaw: This is like trying to describe a complex painting using only the word "blue." You lose all the details. Also, if you only have a picture and no text description, the system gets stuck because it was trained to always expect a text description.

2. The Solution: A Shared "Dream Space"

MUNI creates a shared latent space. Think of this as a central meeting room where the painter, musician, and poet all have a seat.

No Translation Needed: The painter can walk in with a sketch, the musician with a melody, and the poet with a poem. They don't need to translate their work into a common language; they just drop their "vibe" into the room.
Any-to-Any: You can give the system any combination of inputs (just a picture, just sound, or both) and ask it to generate the missing pieces. It's like saying, "Here is a picture of a cat, now imagine what it sounds like," or "Here is a sound of rain, now imagine what it looks like."

3. How It Learns: The "Group Project" Analogy

The paper introduces two main tricks to make this work better than previous attempts:

Trick A: The "One Big Team" Approach
Old methods often trained the artists separately first, then tried to force them to agree later. MUNI trains them all at the same time, as one big team.

The Analogy: Imagine a band practicing together from day one. They learn how to listen to each other while they learn to play their instruments. This means the "shared space" they create is perfectly tuned to how they actually play, rather than being a clumsy compromise between two separate groups.

Trick B: The "Strict Teacher" (The Training Objective)
This is the most important part. The paper realized that if you just let the artists share a room, they might start sharing too much or too little.

The Problem: If the painter shares every tiny detail of their sketch (like the specific shade of red on a single brushstroke) with the musician, the musician gets confused. That detail is private to the painter and doesn't belong in the shared room.
The Fix: MUNI uses a special training rule (a "routed objective") that acts like a strict teacher. It tells the artists: "Only share the things that help everyone understand the main idea. Keep your private details to yourself."
- Coherence: It ensures that if they generate a picture, sound, and text together, they all match perfectly (e.g., if the text says "dog barking," the sound is a bark, and the picture shows a dog).
- Minimality: It forces the shared room to only hold the "common ground," leaving the unique, messy details to the individual artists to handle later.

4. The Results: Better Harmony

The paper tested MUNI on two levels:

Simple Puzzles (PolyMNIST): A controlled test where the AI had to match numbers and shapes. MUNI was much better at keeping the pieces consistent when generating them all at once compared to other methods.
Real World (Images, Text, Audio): They tested it with real photos, sentences, and sounds.
- Conditional Generation: When given a prompt (like "a cat"), MUNI was just as good as the best existing systems at making the cat look, sound, and read correctly.
- Unconditional Generation: This is where MUNI shined. When asked to just "make up a random scene" without any prompt, other systems often produced mismatched results (a picture of a beach with a sound of a city). MUNI produced scenes where the image, text, and sound all felt like they belonged together naturally.

Summary

Think of MUNI as a universal translator that doesn't use words. It builds a shared "vibe" space where different types of data (images, sounds, text) can mix and match freely. By teaching the system to only share the essential "common ground" and keep the private details separate, it creates much more coherent and consistent multi-sensory experiences than previous AI models.

Technical Summary: MUNI: Multimodal Unified Latent Diffusion for Coherent Any-to-Any Generation

Problem Statement

Current multimodal generative models face significant limitations in achieving "any-to-any" generation, where a model must generate any subset of modalities conditioned on any other subset (including unconditional joint sampling). Existing approaches generally fall into two categories with distinct drawbacks:

LLM-based Foundation Models: These represent all modalities as discrete tokens in a unified sequence. While flexible, they struggle to leverage modality-specific continuous generators (e.g., latent diffusion for images, flow matching for audio) and require text-paired data for training, limiting fidelity and the ability to capture direct cross-modal correlations not mediated by language.
Diffusion/Flow-based Extensions: Recent works attempt to extend continuous generative modeling to multimodal settings but often rely on text-aligned embeddings, require learning conditional paths for every subset configuration (demanding fully-paired data), or impose deterministic mappings with matched dimensionality.

The core challenge is to develop a framework that unifies subset-conditioned cross-modal generation and unconditional joint sampling without relying on text-aligned embeddings, fully-paired training data, or pre-computed latent spaces.

Methodology: MUNI

The authors propose MUNI (Multimodal Unified Latent Diffusion), an end-to-end framework that extends the Unified Latents (UL) concept to arbitrary subset conditioning. The framework consists of three main components:

Modality-Specific Encoders and Decoders: Instead of a single unified encoder, MUNI employs specific encoders $q_{\phi_m}(z | x_m)$ for each modality and expressive, modality-specific decoders $p_{\theta_m}(x_m | z)$ .
Shared Stochastic Latent: A single shared latent variable $z$ mediates all generation tasks.
Learned Flow-Based Prior: A shared prior $q_\psi(z)$ is learned jointly with the encoders and decoders, rather than being fitted to a frozen latent space.

Key Architectural and Objective Innovations

MUNI addresses the insufficiency of standard multimodal variational inference aggregation rules (like Product of Experts or Mixture of Experts) when coupled with expressive decoders and learned priors. The authors identify three necessary criteria for the shared latent:

Coherence-sufficiency: The latent must explain dependencies among modalities to enable coherent unconditional joint sampling.
Predictive sufficiency: Subset latents must retain sufficient information to predict missing modalities.
Minimality: The shared latent should not absorb modality-specific details that can be modeled by the expressive decoders.

To satisfy these criteria, MUNI introduces a routed training objective with three specific structural choices:

Non-Mixture Aggregation: For a subset of observed modalities $A$ , the posterior is formed by aggregating all available unimodal experts (via Product or Hellinger aggregation) into a single posterior, rather than sampling from a mixture of experts. This ensures the latent contains information from all observed modalities.
Target-Detached Self-Reconstruction: When reconstructing a modality $x_m$ that is part of the observed subset $A$ , the encoder for $x_m$ is detached (stop-gradient) during the aggregation step. This prevents the shared latent from being forced to encode modality-specific details that are only useful for self-reconstruction, enforcing the minimality criterion.
Leave-One-Out Prior Learning: The prior loss is applied only to latents inferred from full-modality subsets or "leave-one-out" subsets (where all but one modality are observed). This ensures the prior is trained on latents that are coherence-sufficient, avoiding the contamination of the prior with latents inferred from insufficient evidence.

The training objective combines reconstruction losses (for both held-out prediction and target-detached self-reconstruction) and a prior loss gated to specific subset routes.

Key Contributions

End-to-End Multimodal Latent Diffusion: MUNI extends latent diffusion to any-to-any generation by jointly training modality-specific encoders, expressive decoders, and a shared flow-based prior under a single objective, eliminating the need for a two-stage "frozen latent" pipeline.
Novel Training Objective for Variational Inference: The paper identifies that standard aggregation rules fail to satisfy coherence, predictivity, and minimality simultaneously when paired with learned priors. The proposed routed objective explicitly aligns the latent space with these criteria through non-mixture aggregation, target-detached reconstruction, and selective prior learning.
Scalable Any-to-Any Framework: MUNI operates without text-aligned embeddings or fully-paired triplets, enabling training on partially observed multimodal data (e.g., image-text pairs, audio-text pairs) while supporting unconditional co-generation.

Experimental Results

The authors evaluate MUNI on two benchmarks:

PolyMNIST-Quadrant-Labels: A controlled benchmark with image and label modalities. MUNI matches or exceeds strong baselines (MVAE, MMVAE, MoPoE, HELVAE) on conditional generation (single/multi-label to image) and achieves the largest margin in unconditional coherence, significantly outperforming baselines in generating consistent digit identities and quadrant positions across modalities.
Large-Scale Image-Text-Audio Benchmark: Using pairwise datasets (LAION-COCO, AudioCaps, VGGSound) without fully observed triplets.
- Conditional Generation: MUNI achieves state-of-the-art performance among generalist baselines (CoDi, OmniFlow, FlowBind) on many-to-one alignment tasks. Notably, it improves Audio-Image Similarity (AIS) by up to +7.14 over the next-best generalist.
- Unconditional Co-generation: MUNI demonstrates substantial gains in unconditional coherence. Compared to OmniFlow (the only directly comparable generalist supporting unconditional sampling), MUNI improves scores by +5.57 CLIP (Text-Image), +10.60 CLAP (Text-Audio), and +31.85 AIS (Audio-Image).

Significance and Claims

The paper claims that coherent any-to-any generation depends not only on how modality evidence is aggregated but also on what information the shared latent is trained to contain.

By explicitly enforcing criteria of coherence, predictivity, and minimality through the routed objective, MUNI successfully scales the probabilistic multimodal variational inference perspective to large-scale, practical image-text-audio generation. The results demonstrate that this framework can retain competitive conditional generation capabilities while substantially improving unconditional coherence, a task where existing diffusion-based and VAE-based baselines often fail. The work suggests that the multimodal variational inference perspective, when paired with expressive generative components and a learned prior, is a viable path toward unified multimodal generation without relying on text as an intermediary.