🤖 AI

Second-Order Multi-Level Variance Correction for Modality Competition in Multimodal Models

This paper introduces ML-FOP-SOAP, a second-order optimization framework featuring multi-level variance correction and Fisher-Orthogonal Projection to resolve modality competition in autoregressive multimodal models, thereby enabling stable large-batch training and significantly improving both sample efficiency and wall-clock speed compared to AdamW.

Original authors: Yishun Lu, Wes Armour

Published 2026-05-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yishun Lu, Wes Armour

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a single student to be both a world-class painter and a brilliant poet at the same time. You want this student to look at a picture and write a story about it, or read a story and draw the scene. This is what modern "multimodal" AI models do: they try to understand images and text simultaneously.

However, the paper by Lu and Armour identifies a major problem: The student gets confused because the two subjects speak different languages.

The Problem: The "Loud" Student vs. The "Quiet" Student

In the AI's training process, the "painting" part (images) is like a very loud, chaotic student who shouts out huge, messy answers. The "poetry" part (text) is like a quiet, precise student who gives small, very specific answers.

When the teacher (the computer algorithm) tries to average out their answers to decide what to learn next, the loud student drowns out the quiet one. The AI ends up getting really good at generating messy images but forgets how to understand text properly. This is called "Modality Competition."

The Old Way: The Bad Teacher

Most AI models use a standard teaching method called AdamW. The authors compare this to a teacher who only listens to the volume of the students' voices.

Because the image student is so loud, the teacher thinks, "Okay, we'll focus entirely on painting!"
The quiet text student gets ignored, and the AI fails to learn the balance it needs.

The New Solution: A Smarter Teacher

The authors propose a new, smarter teaching framework called ML-FOP-SOAP. They break it down into three clever tricks:

1. The "Second-Order" Glasses (SOAP)

First, they switch to a better type of glasses called SOAP.

Analogy: Instead of just listening to volume, this teacher looks at the shape and direction of the answers.
Result: The teacher realizes that the loud image student is actually shouting in a chaotic direction, while the quiet text student is pointing in a very valuable direction. The teacher stops being fooled by the volume and starts respecting the direction. This helps the AI learn both skills better than before.

2. The "Noise Canceling" Headphones (FOP)

Even with the better glasses, the teacher still struggles because the image student's noise is so strong it accidentally pushes the text student's ideas out of the way.

Analogy: The authors add a Fisher-Orthogonal Projection (FOP). Think of this as a special noise-canceling headphone.
How it works: It listens to the difference between the two students. If the image student is shouting something that contradicts the text student, the teacher uses the headphones to cancel out that specific conflict without silencing the student entirely.
Result: The AI can now learn to paint and write poetry without one ruining the other. It achieves a "Pareto improvement," meaning it gets better at both tasks simultaneously, rather than trading one for the other.

3. The "Telescoping" Zoom (ML-FOP)

Training these models requires looking at massive amounts of data at once (like 8,192 examples at a time). If the teacher tries to analyze every single tiny detail in that huge pile of data, they get overwhelmed and slow down.

Analogy: The authors use a Multi-Level Hierarchical Folding strategy. Imagine looking at a forest. Instead of counting every single leaf (which takes forever), you first look at the whole forest, then zoom in on a few trees, then zoom in on a few branches.
How it works: This method captures the "rough" differences between the students quickly, then zooms in to catch the "fine" details only when necessary.
Result: The teacher can handle massive crowds of data without getting tired or slow.

The Results

When the authors tested this new method on real AI models (named Janus and Emu3), they found:

Faster Learning: The AI learned the same amount of material 1.4 times faster (in terms of data used) and finished the training 1.5 times faster in real-world time compared to the old standard.
Better Balance: The AI didn't just get better at one thing; it got better at both understanding text and generating images at the same time.
Big Data Ready: It worked perfectly even when the class size was huge (8,192 students), a situation where the old methods usually failed and gave up.

In short: The paper shows that by using a smarter way to listen to the "loud" and "quiet" parts of an AI, and by using a special technique to cancel out their arguments, we can train powerful AI models that are faster, more stable, and better at doing everything at once.

Technical Summary: Second-Order Multi-Level Variance Correction for Modality Competition in Multimodal Models

Problem Statement

The unification of image generation and text understanding under a single autoregressive next-token prediction paradigm has introduced a critical challenge known as modality competition. While this architectural convergence enables seamless multimodal foundation models, it induces severe instability during pretraining. The core issue arises from gradient heterogeneity: visual objectives typically involve high-dimensional, spatially diffuse prediction errors with high variance, whereas textual objectives concentrate on a compact semantic manifold with lower variance.

Under standard first-order optimizers like AdamW, which rely on diagonal moment estimates, this geometric mismatch leads to a suboptimal optimization trajectory. The optimizer fails to model the complex, non-diagonal cross-modal covariance structure. Consequently, modalities with larger gradient variance (typically vision) exert a disproportionate influence on parameter updates, suppressing the semantically dense but lower-variance signals from text. This results in a trade-off where improving visual generation often degrades textual understanding, and vice versa. Furthermore, scaling these models to ultra-large global batch sizes (e.g., 8192) exacerbates these issues, as standard variance correction methods applied only at coarse accumulation boundaries fail to preserve fine-grained gradient discrepancies.

Methodology

The authors propose ML-FOP-SOAP, a second-order optimization framework designed to stabilize multimodal alignment through Multi-Level Variance Correction. The methodology is built upon three pillars:

1. Second-Order Preconditioning with SOAP

To address the limitations of first-order methods, the paper leverages the SOAP optimizer. Unlike Shampoo, which uses uncentered second moments vulnerable to early-stage visual outliers, SOAP projects gradients into a skewed eigenbasis and applies coordinate-wise adaptive normalization. This provides a memory-efficient, robust surrogate for the natural gradient ( $P(g) \approx F^{-1}g$ ), effectively capturing local curvature without the prohibitive memory cost of inverting the full Fisher Information Matrix (FIM).

2. Fisher-Orthogonal Projection (FOP)

While SOAP provides a stable baseline, it does not fully resolve the trade-off between modalities. The authors introduce Fisher-Orthogonal Projection (FOP) to actively neutralize cross-modality conflicts.

Mechanism: Instead of applying preconditioning solely to the averaged batch gradient, FOP operates on the gradient differences ( $g_{diff}$ ) between micro-batches within an accumulation window.
Projection: It constructs a lightweight forward metric proxy $M(v) \approx Fv$ using the internal states of the tensor preconditioner. The difference vector is projected orthogonally to the mean descent direction ( $g_{avg}$ ) in the Riemannian manifold defined by the Fisher metric.
Result: This isolates the pure cross-modality variance ( $g_{diff}^\perp$ ) without interfering with the primary optimization trajectory. The final update direction combines the average gradient with this variance-aware residual, effectively expanding the search space beyond the one-dimensional subspace of the batch mean.

3. Multi-Level Hierarchical Folding (ML-FOP)

To make FOP tractable under massive gradient accumulation (required for batch sizes like 8192), the authors propose a Multi-Level Hierarchical Folding mechanism inspired by the Multi-Level Monte Carlo (MLMC) paradigm.

Strategy: Rather than applying projections at every micro-step (which incurs high wall-clock overhead) or only at the window boundaries (which loses fine-grained information), the method discretizes the accumulation window into a dyadic hierarchy.
Execution: It recursively fuses computationally lightweight cumulative snapshots to reconstruct segment-level averages. These multi-scale summaries are then processed bottom-up using the FOP operator. This approach captures fine-grained, cross-modality variance across the accumulation hierarchy while amortizing the computational cost.

Key Contributions

Second-Order Optimization for Modality Competition: The paper systematically demonstrates that while first-order methods (AdamW) and standard uncentered second-order methods (Shampoo) struggle with extreme visual gradient outliers, the SOAP optimizer establishes a robust foundation for multimodal alignment.
Variance Control via FOP: The introduction of a geometry-aware Fisher-Orthogonal Projection mechanism that projects gradient differences orthogonally to the primary descent trajectory. This eliminates the conventional trade-off between textual understanding and visual generation, yielding strict Pareto improvements.
Efficient Multi-Level FOP: A novel ML-FOP hierarchical folding strategy that enables variance correction under massive gradient accumulation without prohibitive computational overhead, preserving intra-step multi-scale variance.
Ultra-Large-Batch Benchmark: Validation of the framework on unified architectures (Janus and Emu3) scaled to a global batch size of 8192, a regime where existing optimizers suffer catastrophic degradation.

Experimental Results

Experiments were conducted on Janus-400M and Emu3-600M models trained on combined LLaVA-3M and LLaVA-12M datasets.

Sample Efficiency: Compared to AdamW, the ML-FOP-SOAP framework improves sample efficiency by up to 1.4×. For instance, on Janus-400M, it reaches convergence thresholds at ~3.1B tokens compared to 4.5B tokens for AdamW.
Wall-Clock Time: The method accelerates end-to-end training by up to 1.5× (e.g., reducing training time from 8.3 hours to 5.5 hours for Janus-400M).
Modality Balance: Decoupled loss curves for Image-to-Text (I2T) and Text-to-Image (T2I) tasks show that while standard SOAP improves T2I generation, it often plateaus on I2T. ML-FOP-SOAP successfully drives both losses downward simultaneously, achieving strict Pareto improvements.
Scalability: At a global batch size of 8192, AdamW and standard SOAP suffer from premature plateauing. Only ML-FOP-SOAP maintains a continuous downward trajectory, demonstrating robust stability in ultra-large-batch regimes.

Significance and Claims

The paper claims to provide a robust optimizer for scaling multimodal foundation models. Its significance lies in shifting the solution to modality competition from architectural modifications (e.g., decoupled encoders) to the optimization infrastructure itself.

The authors assert that by integrating second-order preconditioning with Fisher-aware variance correction, it is possible to:

Stabilize the training of unified autoregressive models against the inherent geometric mismatch of visual and textual gradients.
Resolve the trade-off between modalities, allowing for simultaneous improvement in generation and understanding tasks.
Enable training at unprecedented batch sizes (8192) without the catastrophic degradation seen in standard optimizers, thereby improving hardware utilization on modern GPU clusters.

The work suggests that for future large-scale multimodal pretraining, the choice of optimizer is as critical as the model architecture, with second-order, variance-aware methods offering a necessary path forward for stable, efficient scaling.