🤖 machine learning

M+Adam: Low-Precision Training via Additive-Multiplicative Optimization

The paper proposes M+Adam, a novel optimizer that combines additive and multiplicative updates to overcome the complementary failure modes of low-precision training, thereby enabling stable and accurate training of LLaMA-style models using only BF16, FP8, and FP4 master weights.

Original authors: Xiaoyuan Liang, Sebastian Loeschcke, Mads Toftrup, Anima Anandkumar

Published 2026-07-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xiaoyuan Liang, Sebastian Loeschcke, Mads Toftrup, Anima Anandkumar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a giant, hyper-intelligent robot (a Large Language Model) to write like a human. To do this, you have to adjust millions of tiny dials on the robot's brain. Usually, these dials are adjusted with extreme precision, like using a microscope to turn a screw. But microscopes are heavy, slow, and expensive.

The paper asks: What if we used a cheap, lightweight wrench instead?

This is the world of low-precision training. Instead of using high-precision "master weights" (the robot's memory of the dials), researchers try to store them in a smaller, coarser format like BF16, FP8, or even FP4. It's like trying to measure a mountain's height with a ruler that only has marks every 10 feet.

The Problem: The "Rounding" Trap

The authors discovered that when you use these coarse rulers, standard training methods (called Adam) hit a wall. Here's why:

The Big Mountain Problem: When a dial is set to a huge number, the "ruler" marks are very far apart. If the robot needs to move the dial by a tiny amount, the ruler can't see it. The tiny move gets rounded off to zero, and the robot thinks, "I didn't move anything!" The robot gets stuck, unable to climb the mountain.
The Zero Valley Problem: Other methods tried to fix this by using multiplicative updates (like Madam). Instead of adding a tiny number, they multiply the dial by a factor. This works great for big mountains because the step size grows with the dial. However, if a dial is stuck at exactly zero, you can't multiply it to get it moving. It's like trying to push a car that has no wheels; multiplying zero by anything still gives you zero. Also, if the dial needs to flip from positive to negative (crossing zero), these methods get confused and can't make the switch.

The Solution: M+Adam (The Two-Tool Kit)

The paper introduces M+Adam, a new optimizer that acts like a clever mechanic carrying two different tools at once. It combines the best of both worlds to fix the "complementary failure modes."

Tool 1: The Additive Wrench (Adam-style). This tool is great for small, precise adjustments. It can nudge a dial from zero, flip a sign from positive to negative, and handle tiny movements near the bottom of the valley.
Tool 2: The Multiplicative Lever (Madam-style). This tool is great for big, heavy lifting. When the dial is huge and the additive wrench gets "rounded off" to nothing, the multiplicative lever kicks in. It scales the dial up or down, ensuring the robot keeps making progress even on the tallest mountains.

M+Adam uses both tools simultaneously. If the additive tool gets stuck due to rounding, the multiplicative tool keeps the robot moving. If the multiplicative tool can't escape zero, the additive tool pushes it out.

The Proof: Did it Work?

The authors didn't just guess; they tested this on a massive scale.

The Test: They trained "LLaMA-style" language models ranging from 60 million to 1 billion parameters.
The Budget: They used training budgets from 1x to 8x the standard "Chinchilla" budget (a standard measure of how much data a model sees).
The Precision: They tested with BF16, FP8, and NVFP4 master weights.

The Results:
In every single test, M+Adam consistently beat the standard Adam optimizer.

In the most extreme low-precision setting (NVFP4), M+Adam reduced the "perplexity" (a measure of how confused the model is) by 16.6% for the 350M model compared to Adam.
Even with 1 billion parameters, M+Adam showed clear improvements over Adam in all precision regimes.

The paper proves mathematically that this combined approach guarantees the training loss will go down (a "monotone descent"), meaning the robot is always learning and never getting stuck in a loop.

What It Is NOT

It is important to know what this paper does not claim:

It does not say that M+Adam is a magic bullet that eliminates all memory costs. In fact, the paper notes that M+Adam adds a tiny bit of extra memory (one extra "state" for the multiplicative tool), though they show this can be compressed later.
It does not claim that standard methods like Stochastic Rounding (a technique to fix rounding errors) are useless. They tested Adam with Stochastic Rounding, and M+Adam still performed better.
It does not suggest that we should stop using high-precision hardware entirely. Instead, it shows that if we must use low-precision storage, M+Adam is the right way to do it.

The Bottom Line

The paper suggests that when you are forced to use "coarse" rulers (low-precision weights) to train giant AI models, you can't rely on just one type of movement. You need a hybrid approach. M+Adam is that hybrid, using both additive and multiplicative steps to ensure the robot never gets stuck, whether it's climbing a giant mountain or escaping a zero-valley. It's a principled step toward training massive AI models efficiently without needing the heavy, expensive high-precision memory we usually rely on.

Technical Summary: M+Adam – Low-Precision Training via Additive–Multiplicative Optimization

Problem Statement

Training large language models (LLMs) in full precision (FP32) incurs significant memory, compute, and energy costs. While modern accelerators support reduced-precision formats like BF16, FP8, and FP4, standard training pipelines typically retain high-precision (FP32) "master weights" to ensure stability. Directly training with low-precision master weights often leads to degraded accuracy and instability.

The authors identify a specific failure mode in low-precision optimization: coarse quantization resolution at large weight magnitudes. In floating-point systems, the spacing between representable values (quantization bins) grows with the magnitude of the number. Consequently, standard additive optimizers (like Adam) often fail to make progress when weight magnitudes are large, as the calculated update steps fall below the local quantization step and are rounded to zero. Conversely, purely multiplicative optimizers (like Madam), which scale weights relative to their current magnitude, fail near zero or when sign changes are required, as they cannot escape exact zero initialization or flip signs effectively.

Methodology: M+Adam

The paper proposes M+Adam, a novel optimizer that combines the complementary strengths of additive and multiplicative update mechanisms to overcome the limitations of low-precision arithmetic.

Core Mechanism

M+Adam maintains two parallel update branches that are combined at each step:

Additive Branch (Adam-style): Operates directly on the weight space. It computes standard first and second moments of the gradient ( $g$ ) to produce an additive update $u^{(a)}$ . This branch is effective for handling small magnitudes, escaping zero initialization, and facilitating sign changes.
Multiplicative Branch (Madam-style): Operates on the logarithmic scale (exponent space). It computes the gradient with respect to the exponent, $g^{(e)} = (\ln 2) w_t g$ , and uses adaptive moments to produce a relative multiplicative update $u^{(m)}$ . This branch ensures progress at large magnitudes where additive updates would be rounded away.

The weight update rule is defined as:
$w_{t+1} = w_t + w_t u^{(m)}_t + u^{(a)}_t$
This structure allows the optimizer to perform scale-aware relative updates when magnitudes are large (via the multiplicative term) while retaining the flexibility to change signs and update near-zero values (via the additive term).

Theoretical Analysis

The authors provide a theoretical guarantee for M+Adam under standard smoothness assumptions. They model the update as a "scale-then-add" operation in local coordinates. Theorem 4.1 establishes that if the learning rates and smoothness constants satisfy specific conditions, the method ensures monotone descent ( $L(w_{t+1}) < L(w_t)$ ) unless the gradient is zero. This proves that the combined geometry is a valid optimization strategy.

Implementation Details

Precision Support: The method is designed for BF16, FP8, and NVFP4 master weight storage.
State Management: M+Adam introduces one additional optimizer state (the second-moment accumulator for the multiplicative branch). The authors demonstrate that this state can be compressed using low-rank approximations (specifically Apollo-E, a rank-4 approximation) to minimize memory overhead, keeping the footprint close to standard BF16-state AdamW.
No Stochastic Rounding: Unlike some prior approaches, M+Adam achieves stability without requiring stochastic rounding (SR) or Kahan-style compensated accumulation, though it is compatible with them.

Key Contributions

Identification of Complementary Failure Modes: The paper formally characterizes the failure modes of additive optimizers (stalling at large magnitudes due to rounding) and multiplicative optimizers (stalling at zero and inability to flip signs), showing they are complementary.
M+Adam Algorithm: A new optimizer that unifies these two geometries, providing a principled approach to low-precision training without FP32 master weights.
Theoretical Descent Guarantee: A proof of monotone descent for the combined additive-multiplicative update rule under standard smoothness assumptions.
Empirical Validation at Scale: Extensive experiments on LLaMA-style models ranging from 60M to 1B parameters across 1–8× Chinchilla training budgets.

Experimental Results

The authors evaluated M+Adam across various precision regimes: BF16/BF16, BF16/FP8, FP8/FP8, and NVFP4/FP8.

Consistent Improvement: M+Adam consistently outperformed tuned AdamW baselines across all model sizes and precision regimes.
Perplexity Reductions: In the most aggressive low-precision setting (NVFP4/FP8), M+Adam reduced perplexity by significant margins compared to AdamW (e.g., 16.6% improvement for the 350M model). Even in BF16/BF16 training, M+Adam reduced perplexity by 2.8% to 13.3% depending on model size.
Robustness: The method remained stable without stochastic rounding. It also outperformed AdamW combined with stochastic rounding (AdamW+SR) and Kahan compensation.
Scaling: The performance gains persisted as the training budget increased (up to 8× Chinchilla) and as the model size scaled to 1B parameters.
Memory Efficiency: Using the Apollo-E compression technique, the additional memory state required for the multiplicative branch was reduced, resulting in an optimizer state footprint only slightly larger than standard BF16 AdamW, while retaining most of the perplexity gains.

Significance and Claims

The paper claims that combined additive–multiplicative updates are especially useful when low-precision master weights become a dominant source of optimization error.

The significance of this work lies in providing a principled solution to the numerical instability of low-precision training. By aligning the optimizer's update geometry with the non-uniform structure of floating-point numbers, M+Adam mitigates the rounding errors that plague standard additive optimizers at large magnitudes and the stagnation issues of multiplicative optimizers near zero.

The authors conclude that M+Adam represents a step toward end-to-end low-precision training that does not rely on FP32 master weights. This capability is crucial for leveraging modern AI hardware (tensor cores) optimized for FP8 and FP4, potentially reducing memory overhead and energy costs for training large-scale models. The work suggests that future optimizers should be designed with the specific constraints of floating-point representation in mind, rather than treating low precision merely as a quantization problem to be solved by rounding techniques.