💬 NLP

Lost in Interpolation: Why Predictive Feedback Fails in Diffusion Language Models

This paper identifies that the Euclidean linear interpolation used in existing Masked Diffusion Language Models is geometrically mismatched with their hyperspherical embedding space, and proposes Spherical Soft-Masking (S-SM), a method using spherical linear interpolation that significantly improves generative quality and perplexity without degrading convergence.

Original authors: Lavanya Nigam, Ishaan Bansal, Aryan Sood, Vidit Aggarwal, Gaurav Kumar Nayak

Published 2026-08-10

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Lavanya Nigam, Ishaan Bansal, Aryan Sood, Vidit Aggarwal, Gaurav Kumar Nayak

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to write a story, but instead of letting it write word-by-word like a human, you give it a blank page where every word is hidden behind a "MASK." The robot's job is to peek at the whole page, guess what words should go in the blanks, and then slowly reveal them one by one. This is how a type of AI called a "Diffusion Language Model" works. It's like a game of "Guess the Word" played in reverse, where the AI starts with total confusion and gradually clears up the picture.

To make this game faster and smarter, researchers use a trick called "soft-masking." Instead of the robot just saying, "Yes, I'm sure this word is 'cat'" or "No, keep it masked," it gets to say, "I'm 80% sure it's 'cat' and 20% sure it's 'dog'." It blends these guesses together to create a new, fuzzy hint for the next round. The big question this paper tackles is: How do you mathematically blend these guesses? Most people have been doing it the old-fashioned way, as if the robot's brain is a flat, straight-line map. But what if the robot's brain is actually shaped like a giant, curved globe?

The Lost in Interpolation Mystery

The authors of this paper, a team from the Indian Institute of Technology Roorkee, decided to investigate the shape of the "brain" inside these AI models. They discovered that the way these models store words isn't flat like a sheet of paper; it's actually curved, like the surface of a sphere.

Think of the AI's vocabulary as a giant, invisible globe. Every word is a point on the surface of this ball. When the AI makes a guess, it's pointing to a spot on that globe. The problem is that the standard method for blending guesses (called "Linear Interpolation" or LERP) treats the globe like a flat map. If you draw a straight line between two cities on a flat map, you might cut right through the center of the Earth. But if you are walking on the surface of the globe, you have to follow the curve of the ground. The paper shows that the standard method is forcing the AI to walk through the center of the Earth, which is a weird, unnatural path that confuses the model and slows it down.

The "Lost in Interpolation" Discovery

The researchers found that when the AI tries to blend its predictions using the old, flat-line method, it gets "lost." They measured the angle between the "masked" state (the blank) and the "predicted" state (the guess) and found it stays at a steady 73 degrees throughout the entire training process. They also noticed that the "size" (or norm) of the word vectors stays almost exactly the same, no matter how common or rare the word is. These two clues are like footprints that prove the AI is walking on a hypersphere (a multi-dimensional ball), not a flat plane.

Because the AI lives on a sphere, the authors argue that using straight lines to mix information is the wrong tool. It's like trying to measure the distance between New York and London with a ruler laid on a table instead of a string wrapped around a globe.

The Solution: Spherical Soft-Masking (S-SM)

To fix this, the team invented a new method called Spherical Soft-Masking (S-SM). Instead of drawing a straight line through the center of the Earth, S-SM forces the AI to walk along the surface of the globe.

Here is how they did it:

The Globe Walk: Instead of averaging the guesses in a straight line, they used a special math trick called a "Fréchet mean" to find the average spot on the sphere's surface.
The Curved Blend: They used a technique called SLERP (Spherical Linear Interpolation). Imagine a string stretched tight between two points on a balloon; SLERP follows that string along the curve of the balloon, never leaving the surface.
The Result: They tested this new method on a 169-million-parameter AI model. The results were impressive. The new method didn't just work; it worked better.

What the Numbers Say

The team ran their new S-SM method against the old standard (TopK/LERP) and a version with no feedback at all. They tested it with different amounts of "thinking time" (called NFE budgets, ranging from 64 to 512 steps).

Better Quality: The new method produced text that was much closer to human writing. They measured this using a score called MAUVE. At higher budgets (like 512 steps), S-SM improved the MAUVE score by up to 2 times compared to the old method, and by 27.5% to 56.1% compared to the standard TopK/LERP approach.
Fewer Mistakes: The AI made fewer confusing errors. The "generative perplexity" (a measure of how surprised the AI is by its own text) dropped by 16.9% to 19.6% compared to the baseline.
No Trade-offs: Usually, when you make AI smarter, it becomes less creative (it repeats itself). But the authors found that S-SM kept the "entropy" (a measure of creativity and randomness) exactly the same as the old methods. It got smarter without getting boring.

Why the Old Way Failed

The paper explicitly rules out the idea that the flat, straight-line method is just "okay" or "good enough." The data suggests it is fundamentally wrong for this type of AI. When the researchers looked at the "confidence weight" (a number the AI learns to decide how much to trust its own guesses), they saw something interesting. Under the new S-SM method, the AI learned to trust its geometric feedback much more, settling on a weight of about 0.056, whereas the old method only trusted it at 0.030. This suggests the AI "felt" more confident when it wasn't being forced to walk through the center of the Earth.

The Bottom Line

This paper doesn't just suggest a new idea; it provides strong evidence that the geometry of AI brains is spherical, and we have been using the wrong math to talk to them. By switching from straight lines to curved paths (SLERP), the authors showed that we can make these models generate better text, faster, without losing their spark. It's a reminder that sometimes, to move forward, you have to stop walking in a straight line and start following the curve.

Technical Summary: Lost in Interpolation: Why Predictive Feedback Fails in Diffusion Language Models

Problem Formulation
Masked Diffusion Language Models (MDLMs) generate text by iteratively denoising a fully masked sequence. To prevent information loss inherent in binary unmasking decisions, Soft-Masking (SM) was introduced, replacing the binary choice with a continuous, confidence-weighted blend of the [MASK] embedding and the top- $k$ predicted token embeddings.

Existing formulations construct this blend using Linear Interpolation (LERP) in the raw embedding space. This approach implicitly assumes the embedding space is Euclidean. However, the authors identify a critical geometric mismatch: trained token embeddings in MDLMs concentrate on a hyperspherical shell of near-constant radius rather than filling the ambient Euclidean space.

The paper presents two empirical diagnostics confirming this hyperspherical geometry:

Norm Stability: The $L_2$ norms of token embeddings remain essentially flat across four orders of magnitude of vocabulary-frequency rank, indicating a constant-radius shell.
Geodesic Angle: The angle between the normalized [MASK] embedding and the normalized mean of the top- $k$ predictions remains stable at approximately 73° throughout training.

In a hyperspherical geometry, linear interpolation (a straight line) cuts through the interior of the sphere, producing a point off the manifold. This "off-manifold" point fails to accurately capture the semantics of the interpolation, leading to training degradation and suboptimal generation quality when using LERP.

Methodology: Spherical Soft-Masking (S-SM)
The authors propose Spherical Soft-Masking (S-SM) as a geometrically principled replacement for the linear feedback operator. S-SM is designed as a "drop-in" replacement that preserves the training objective, unmasking rules, and confidence schedules of existing MDLMs while altering only the interpolation primitive.

The S-SM operator performs three distinct steps:

Spherical Aggregation: Instead of a Euclidean weighted mean, the top- $k$ predicted embeddings are aggregated using a Fréchet mean (specifically, the Karcher mean) computed intrinsically on the unit hypersphere. This is approximated via a few iterations of Karcher flow, starting from the top-1 prediction.
Spherical Interpolation: The blend between the [MASK] direction and the aggregated Fréchet mean is performed using Spherical Linear Interpolation (SLERP). This traces the great-circle arc (geodesic) on the sphere's surface rather than a chord through the interior.
Norm Restoration: The resulting unit vector is rescaled to the native norm of the [MASK] token before being passed back to the backbone, ensuring norm compatibility with the pre-trained model.

The method includes numerical safeguards (clamping inner products and denominators) to handle cases where the mask and target directions are nearly identical, falling back to a normalized linear blend in such edge cases.

Key Contributions

Identification of Hyperspherical Geometry: The authors demonstrate that MDLMs operate on a hyperspherical embedding manifold, characterized by a stable $\approx 73^\circ$ angle between mask and prediction directions and flat embedding norms. This geometry renders linear interpolation geometrically invalid.
S-SM Operator: The introduction of a geometric drop-in replacement that utilizes Fréchet means and SLERP to aggregate and blend embeddings while respecting the manifold structure.
Empirical Validation: Comprehensive evaluation showing that S-SM avoids the training degradation induced by LERP and consistently improves generation quality across various sampling budgets and random seeds.

Results
The authors evaluated S-SM on the continued pre-training of a released 169M-parameter MDLM checkpoint on OpenWebText. Comparisons were made against a vanilla MDLM baseline (no feedback) and the standard TopK/LERP baseline.

Generative Perplexity (Gen PPL): S-SM achieved consistently lower perplexity, showing improvements of 16.9–19.6% over the vanilla baseline and 12.2–14.2% over TopK/LERP across all tested NFE (Number of Function Evaluations) budgets ( $T \in \{64, 128, 256, 512\}$ ).
MAUVE Score: S-SM delivered significant gains in distributional similarity (MAUVE). At larger inference budgets ( $T=256, 512$ ), it nearly doubled the MAUVE score of the vanilla baseline and improved over TopK/LERP by 27.5–56.1%.
Training Dynamics: Unlike LERP, which induced training degradation (higher perplexity than the no-feedback baseline), S-SM maintained or improved training perplexity.
Learned Confidence: The learned mixing weight $\lambda$ converged to a higher value under S-SM ( $\lambda \approx 0.056$ ) compared to LERP ( $\lambda \approx 0.030$ ), suggesting the optimizer places greater trust in the geometrically well-formed feedback.
Entropy: Output entropy remained comparable across configurations, indicating that quality gains were not achieved at the cost of diversity.

Significance and Claims
The paper claims that the failure of predictive feedback in current MDLMs is not due to the concept of soft-masking itself, but rather the incorrect interpolation primitive (LERP) used in a hyperspherical space. By aligning the feedback mechanism with the intrinsic geometry of the embedding space, S-SM unlocks the full potential of soft-masking.

The authors emphasize that S-SM is a minimal, drop-in modification that requires no changes to the backbone architecture, training objectives, or unmasking schedules. The results suggest that respecting the hyperspherical nature of token embeddings is crucial for the stability and performance of diffusion-based language models, particularly when utilizing continuous feedback mechanisms.

Technical Summary: Lost in Interpolation: Why Predictive Feedback Fails in Diffusion Language Models

More like this