🤖 machine learning

Weight-Space Geometry of Offline Reasoning Training

This paper analyzes the weight-space geometry of six offline reasoning training methods on a Qwen3-4B model, revealing that while SFT, RFT, and RIFT converge to nearly colinear updates with similar accuracy, DPO adopts a distinct, near-orthogonal subspace that achieves significantly superior performance on GSM8K and AIME26 benchmarks, whereas Offline GRPO introduces a substantial orthogonal component while remaining within the SFT loss basin.

Original authors: Aleksandr Nikolich, Igor Kiselev, Vladimir Platonov, Karina Romanova

Published 2026-06-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Aleksandr Nikolich, Igor Kiselev, Vladimir Platonov, Karina Romanova

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, giant math tutor (the "Teacher") who can solve complex problems. You want to teach a smaller, cheaper student (the "Student") to think like that tutor. You give the student the Teacher's step-by-step solutions (called "rollouts") and ask the student to learn from them.

The paper asks a simple but deep question: Does it matter how we tell the student to learn?

There are many different "teaching formulas" (loss functions) researchers use. Some say, "Only look at the correct answers." Others say, "Look at all answers, but give extra credit to the good ones." Some say, "Compare the good answers to the bad ones."

The authors wanted to know: Do these different formulas make the student's brain (the computer's weights) change in the same way, or do they create completely different kinds of thinkers?

Here is the breakdown of their findings, using simple analogies:

1. The "Copycat" Group (SFT, RFT, RIFT)

The Analogy: Imagine three students trying to copy a drawing.

Student A copies every single line.
Student B only copies the lines that are perfect.
Student C copies every line, but draws the perfect ones slightly darker.

The Finding: Even though they use slightly different rules, they all end up drawing almost the exact same picture.

Mathematically, the changes to their brains are nearly identical (97% similar).
They all get about the same score on math tests (around 87–88%).
Takeaway: If you just want a student to mimic the teacher's reasoning, it doesn't matter much which of these three specific formulas you use. They are effectively doing the same thing.

2. The "Self-Adjuster" (DFT)

The Analogy: This student uses the same paper as Student A, but they have a weird habit: they automatically make the pen lighter when they are already confident and darker when they are unsure.

The Finding: This small tweak changes the direction of the drawing significantly. The student's brain changes in a way that is very different from the Copycat group, even though they didn't use any "reward" scores to guide them. It's a unique path, but it doesn't necessarily lead to a better score.

3. The "Side-Step" (Offline GRPO)

The Analogy: This student looks at the teacher's work but decides to take a big step sideways. They still stay in the same general neighborhood (the "loss basin"), but they are walking a different path.

The Finding: This method pushes the student's brain in a direction that is 67% different from the Copycat group. In the later layers of the brain (the "final thoughts"), this difference grows to nearly 86%.

However, they still end up with a similar test score (around 87%).
Takeaway: This method explores a new direction, but it doesn't seem to unlock a "superpower" in terms of accuracy compared to the Copycats.

4. The "Outlier" (DPO)

The Analogy: While everyone else is drawing on the same piece of paper in the same room, this student is drawing on a completely different canvas, in a different room, using a different style entirely.

The Finding:

Geometry: The changes to this student's brain are almost 90 degrees (orthogonal) to everyone else. They are doing something fundamentally different.
The Barrier: If you tried to mix this student's brain with the Copycat's brain (a "linear interpolation"), the result would break. They are in different "universes" of solutions.
The Score: Despite being so different, this student got the highest scores (93.5% on math, 30% on hard puzzles).
The Catch: This student was trained with a 10x smaller learning rate (they took much smaller, more careful steps). The authors warn that we can't be 100% sure if the high score is because of the formula or just because they took smaller, more precise steps.

5. The "Live vs. Recorded" Surprise

The paper also looked at what happens if the student learns from live practice (Online) vs. recorded practice (Offline).

Offline: When learning from a fixed set of recorded teacher answers, the "Side-Step" student (Offline GRPO) stays somewhat close to the Copycats.
Online: When the student generates their own practice problems (Online GRPO), they become completely orthogonal (totally different) from the Copycats.
Conclusion: The fact that the "Offline" methods look similar to the Copycats is partly because they are all staring at the exact same fixed set of teacher answers. If you let them generate their own practice, they diverge wildly.

Summary

Most methods (SFT, RFT, RIFT) are just different ways of saying "Copy the teacher." They result in the same brain structure and similar scores.
DPO is the outlier. It builds a completely different brain structure. It gets the best scores, but it does so with a very specific, cautious training style (small learning rate) that makes it hard to compare directly to the others.
The "Offline" nature of the training data is a major reason why many of these methods end up looking so similar to each other.

The paper essentially maps out the "geography" of these learning methods, showing which ones are neighbors and which ones are living on different planets.

Technical Summary: Weight-Space Geometry of Offline Reasoning Training

Problem Statement

Offline reinforcement learning (RL) losses—such as RFT, RIFT, DFT, Offline GRPO, and DPO—are increasingly used to distill reasoning capabilities from large teacher models into smaller student models. While these methods are routinely compared based on downstream benchmark accuracy, their mechanistic distinctness remains unknown. Specifically, it is unclear whether different loss formulations produce qualitatively different weight updates or if they merely converge to similar directions in weight space. This distinction is critical for practitioners selecting loss functions and for researchers interpreting whether "offline RL" represents a single mechanism or a diverse family of approaches.

Methodology

The authors conducted a controlled, weight-space comparison of six offline reasoning training methods: SFT (Supervised Fine-Tuning), RFT (Reward Fine-Tuning), DFT (Down-weighted Fine-Tuning), RIFT (Reward-Weighted In-Context Fine-Tuning), Offline GRPO, and DPO (Direct Preference Optimization).

Experimental Setup:

Base Model: Qwen3-4B-Instruct.
Data: Identical math rollouts generated by a teacher model (DeepSeek-V4-Flash) on DeepScaleR prompts (~40k verified math problems). All methods used the same data, except DPO which consumed ~1.8K paired (chosen, rejected) samples derived from the same rollouts, while others used ~75K rows.
Architecture: LoRA applied only to attention projections (q, k, v, o) with rank 32 and $\alpha=64$ across 144 modules.
Training Protocol: Identical hyperparameters (batch size 32, cosine schedule, 5% warmup) except for learning rates. All methods used a peak learning rate (LR) of $5 \times 10^{-6}$ , except DPO, which used $5 \times 10^{-7}$ (a 10 $\times$ smaller LR per standard convention). Training ran for 1,500 steps, with results reported at step 1,000.
Analysis Metrics: The study analyzed the resulting LoRA weight deltas ( $\Delta W$ $Δ W$ ) using:
1. Cosine similarity (global and per-layer).
2. Principal-angle subspace analysis (SVD of top-10 singular vectors).
3. Linear mode connectivity (interpolating deltas and measuring cross-entropy on GSM8K).
4. Centered Kernel Alignment (CKA) of hidden states.

Key Contributions and Results

1. Convergence of Reward-Weighted MLE Methods

The authors found that SFT, RFT, and RIFT converge to nearly colinear weight deltas.

Geometry: Cosine similarities between these methods are $\ge 0.97$ , with top-1 principal angles around $7^\circ$ (median).
Performance: They achieve statistically indistinguishable GSM8K accuracy (87–88%, $p \ge 0.15$ via McNemar's test).
Implication: Filtering negative samples (RFT) or reward-weighting them (RIFT) does not change the direction of the update relative to plain SFT; it primarily adjusts the step size (Frobenius norm).

2. DFT Divergence

Despite being a minimal modification of SFT (down-weighting confident tokens via a single multiplicative factor), DFT produces a significantly more distinctive update than any explicitly reward-weighted method.

Geometry: DFT shows a cosine of $\sim 0.57$ to SFT, diverging more than reward-weighted methods.
Performance: Accuracy remains comparable to SFT/RFT/RIFT, suggesting the loss basin is shared despite the directional shift.

3. Offline GRPO: Orthogonal Components within a Basin

Offline GRPO introduces a substantial component orthogonal to the SFT direction while remaining within the same loss basin.

Geometry: It adds a globally orthogonal component of $\sim 67\%$ , rising to $\sim 86\%$ in late layers. The cosine to the SFT cluster is $\sim 0.74$ .
Connectivity: Linear interpolation between Offline GRPO and SFT/RIFT shows no barrier, indicating they share the same optimization basin.
Performance: Accuracy is comparable to SFT/RFT/RIFT (87.3%).

4. DPO: A Distinct Subspace and Accuracy Peak

DPO occupies a near-orthogonal subspace and achieves the highest accuracy in the study's protocol.

Geometry: DPO is nearly orthogonal to all other methods (cosine $\le 0.065$ to SFT/RFT/RIFT). It operates in a higher effective rank subspace (rank $\sim 24.5$ vs. $\sim 16$ for others) but with a much smaller Frobenius norm.
Connectivity: Linear interpolation between DPO and other methods reveals a sharp barrier (non-monotonic increase in loss), indicating distinct solution basins.
Representational Similarity: CKA collapses to $\sim 0.46$ in late layers, confirming distinct internal representations.
Performance: DPO achieves 93.5% on GSM8K and 30.0% on AIME26, significantly outperforming all other methods ( $p < 10^{-9}$ on GSM8K).
Caveat: The authors note that DPO's superior performance is entangled with its 10 $\times$ smaller learning rate and the use of ~40 $\times$ fewer training rows. The accuracy gap cannot be cleanly attributed solely to the loss function without a learning-rate-matched comparison.

5. The Role of On-Policy Sampling

A critical finding involves the comparison between Offline GRPO and Online GRPO.

Offline GRPO (trained on fixed rollouts) remains partially aligned with SFT (orthogonal fraction 0.67).
Online GRPO (trained on on-policy rollouts) becomes almost entirely orthogonal to SFT (orthogonal fraction 0.998).
Conclusion: The directional convergence observed between SFT and Offline GRPO is partly an artifact of training on the same fixed rollouts. Replacing these with on-policy samples breaks this convergence, pushing the update into a completely different subspace.

Significance and Claims

The paper argues that the "offline reasoning" landscape is not monolithic but consists of distinct geometric regimes:

Reward-Weighted MLE Cluster: SFT, RFT, and RIFT are mechanistically similar, differing only in step magnitude.
Intermediate Shifters: DFT and Offline GRPO shift direction but remain within the SFT loss basin.
Orthogonal Regimes: DPO and On-Policy methods (like Online GRPO) operate in near-orthogonal subspaces with distinct representational geometries.

The authors emphasize that while DPO achieves the highest accuracy, this result is confounded by optimizer settings (specifically the learning rate). Therefore, the paper does not claim DPO is causally superior due to its loss function alone, but rather that it occupies a unique geometric space that, under the specific protocol used, yields better results. The study highlights that "offline RL" does not name a single mechanism; rather, the choice of data (fixed vs. on-policy) and loss function jointly determine the weight-space trajectory.

Limitations: The study is limited to a single domain (math), a single base model checkpoint, and attention-only LoRA. The DPO comparison is acknowledged as imperfect due to the learning rate discrepancy, and a full sweep of the contrastive family (IPO, KTO, SimPO) is left for future work.