🤖 machine learning

Training Without Orthogonalization, Inference With SVD: A Gradient Analysis of Rotation Representations

This paper provides a theoretical foundation for training rotation networks with direct 9D regression and applying SVD projection only at inference by demonstrating that SVD orthogonalization during training introduces severe, condition-number-dependent gradient distortion and direction errors, whereas its removal avoids these tradeoffs while 6D Gram-Schmidt methods suffer from asymmetric gradient signals.

Original authors: Chris Choy

Published 2026-04-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chris Choy

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot how to understand 3D rotations (like how a camera turns or how a human arm moves). To do this, the robot needs to output a "rotation matrix"—a specific 3x3 grid of numbers that describes a perfect turn.

The problem is that this grid has strict rules: it must be "orthogonal" (the lines must be perfectly perpendicular, like the corners of a perfect cube). If the robot guesses a grid that is slightly crooked, it's mathematically invalid.

For a long time, researchers tried to fix this by forcing the robot to "straighten" its guess every single time it made a mistake during training. They used a mathematical tool called SVD (Singular Value Decomposition) to snap the crooked grid into a perfect cube instantly.

This paper argues that this "instant fixing" is actually hurting the robot's learning.

Here is the breakdown of the paper's findings using simple analogies:

1. The "Over-zealous Editor" Analogy (Why SVD hurts training)

Imagine you are writing a story, and your editor (the SVD algorithm) stops you after every sentence to rewrite it into perfect, grammatically flawless English before you can write the next one.

The Problem: Because the editor is so aggressive, they change your original intent. If you wrote "The cat sat," and the editor hears "The cat sat slightly," they might snap it to "The cat sat perfectly."
The Gradient Issue: In machine learning, the "gradient" is the signal telling the robot how to change its guess to get better. When you use SVD during training, the editor distorts this signal.
- The "Explosion": If the robot's guess is very bad (very crooked), the editor panics and screams, sending a massive, chaotic signal back to the robot. This is like the editor screaming "YOU ARE WRONG!" so loudly the robot can't hear the actual instructions on how to fix it.
- The "Blind Spot": The editor only cares about making the grid look perfect. They ignore the parts of the signal that tell the robot, "Hey, you are far away from the target." It's like the editor only checking if the spelling is right, but ignoring that you are writing about the wrong topic entirely.

The Paper's Discovery: The authors did a deep mathematical autopsy of this "editor." They found that the editor's signal is distorted and uneven. Sometimes it amplifies noise, sometimes it deletes important information. It's like trying to steer a car while the steering wheel is being randomly spun by a chaotic force.

2. The "Two-Step Dance" (The Solution)

The paper proposes a much simpler strategy: Stop editing while you practice.

Training (The Practice): Let the robot output a "pseudo-rotation." It can be a crooked, messy, imperfect grid. The robot learns by comparing this messy grid directly to the target.
- Why this works: The signal is now clean. If the robot is off by 10 degrees, it gets a clear, proportional signal to move 10 degrees closer. No screaming, no distortion. The robot learns the shape of the problem perfectly.
Inference (The Performance): Only after the robot has finished learning and is ready to take the test (or control the real robot), do we apply the "editor" (SVD).
- Why this works: At this point, the robot is already very good. Its guess is almost a perfect cube. The editor just does a tiny, final polish to snap it into the perfect legal shape. Since the robot is already close, the editor doesn't panic or distort the signal.

3. The "9D vs. 6D" Debate (Why more numbers are better)

There was another debate: Should the robot guess 6 numbers (a 6D representation) or 9 numbers (a 9D representation)?

The 6D Approach (Gram-Schmidt): This is like building a house by laying the first brick, then the second, then calculating the third based on the first two.
- The Flaw: If you mess up the first brick, the whole house is crooked, and the error ripples down. The signal gets "asymmetric"—the first brick gets all the attention, but the later bricks get confused signals.
The 9D Approach: This is like laying all 9 bricks independently.
- The Advantage: Every brick gets a clear, equal signal. If one is wrong, you fix that specific brick without messing up the others. The paper proves mathematically that this "9D direct regression" is the most stable way to learn.

4. The "Best of Both Worlds" Conclusion

The paper combines these two insights into a winning strategy:

Train using 9D direct regression (let the robot guess all 9 numbers freely, without forcing them to be perfect). This gives the cleanest, most honest learning signal.
Infer using SVD (apply the "perfect snap" only at the very end). This ensures the final output is mathematically valid and, surprisingly, reduces the final error by about 3x compared to other methods.

Summary

Think of it like learning to ride a bike.

Old Way (SVD-Train): You wear a helmet that instantly corrects your balance while you are learning. It's so aggressive it confuses your brain about which way to lean. You never learn the true feeling of balance.
New Way (SVD-Inference): You learn to ride on a wobbly, unbalanced bike. You feel the wobble and learn to correct it naturally. Once you are a pro, you switch to a perfectly balanced bike for the race.

The paper proves mathematically that learning on the wobbly bike (training without orthogonalization) and switching to the perfect bike only at the end (inference with SVD) is the fastest and most accurate way to master rotation.

1. Problem Statement

Representing 3D rotations ($SO(3)$) for deep learning is challenging because $SO(3)$ is a 3-dimensional manifold embedded in $\mathbb{R}^{3\times3}$ with orthogonality constraints ( $R^\top R = I, \det(R)=1$ ).

The Conflict: Recent empirical work (Gu et al., 2024) showed that removing orthogonalization during training and applying it only at inference (using SVD) yields superior results compared to applying orthogonalization (SVD or Gram-Schmidt) during training.
The Gap: While empirical evidence favors "Direct 9D Regression + SVD at Inference," the theoretical reasons were incomplete. Specifically:
1. Why does SVD orthogonalization specifically harm training gradients?
2. Why is SVD preferred over Gram-Schmidt (GS) at inference?
3. Why is 9D regression superior to 6D (GS-based) regression in the unorthogonalized regime?

2. Methodology

The authors provide a rigorous Jacobian analysis of the backward pass for rotation representations. They specialize the derivation for $3\times3$ matrices and $SO(3)$ projection to derive exact spectral properties.

SVD Analysis: They derive the exact spectrum of the Jacobian for the mapping $M \to R = UV^\top$ (where $M = U\Sigma V^\top$ ). They analyze the singular values, condition number, and rank of this Jacobian.
Gram-Schmidt Analysis: They analyze the Jacobian of the 6D representation (two columns recovered via GS) to characterize gradient coupling and asymmetry.
Stability Analysis: They compare standard SVD backpropagation against stabilized variants (Taylor expansion approximations) and direct regression.
Convergence Analysis: They model gradient descent convergence rates for Direct Regression vs. SVD-Train under Frobenius and Geodesic losses.

3. Key Contributions & Theoretical Results

A. SVD Gradient Pathology (The Convergence Paradox)

The paper derives the exact spectrum of the SVD Jacobian ( $J_{SVD}$ ):

Rank: The Jacobian has rank 3 (matching the dimension of $SO(3)$), meaning 6 of the 9 gradient dimensions are projected to zero.
Singular Values: The three non-zero singular values are $\sigma_{ij} = \frac{2}{s_i + s_j}$ for $i < j$ , where $s_i$ are the singular values of the predicted matrix $M$ .
Condition Number: $\kappa(J_{SVD}) = \frac{s_1 + s_2}{s_2 + s_3}$ .
Implication:
- Gradient Explosion: When the network is far from $SO(3)$ (early training), $s_3 \approx 0$ . This causes the condition number to blow up ( $O(1/s_3)$ ) and the spectral norm to explode.
- Gradient Distortion: The gradient signal is anisotropic; perturbations in the $(2,3)$ singular vector plane are amplified relative to the $(1,2)$ plane.
- Information Loss: SVD backpropagation discards 2/3 of the gradient energy (the normal component to the manifold). The optimizer receives no signal pushing the matrix toward orthogonality, whereas direct regression retains this signal.
- Stabilization Failure: Even state-of-the-art stabilization techniques (clamping singular values) cannot eliminate this pathology without introducing significant gradient direction errors.

B. Gram-Schmidt (6D) Gradient Asymmetry

The authors prove that the 6D Gram-Schmidt Jacobian has an asymmetric spectrum:

One-Directional Coupling: The first column parameters ( $t'_1$ ) receive gradient signals from all three output rotation columns. However, the second column parameters ( $t'_2$ ) are "blind" to errors in the first column ( $\partial r'_1 / \partial t'_2 = 0$ ).
Conditioning: The condition number diverges as $t'_1$ and $t'_2$ become parallel.
Result: This asymmetry leads to unequal learning signals and increasing projection error from column 1 to column 3 at inference.

C. Optimality of 9D + SVD-Inference

Training: Direct 9D regression (predicting raw matrix entries) has a Jacobian of $I_9$ (Identity). It has condition number $\kappa=1$ , full-rank gradients, and no cross-coupling. It avoids all singularities.
Inference: SVD is the Least Squares Optimal Projector onto $SO(3)$.
- Under isotropic Gaussian noise, SVD projection reduces the expected squared error by a factor of 3 compared to the raw matrix (retaining only the 3 tangential dimensions, removing the 6 normal dimensions).
- SVD is coordinate-independent (equivariant under rotation), whereas Gram-Schmidt is not.

4. Results

Theoretical Justification: The paper mathematically proves why the "Train with 9D, Inference with SVD" paradigm works:
1. Training: Avoids the $O(1/\delta)$ gradient explosion and rank deficiency of SVD-Train and the asymmetry of 6D-GS.
2. Inference: Leverages SVD's optimality as a projector to minimize reconstruction error.
Convergence Rates:
- Direct Regression: Converges linearly with a rate independent of the state.
- SVD-Train: Converges $\sim 2\times$ slower even when near $SO(3)$ (due to rank-3 projection) and significantly slower when far from $SO(3)$ due to ill-conditioning.
Geodesic Loss: The paper shows that using Geodesic loss with SVD-Train compounds singularities (both $1/\sin\theta$ from the loss and $1/(s_2+s_3)$ from SVD), making it strictly worse than Frobenius loss on raw matrices.

5. Significance

Resolves Empirical Mysteries: This work provides the rigorous mathematical foundation for the empirical success of Pseudo Rotation Matrices (PRoM) and similar methods, moving beyond "it works empirically" to "here is exactly why it works."
Design Principle: It establishes a clear design principle for rotation estimation: Decouple training and inference. Train with unconstrained, full-rank regression (9D) to ensure stable, isotropic gradients, and apply the optimal geometric projection (SVD) only at inference.
Broader Impact: The analysis of SVD gradient pathologies (specifically the $1/(s_i+s_j)$ scaling) applies to any differentiable pipeline involving SVD, suggesting that similar "train without projection, project at inference" strategies may benefit other manifold learning tasks.

In summary, the paper demonstrates that orthogonalization layers during training introduce unavoidable gradient distortions and information loss. The optimal strategy is to treat the rotation matrix as a standard 9D regression target during training and apply the geometrically optimal SVD projection only when a valid rotation is required at inference.