Revisiting an Old Perspective Projection for Monocular 3D Morphable Models Regression

This paper introduces a novel camera model that extends orthographic projection with a shrinkage parameter to effectively capture perspective distortion in close-up monocular 3D Morphable Model regression, enabling stable and accurate fitting for head-mounted camera footage.

The Gist

Imagine you are trying to build a perfect clay sculpture of a friend's face based on a single photo. This is what computer scientists call 3D Morphable Model (3DMM) regression. They use math to turn a flat 2D picture into a 3D digital head.

For a long time, these digital sculptors have used a very simple rule of thumb: "Everything is the same size, no matter how far away it is."

In the paper you shared, Toby Chong and Ryota Nakajima from Toei Company say, "Hey, that rule doesn't work for close-up photos!" Here is the story of their fix, explained simply.

The Problem: The "Floating Jaw" and the "Tiny Nose"

Imagine you take a selfie with your phone held very close to your face. Because of perspective, your nose looks huge, and your ears look small and far away. This is how our eyes and real cameras work.

However, most computer vision software uses a "flat" projection (called Orthogonal Projection). It's like looking at your face through a window where everything is flattened.

• The Result: When the computer tries to recreate a close-up selfie, it gets confused. It thinks, "If the nose is big in the photo, it must be a big nose." But since the software assumes the face is flat, it shrinks the nose to fit the math.
• The Glitch: The final 3D model ends up with a tiny nose and a jawline that looks like it's floating in mid-air. It also creates a weird "Expanding Brain" effect, where the top of the head looks like it's bulging outward.

The Solution: The "Magic Shrinkage Knob"

The authors realized they didn't need to throw away the old, stable math. Instead, they invented a new camera model that acts like a "magic knob."

1. The Old Way: The computer draws the face flat.
2. The New Way: They added a learnable parameter (let's call it $\rho$ or "Rho") that acts like a perspective dial.
   • Turn the dial to 0: The face looks flat (like the old way).
   • Turn the dial to 5: The face starts to warp, making the nose look bigger and the ears smaller, just like a real close-up photo.

Think of it like a lens filter on a camera. The software learns to twist this "lens" just enough to match the distortion in the photo, without breaking the rest of the math.

The "Head-Mounted Camera" Dataset

To teach their computer this new trick, they needed photos that showed extreme distortion. They couldn't just use standard celebrity photos (which are usually taken from far away).

So, they created a special dataset called HMC1M.

• The Setup: They strapped cameras to the heads of 200 professional actors.
• The Result: These cameras were only 15–30 cm (6–12 inches) away from the actors' faces. This created the "extreme close-up" look where the nose is huge and the perspective is wild.
• The Training: They took an existing AI model (trained on flat photos) and "fine-tuned" it using these crazy close-up photos. The AI learned to turn that "Magic Shrinkage Knob" automatically.

Why This Matters

This isn't just about making better 3D models; it's about fixing the "Uncanny Valley."

• Before: If you tried to put a 3D face on a video game character or a movie actor using a head-mounted camera, the face would look weird, with a tiny nose and a floating chin.
• After: With this new "Shrinkage Knob," the computer understands that close-up = distortion. It can now recreate a realistic 3D face from a selfie or a camera strapped to a helmet, making the nose look big and the jaw look grounded.

The Bottom Line

The authors didn't reinvent the wheel; they just added a suspension system to it.

They took a method that was great for driving on straight highways (standard photos) and added a suspension that allows it to handle bumpy, off-road terrain (extreme close-ups). Now, whether the camera is far away or right in your face, the 3D model looks real, not like a cartoon with a tiny nose.

Technical Summary

Here is a detailed technical summary of the paper "Revisiting an Old Perspective: Projection for Monocular 3D Morphable Models Regression" by Toby Chong and Ryota Nakajima.

1. Problem Statement

Monocular 3D Morphable Model (3DMM) regression is a critical technique for generating 3D facial geometry from 2D images, widely used in content creation and animation.

• The Limitation of Orthogonal Projection: Most state-of-the-art regression-based methods (e.g., DECA, EMOCA, SMIRK) rely on orthogonal projection to map 3D models to 2D image space. This simplification eliminates the ambiguity between focal length ($f$) and object distance ($t_z$), making training more stable.
• The Consequence: However, orthogonal projection ignores perspective distortion. This leads to significant artifacts in close-up images (such as those from head-mounted cameras or selfies), where the model fails to capture the natural "shrinkage" effect.
  • Specific Artifacts: The nose appears unnaturally small, and the jawline or face contour often bends outward, creating an "expanding brain" effect.
• The Challenge: Directly regressing full perspective parameters (focal length and distance) is difficult because the network struggles to learn the complex inverse relationship between $f$ and $t_z$ without ground truth, often resulting in unstable training or failure to capture distortion.

2. Methodology

The authors propose a Pseudo-Perspective Camera Model that extends existing orthogonal projection methods with a learnable shrinkage parameter ($\rho$).

A. The Pseudo-Perspective Projection Model

Instead of replacing the projection equation entirely, the authors modify the standard orthogonal projection to include a perspective-like term:
$$\begin{pmatrix} u \\ v \end{pmatrix} = \begin{pmatrix} S\frac {v_x}{1+\rho v_z}\\ S\frac {v_y}{1+\rho v_z} \end{pmatrix}$$

• $\rho = 0$: The model behaves exactly as standard orthogonal projection.
• $\rho > 0$: The model introduces a perspective effect where points further in depth ($v_z$) are scaled down more aggressively.
• Relation to Physics: The parameter $\rho$ is mathematically related to focal length ($f$) and distance ($v_z$) by the approximation $\rho \approx \frac{S}{f} - \frac{1}{v_z}$.
• Benefit: This formulation isolates the distortion effect into a single parameter, allowing for smooth interpolation between orthogonal and perspective projection via backpropagation.

B. Fine-Tuning Strategy

To adapt existing models trained with orthogonal projection to this new model, the authors propose a specific fine-tuning pipeline:

1. Architecture Modification: A single linear layer with a sigmoid activation is added to the camera pose encoder ($E_\beta$) to regress the new parameter $\rho$.
2. Initialization: The new layer is initialized with small weights/biases to ensure the initial projection remains compatible with the pre-trained network.
3. Unsupervised Learning with Priors: Since close-up footage is often uncalibrated, the authors use a dataset-specific prior ($\rho_{prior}$).
   • They manually adjust $\rho$ on a sample of the target dataset (e.g., Head-Mounted Camera footage) to find the visual optimum.
   • An L2 loss term ($L_\rho = \lambda_p ||\rho - \rho_{prior}||^2$) is added to the training objective to guide the network toward the correct distortion strength without requiring ground-truth focal lengths.
4. Masking Technique: To address ambiguities where the nose and face contour are most prone to mismatch, the authors modify the training mask. They erode the mask around the face contour and specifically exclude pixels around the nose during the rendering loss calculation, forcing the network to rely on other facial features for geometry estimation.

C. New Dataset: HMC1M

The authors curated a new dataset, HMC1M, containing 1 million images captured with head-mounted cameras (15–30 cm from the subject). This dataset exhibits extreme perspective distortion, serving as a rigorous testbed for the proposed method.

3. Key Contributions

1. Novel Camera Model: Introduction of a differentiable, learnable shrinkage parameter ($\rho$) that extends orthogonal projection to capture perspective distortion while maintaining training stability.
2. Backward-Compatible Fine-Tuning: A technique to convert existing 3DMM regression models (trained on orthogonal projection) into the new pseudo-perspective model using uncalibrated images and dataset-specific priors.
3. HMC1M Dataset: A large-scale dataset of extreme close-up facial images specifically designed to evaluate perspective distortion in 3DMMs.
4. Masking Strategy: A refined masking technique to resolve ambiguities in nose and contour regions during the fine-tuning process.

4. Results and Evaluation

The method was evaluated against state-of-the-art baselines (SMIRK, DECA, MICA) on multiple datasets.

• Quantitative Evaluation:
  • 2D Landmarks: On the HMC1M dataset, the proposed method achieved the lowest reconstruction loss for facial landmarks, significantly outperforming the baseline.
  • 3D Mesh Reconstruction: On the NoW Selfie subset (which contains close-up images), the proposed method achieved a lower reconstruction loss (1.2143) compared to the baseline SMIRK (1.2718). The improvement was less pronounced on neutral/standard datasets, confirming the method's specific utility for close-ups.
• Qualitative Evaluation:
  • Visual Artifacts: The baseline models exhibited the "expanding brain" effect and unnaturally small noses in close-ups. The proposed method successfully corrected these, producing realistic nose sizes and natural jawlines.
  • Perception Study: In a crowdsourced study (Amazon Mechanical Turk), 44.4% of respondents preferred the geometry generated by the proposed method over the baseline (23.4%) and the retrained baseline (32.1%).
• Ablation on Full Perspective: Attempts to regress full focal length ($f$) and distance ($t_z$) directly failed to stabilize or capture distortion effectively, validating the necessity of the simplified $\rho$ parameter approach.

5. Significance

This paper addresses a critical gap in 3D facial reconstruction: the inability of standard regression models to handle close-up footage due to the reliance on orthogonal projection.

• Practical Impact: The method enables high-quality 3D reconstruction for applications involving head-mounted cameras, VR/AR avatars, and selfie-based modeling, where perspective distortion is dominant.
• Efficiency: By introducing a single learnable parameter rather than retraining the entire network architecture from scratch with complex perspective constraints, the approach offers a computationally efficient path to upgrading existing state-of-the-art models.
• Insight: The work demonstrates that while full perspective regression is theoretically ideal, a controlled, parameterized approximation (pseudo-perspective) is often more robust and effective for deep learning-based 3DMM fitting.