HyPlaneHead: Rethinking Tri-plane-like Representations in Full-Head Image Synthesis

Technical Summary: HyPlaneHead

HyPlaneHead: Rethinking Tri-plane-like Representations in Full-Head Image Synthesis

1. Problem Statement

The paper addresses fundamental limitations in current 3D-aware Generative Adversarial Networks (GANs) used for photorealistic full-head synthesis. Existing methods primarily rely on two types of representations, both of which suffer from distinct geometric and architectural flaws:

• Tri-plane Representations (e.g., EG3D, PanoHead): These use Cartesian coordinate projections. While efficient and good at capturing symmetry, they suffer from feature entanglement, which leads to mirroring artifacts (e.g., a face appearing on the back of the head) because symmetric 3D points query the same feature.
• Spherical Tri-plane Representations (e.g., SphereHead): These use spherical coordinate systems to mitigate mirroring. However, they introduce non-equal-area warping (mapping a square feature map to a sphere), causing features to be overly dense at the poles and sparse at the equator. This leads to inefficient feature utilization and artifacts in the seam/polar regions.
• Feature Penetration (The "Subtle" Issue): The authors identify a pervasive problem where convolutional channels interfere with one another. Because different feature planes (XY, YZ, etc.) have different spatial meanings but are processed by the same convolutional kernels, features from a "dominant" plane bleed into others, limiting the model's expressiveness.

2. Methodology

To resolve these issues, the authors propose the Hy-plane (Hybrid-plane) representation and several supporting technical innovations:

• Hy-plane Representation: A hybrid approach that combines planar planes (to capture symmetric features efficiently) and spherical planes (to capture anisotropic/asymmetric features like the back of the head).
  • Hy-plane (3+1): Uses three orthogonal planar planes and one spherical plane.
  • Hy-plane (2+2): A more robust variant using two planar planes and two spherical planes with opposing poles to ensure seamless coverage of the entire head.
• Near-Equal-Area Warping: To fix the spherical mapping issue, they employ Lambert Azimuthal Equal-Area (LAEA) projection combined with Elliptical Grid Mapping. This transforms the spherical surface into a square feature map with near-uniform area distribution, maximizing feature utilization and eliminating seam artifacts.
• Unify-Split Strategy: To eliminate inter-channel feature penetration, the authors move away from using separate channels for different planes. Instead, they generate a single-channel unified feature map and then spatially split it into distinct regions for each plane. This ensures that each plane's features are spatially disentangled, preventing interference during convolution.
• Area-Biased Splitting: An optimization where the area of the unified feature map is allocated disproportionately (e.g., giving more area to the spherical plane) to match the complexity and importance of different head regions.

3. Key Contributions

1. Systematic Analysis: The first work to identify and analyze the "feature penetration" issue across convolutional channels in tri-plane-like representations.
2. Hybrid Representation: Introduction of the Hy-plane, which successfully merges the benefits of Cartesian and spherical coordinate systems.
3. Geometric Innovations: Development of a novel warping strategy (LAEA + Elliptical Grid Mapping) that solves the non-equal-area problem in spherical representations.
4. Architectural Innovation: The Unify-Split strategy, which fundamentally prevents feature interference by treating planes as spatial regions rather than separate channels.

4. Results

• Quantitative Performance: HyPlaneHead achieves state-of-the-art (SOTA) performance in full-head image synthesis. It outperforms EG3D, PanoHead, and SphereHead across key metrics, specifically showing lower FID (fidelity) and FID-random (robustness to arbitrary viewpoints).
• Qualitative Superiority: Visual comparisons demonstrate that HyPlaneHead eliminates mirroring artifacts (unlike tri-planes) and avoids the blurry textures or seam artifacts found in spherical tri-planes. It produces high-resolution, view-consistent details in complex areas like hair and the back of the head.
• Inversion Capabilities: In single-view 3D reconstruction tasks (using PTI), HyPlaneHead produces significantly higher-quality and more geometrically accurate results compared to PanoHead and SphereHead.

5. Significance

This work is significant because it provides a more robust and mathematically sound foundation for 3D-aware generative modeling. By rethinking how 3D information is mapped to 2D feature grids, the authors have moved the field closer to achieving seamless, high-fidelity, and view-consistent digital human synthesis. The findings regarding feature penetration offer a new perspective for researchers designing any multi-modal or multi-view neural representation.