Technical Summary: GeoDualNet

1. Problem Statement

Multi-view video plus depth (MVD) coding is essential for immersive applications like virtual reality and free-viewpoint video, where scenes are captured by multiple calibrated cameras to enable interactive view synthesis. However, the data volume scales linearly with the number of views, necessitating highly efficient compression.

Current solutions face a fundamental limitation: they fail to jointly exploit the projective duality between disparity vectors (DV) and depth maps.

• Traditional Codecs (e.g., 3D-HEVC): Treat DV prediction and depth coding as independent, hand-crafted modules. While they use depth to influence disparity derivation, there is no joint optimization or shared representation, missing the massive mutual information implied by the geometric relationship $d = fB/Z$.
• Learned Codecs (e.g., LMVC): While end-to-end learned multi-view codecs exist, the state-of-the-art (LMVC) avoids explicit disparity estimation and omits depth coding entirely. This "disparity-free" approach forgoes strong geometric priors and cannot support the MVD format required for Depth-Image-Based Rendering (DIBR).

The core problem is the lack of a unified, end-to-end neural framework that jointly learns disparity vector prediction and depth map coding to exploit their geometric redundancy.

2. Methodology: GeoDualNet Architecture

GeoDualNet is proposed as the first end-to-end neural codec to jointly learn DV prediction and depth coding through a unified geometric latent space. The framework adopts a layered coding structure where a base view is coded using a frozen state-of-the-art learned B-frame codec (DCVC-B), while dependent views and all depth maps are processed through five novel, geometrically coupled components:

2.1 Dual-Latent Mutual Refinement (DLMR)

This is the core iterative mechanism where disparity and depth latent representations ($z_d$ and $z_z$) refine each other.

• Mechanism: The encoder produces initial latents, which are then passed through $K$ iterations of cross-refinement. In each step, the disparity latent is updated using information from the depth latent, and vice versa.
• Theoretical Guarantee: The authors prove that under a Lipschitz condition (enforced via spectral normalization), the DLMR converges to a unique geometric-consistency fixed point.
• Convergence: Experiments show that over 95% of the improvement is captured within $K=3$ iterations.

2.2 Epipolar Cross-Attention (ECA)

To couple the latent spaces efficiently, GeoDualNet introduces a geometry-constrained attention mechanism.

• Epipolar Masking: Instead of global attention, the mechanism restricts interactions to projectively valid correspondences along epipolar lines defined by the fundamental matrix. This reduces complexity by approximately $6\times$ compared to unconstrained attention.
• Geometric Positional Encoding (GPE): Standard sinusoidal encodings are replaced with camera-conditioned encodings that inject intrinsic and extrinsic parameters ($\Pi$) directly into the query and key projections.
• Scale-Aware Heads: Attention heads are partitioned to operate on specific disparity/depth scale bands to prevent interference between near and far objects.

2.3 Implicit Geometric Field (IGF)

This component replaces all hand-crafted warping operations (3D warping, DIBR) with a single learnable neural field.

• Function: It maps latent representations between views via a coordinate-conditioned MLP with Feature-wise Linear Modulation (FiLM) layers, conditioned on source and target camera parameters.
• Advantages: It operates in the latent space (providing robustness to occlusions and computational efficiency) and is fully differentiable, avoiding discretization artifacts inherent in pixel-domain warping.

2.4 Depth-Gated Disparity Flow (DGDF)

This module accelerates and constrains disparity prediction using coarse depth priors.

• Causal Chain: It establishes a differentiable chain: $Depth \rightarrow DV \rightarrow Refined Depth$.
• Mechanism: A lightweight depth predictor generates a coarse depth map, which is converted into an adaptive geometric search corridor (disparity range). A gated correlation volume is then computed only within this corridor, reducing computational cost and memory by $4\text{--}10\times$ compared to full correlation volumes.

2.5 Joint Geometric Entropy Model (JGEM)

To maximize compression efficiency, GeoDualNet employs a cross-component conditional entropy model.

• Factorization: The joint entropy of disparity, depth, and texture residuals is factorized conditioned on a shared geometric prior derived from a transformer-fused hyperprior.
• Benefit: By conditioning the depth entropy model on the decoded disparity latent (and vice versa), the codec exploits the conditional mutual information $I(z_d; z_z | \Pi)$, theoretically saving bits proportional to this redundancy.

2.6 Training Strategy

The system is trained end-to-end using a multi-objective loss function that balances rate, texture distortion, depth distortion, synthesis quality, and geometric consistency. A geometric consistency regularizer enforces the projective duality $d = fB/Z$ as a self-supervised anchor. Training follows a three-phase curriculum: independent view warmup, geometric coupling activation, and joint fine-tuning.

3. Key Contributions

1. First Joint DV-Depth Codec: GeoDualNet is the first end-to-end neural codec to jointly optimize disparity vector prediction and depth map coding, explicitly exploiting the projective duality $d = fB/Z$.
2. Novel Architectural Components: The introduction of DLMR (with provable convergence), ECA (with epipolar constraints), IGF (learnable warping), DGDF (gated search), and JGEM (cross-component entropy).
3. Theoretical Foundations: The paper provides theoretical proofs for the convergence of the mutual refinement loop and lower bounds on rate savings derived from conditional mutual information.
4. Comprehensive Evaluation: Extensive testing on the 3D-HEVC Common Test Conditions (CTC) covering texture, depth, and synthesized view quality.

4. Experimental Results

Evaluated on standard 3D-HEVC test sequences (e.g., Poznan Street, GT Fly, Shark) under 3-view and 5-view configurations:

• Texture Coding: GeoDualNet achieves average BD-rate savings of 39.1% (PSNR), 42.5% (MS-SSIM), and 45.8% (VMAF) over 3D-HEVC.
• Depth Coding: It achieves 47.6% average BD-rate savings for depth maps over 3D-HEVC.
• Comparison with LMVC: Compared to the state-of-the-art learned multi-view codec (LMVC), GeoDualNet provides an additional 15.5 percentage points of texture savings while adding native depth coding support (which LMVC lacks).
• Synthesized View Quality: The decoded MVD representation yields a +2.12 dB improvement in synthesized view PSNR and +4.8 VMAF points over 3D-HEVC.
• Ablation Studies:
  • Removing DLMR causes the largest degradation (+14.3% texture BD-rate loss), confirming mutual refinement is the primary driver.
  • Removing the geometric consistency loss ($L_{geo}$) has a disproportionate impact on synthesis quality (+11.2% degradation), highlighting the critical role of geometric coherence for DIBR.
  • The JGEM contributes significantly to depth savings (+12.4% BD-rate loss when removed), validating the exploitation of cross-component mutual information.
• Scalability: Performance gains increase with the number of views (from -41.5% in 3-view to -47.7% in 5-view configurations).
• Complexity: Encoding time is approximately 0.84s per frame on an A100 GPU (342$\times$ faster than 3D-HEVC on CPU), with a moderate 35% overhead compared to LMVC due to the added depth and geometric modules.

5. Significance and Claims

The paper claims that GeoDualNet establishes a new paradigm for MVD compression by demonstrating that explicit geometric inductive biases (epipolar constraints, projective duality, camera-conditioned representations) yield substantial gains over geometry-agnostic approaches, even when those approaches use general-purpose mechanisms like unconstrained attention.

The work asserts that:

• Jointly learning DV and depth is fundamentally more efficient than independent coding, as it exploits the physical redundancy inherent in multi-view geometry.
• The "disparity-free" philosophy of previous learned codecs (like LMVC) is suboptimal for MVD formats requiring depth; explicit geometric modeling is superior when implemented with appropriate learned modules.
• The success of the JGEM proves that multi-stream compression problems benefit significantly from joint entropy modeling when streams share underlying physical structure.

The authors position this work as a foundational step toward unified compression for immersive media, bridging the gap between traditional multi-view coding and emerging volumetric representations, while noting that future work could address uncalibrated cameras, occlusion-aware constraints, and dense light field arrays.