GaINeR: Geometry-Aware Implicit Network Representation

1. Problem Statement

Implicit Neural Representations (INRs) have revolutionized continuous signal modeling (images, 3D shapes) by mapping coordinates to output values via neural networks, offering resolution independence and high compression. However, conventional INRs (e.g., SIREN, WIRE, FINER) suffer from two critical limitations:

1. Lack of Explicit Geometric Structure: They treat the image as a global function, making local editing difficult and preventing integration with physical simulation engines.
2. Discrete vs. Continuous Trade-off: Recent attempts to add editability using Gaussian Splatting (e.g., MiRaGe, GaussianImage) rely on discrete primitives. While editable, these discrete representations often produce artifacts (spiky boundaries, transparency issues) when subjected to significant geometric deformations or physics simulations, failing to maintain true continuous coherence.

The core challenge is to create a representation that retains the continuous, high-fidelity reconstruction of INRs while possessing the explicit, manipulable geometric structure of Gaussian-based models.

2. Methodology: GaINeR Framework

GaINeR (Geometry-Aware Implicit Network Representation) proposes a hybrid architecture that combines trainable Gaussian distributions with a neural network-based INR decoder.

Core Architecture

• Learnable Gaussian Components: Instead of a fixed grid, the image is represented by a set of $N$ learnable Gaussian components. Each component $i$ is parameterized by:
  • Mean position $\mu_i \in \mathbb{R}^2$ (or $\mathbb{R}^3$ for 3D).
  • Covariance matrix $\Sigma_i$ (defining scale and orientation).
  • A learnable feature embedding vector $e_i$.
  • Enhancement: To capture fine details, embeddings are derived from a multi-resolution hashgrid $H(\mu_i)$ during training, though the hashgrid is discarded at inference, leaving only the optimized Gaussian parameters.
• KNN Aggregation Mechanism: For any query coordinate $\mathbf{x}$, the model does not query the MLP directly. Instead:
  1. It identifies the $K$ nearest Gaussian centers using a radius-limited K-Nearest Neighbors (KNN) search.
  2. It aggregates the feature vectors of these neighbors using Gaussian-weighted averaging. The weight is determined by the Gaussian probability density function $\mathcal{N}_i(\mathbf{x})$.
  3. This produces a locally geometry-aware embedding $\mathbf{e}_{KNN}(\mathbf{x})$.
• MLP Decoder: The aggregated embedding is fed into a Multi-Layer Perceptron (MLP) $f_\theta$ to predict the final RGB color (and optionally density $\sigma$ for 3D).
  $$f_\theta(\mathbf{x}) = \text{MLP}_\theta(\mathbf{e}_{KNN}(\mathbf{x}, \mathcal{G}))$$

Training and Optimization

• Objective: The model minimizes a Smooth L1 loss between the predicted and ground-truth pixel colors.
• Joint Optimization: The system jointly optimizes the Gaussian parameters ($\mu, \Sigma$), the feature embeddings, and the MLP weights.
• Alpha Masking: For RGBA images, training focuses only on non-transparent regions to improve efficiency. A separate mask model is trained to reconstruct the alpha channel during inference.

Editing and Transformation

• Geometric Manipulation: Editing is performed by directly modifying the positions ($\mu_i$) and shapes ($\Sigma_i$) of the Gaussian components.
• Spatial Equivariance: Because the MLP decoder relies on relative aggregated embeddings rather than absolute coordinates, moving Gaussians induces smooth, artifact-free deformations in the output image. This allows for bending, stretching, and rotation without the "pixelation" or "spikiness" seen in discrete Gaussian splatting.
• Physics Integration: The Gaussian centers can be treated as material points in physics engines (e.g., Material Point Method), allowing the image to interact with physical forces (gravity, collisions) while the MLP maintains visual continuity.

2D-to-3D Lifting

The framework naturally extends to 3D by:

1. Lifting Gaussian means into 3D space using monocular depth maps.
2. Modifying the MLP to output both color $c$ and volume density $\sigma$.
3. Using volumetric rendering (NeRF-style) to synthesize novel views.

3. Key Contributions

1. Novel Hybrid Architecture: GaINeR is the first model to successfully combine the continuous flexibility of INRs with the explicit editability of Gaussian embeddings, overcoming the limitations of purely implicit or purely discrete methods.
2. Geometry-Aware Editing: It enables flexible, local image manipulation (2D and 2D-to-3D) that is consistent with physical simulations, preserving structural integrity where other methods fail.
3. Arbitrary Resolution Synthesis: As a fully continuous representation, GaINeR supports seamless super-resolution and upscaling without retraining, avoiding the grid-based artifacts of discrete methods.
4. State-of-the-Art Performance: The method achieves superior reconstruction quality and editability compared to SIREN, WIRE, 3DGS, and MiRaGe.

4. Experimental Results

The authors evaluated GaINeR on the Kodak and DIV2K datasets across reconstruction, editing, and 3D lifting tasks.

• Reconstruction Quality:
  • GaINeR achieved a PSNR of 77.09 on Kodak and 62.20 on DIV2K.
  • This significantly outperforms the strongest baseline, MiRaGe (59.52 on Kodak, 54.54 on DIV2K), and traditional INRs like SIREN and WIRE.
  • MS-SSIM scores were near-perfect (0.9999) across all datasets.
• Editing and Physics:
  • Visual Comparison: Under geometric transformations (bending, cutting, stretching), MiRaGe and GaussianImage exhibited severe artifacts (spiky boundaries, transparency leaks). GaINeR maintained smooth, realistic surfaces.
  • Physics Simulation: In Material Point Method (MPM) simulations involving fluids and granular dynamics, GaINeR preserved high-frequency textures and cohesive surfaces, whereas discrete methods broke down under topological changes.
• Super-Resolution:
  • At extreme upscaling factors ($\times8, \times10, \times12$), GaINeR preserved sharp edges and fine textures. Competing methods (3DGS, MiRaGe) showed visible Gaussian artifacts and over-smoothing.
• 2D-to-3D:
  • Successfully lifted 2D images to 3D, generating consistent depth maps and novel views, demonstrating the framework's dimension-agnostic nature.

5. Significance and Impact

GaINeR represents a significant step forward in interactive image editing and simulation. By bridging the gap between implicit neural fields and explicit geometric primitives, it solves the "editability vs. continuity" dilemma.

• For Computer Vision: It offers a new paradigm for high-fidelity image compression and reconstruction that is inherently editable.
• For Graphics & Simulation: It enables the integration of 2D images into physics engines for dynamic animation without losing visual quality, a capability previously limited to 3D models.
• Future Directions: The work paves the way for continuous, geometry-aware scene representations that can seamlessly transition between 2D perception and 3D reasoning, potentially impacting AR/VR, digital twins, and generative media.

Limitations: The quality of the 3D lifting is currently dependent on the accuracy of off-the-shelf monocular depth estimators; errors in depth prediction can propagate to geometric artifacts in the 3D reconstruction.