💻 computer science

Affine-Equivariant Kernel Space Encoding for NeRF Editing

This paper introduces Affine-Equivariant Kernel Space Encoding (EKS), a novel spatial encoding for Neural Radiance Fields that utilizes anisotropic Gaussian kernels and feature distillation to enable localized, deformation-aware scene editing with high-fidelity rendering while maintaining a compact, grid-free representation.

Original authors: Mikołaj Zieliński, Krzysztof Byrski, Tomasz Szczepanik, Dominik Belter, Przemysław Spurek

Published 2026-02-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mikołaj Zieliński, Krzysztof Byrski, Tomasz Szczepanik, Dominik Belter, Przemysław Spurek

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Spaghetti" of 3D Scenes

Imagine you have a digital 3D scene, like a room full of furniture. Traditional AI methods (called NeRFs) are amazing at creating these scenes so they look photorealistic. However, they store the scene like a giant, tangled bowl of spaghetti. Every piece of information is mixed together globally.

If you want to move just one chair in this "spaghetti bowl," the AI gets confused. Because everything is entangled, moving the chair might accidentally warp the table, the floor, or the lighting. To fix it, you usually have to throw away the whole model and retrain it from scratch, which takes a long time.

Other methods tried to fix this by using a grid of dots (like a pixelated map) to hold the scene. But if you stretch or squash the scene, the dots get squished too, and the image breaks, leaving holes or weird distortions.

The Solution: EKS (The "Smart Fog" Approach)

The authors introduce EKS, a new way to store 3D scenes. Instead of a tangled bowl of spaghetti or a rigid grid of dots, imagine the scene is made of smart, invisible fog clouds.

Here is how it works, broken down into simple concepts:

1. The Fog Clouds (Anisotropic Gaussians)

Instead of storing data at specific points, EKS stores it inside "fog clouds" (mathematically called anisotropic Gaussian kernels).

The Analogy: Think of these clouds as soft, stretchy balloons. Each balloon holds a little bit of the scene's color and texture.
The Magic: These balloons aren't just round; they can be stretched, squashed, or rotated (like a rugby ball vs. a beach ball). This shape is crucial. When you move or deform the scene, the balloons stretch along with it, keeping the information inside them safe and consistent.

2. The "Stretchy" Search (Mahalanobis Distance)

When the computer needs to know what color a specific spot in the scene is, it looks at the nearby fog clouds.

The Old Way: Imagine asking, "Who is the closest person to me?" If you stretch a rubber sheet with people drawn on it, the "closest" person might suddenly jump to a different spot, causing the image to glitch.
The EKS Way: EKS uses a special "stretchy ruler" (called Mahalanobis distance). It asks, "Who is the closest person considering how the rubber sheet is currently stretched?"
The Result: Even if you squash or twist the scene, the computer always finds the right "neighbors" to blend together. This prevents the image from tearing or developing holes.

3. The "Teacher-Student" Trick (Feature Distillation)

Training a 3D model usually requires a massive, high-resolution grid to get all the tiny details (like the grain on a wooden table). But grids are hard to edit.

The Analogy: Imagine a master chef (the Grid) who knows every secret recipe. EKS is a student chef. During training, the student chef tastes the master chef's dishes and learns the recipes, but instead of memorizing the whole kitchen, the student writes the recipes down on a few specific, flexible cards (the Fog Clouds).
The Result: Once training is done, the student throws away the massive kitchen (the grid) and only keeps the cards. Now, you can edit the scene by moving the cards around, and the student still remembers all the high-quality details because they were "distilled" into the cards.

What Can You Do With This?

The paper shows that because the scene is built on these flexible, stretchy fog clouds, you can do things that were previously impossible without retraining:

Physics Simulations: You can drop a virtual object onto a table, and the scene will react realistically. The "fog clouds" stretch and compress just like real soft clay or fabric.
- Example from the paper: Squashing a Lego bulldozer, making fabric fall over a cup, or simulating leaves falling.
Direct Editing: You can grab a part of the scene and move it, rotate it, or stretch it, and the AI updates the image instantly. The lighting and textures stay perfect because the "fog clouds" moved with the object.

Summary

In short, EKS replaces the rigid, breakable grids of old 3D models with stretchy, intelligent fog clouds.

Old way: Like trying to edit a photo by moving individual pixels; it looks blocky and breaks easily.
EKS way: Like editing a sculpture made of soft, magical clay. You can pull, push, and twist it, and the surface remains smooth and perfect because the "clay" knows how to stretch without losing its shape.

This allows for high-quality 3D scenes that are not only beautiful to look at but also fun and easy to play with, deform, and animate.

Technical Summary: Affine-Equivariant Kernel Space Encoding for NeRF Editing

Problem Statement

Neural Radiance Fields (NeRFs) achieve high-fidelity rendering by encoding 3D scenes as continuous functions. However, their latent spaces are typically implicit and globally entangled, making localized editing and physically grounded manipulation difficult. Existing solutions attempt to mitigate this by introducing explicit control structures (e.g., point-based conditioning, mesh-based control, or primitive-based representations). While these approaches enable some manual editing, they often suffer from limited locality, sensitivity to deformations, and visual artifacts. Specifically, point-based latent representations store features at discrete spatial locations; when these points deform, the local neighborhood changes inconsistently, leading to unstable feature interpolation and visual holes or distortions. Furthermore, standard positional encodings (like multi-resolution hash grids) entangle features globally, causing localized modifications to propagate undesirably.

Methodology: Affine-Equivariant Kernel Space Encoding (EKS)

The authors propose EKS, a spatial encoding mechanism that represents scene features in a continuous kernel space using anisotropic Gaussian kernels.

1. Kernel Space Encoding
Instead of querying latent features at discrete points or grid vertices, EKS aggregates features through a field of learnable anisotropic Gaussian kernels. Each kernel defines a localized region of influence.

Representation: The scene is represented by a set of Gaussian kernels $\{N(\mu_i, \Sigma_i)\}$ , where $\mu_i$ is the mean and $\Sigma_i$ is the full anisotropic covariance. Each kernel is associated with a trainable latent feature vector $v_i$ .
Interpolation: For a query point $x$ , features are interpolated from the $k$ nearest neighbors using a Mahalanobis-distance-based weighting scheme. The weight $w_i$ is calculated as:
$w_i(x, G) = \exp\left(-\frac{1}{2}(x - \mu_i)^\top \Sigma_i^{-1} (x - \mu_i)\right)$
This formulation ensures that the interpolation respects local geometry and remains stable under spatial transformations (affine equivariance), unlike Euclidean distance-based KNN which fails under deformation.

2. Ray-Traced Gaussian Proximity Search (RT-GPS)
To efficiently find the nearest neighbors under the Mahalanobis metric, the authors introduce RT-GPS.

The search is restricted to Gaussians whose confidence ellipsoids (defined by a quantile parameter) contain the query point.
This is approximated by tracing rays from the query point and checking intersections with circumscribed stretched icosahedra representing the ellipsoids.
A sorted hit buffer maintains the top $k$ candidates based on squared Mahalanobis distance.

3. Hash Grid Feature Distillation
To retain the high-frequency detail of multi-resolution hash grids while achieving a grid-free representation at inference, EKS employs a training-time feature distillation mechanism.

During training, both hash-grid parameters and Gaussian positions are optimized jointly.
Gaussian features are sampled from the hash-grid encoding at the kernel centers.
At inference, the hash grid is discarded. The Gaussians retain their learned feature vectors, which are fixed. This allows for direct manipulation of Gaussian positions, rotations, and scales without retraining the network.

4. View-Direction Restoration
To handle view-dependent effects after deformation, the method leverages the principal axes of the Gaussian kernels as local coordinate systems. Using the Kabsch algorithm, the system tracks spatial transformations of these axes to restore view-dependent features as if no deformation had occurred, without increasing the representation size significantly.

5. Editing and Pruning

Editing: Gaussians can be exported as tetrahedra (where orthogonal arms correspond to principal axes) for manipulation via standard mesh deformation tools or physics engines. The modified Gaussians are converted back to parameters for the kernel space encoding.
Pruning/Densification: The method adopts strategies to regulate the number of Gaussians. New Gaussians are initialized by sampling from the hash-grid encoding (rather than shading information) to ensure alignment with the latent field. Unused Gaussians are pruned based on their selection frequency in RT-GPS.

Key Contributions

Affine-Equivariant Encoding: A novel positional encoding that uses anisotropic Gaussian kernels and Mahalanobis-distance interpolation to maintain stable feature structures under spatial transformations, addressing the instability of point-based methods.
Grid-Free Inference: A feature distillation mechanism that transfers multi-resolution details from hash grids into a compact, fully grid-free kernel representation, enabling high-fidelity rendering without the computational overhead of grid lookups during inference.
Direct Editability: A framework that allows intuitive, localized scene editing and physics-based manipulation (rigid body, soft body, cloth) directly via Gaussian kernel parameters without retraining the neural network.
RT-GPS: An efficient algorithm for nearest-neighbor search in anisotropic kernel space using ray tracing and ellipsoid intersection tests.

Results

The authors evaluate EKS on the NeRF-Synthetic dataset, Mip-NeRF 360, and custom deformable assets.

Static Reconstruction: EKS achieves reconstruction quality (PSNR) comparable to state-of-the-art non-editable methods (e.g., Instant-NGP) and competitive with other editable methods.
Editing Performance: On a benchmark for edited scenes, EKS outperforms prior state-of-the-art methods (including Point-NeRF, NeuralEditor, and GaMeS) in PSNR, SSIM, and LPIPS metrics.
Qualitative Analysis: EKS produces smoother surfaces and preserves fine details better than point-based methods, which often exhibit granularity and holes. It avoids the "bleeding" artifacts seen in Gaussian Splatting-based methods where primitives become excessively scaled.
Physics Simulations: The method successfully integrates with physics engines to simulate falling leaves, soft-body deformation (squishing a Lego dozer), and cloth simulation, maintaining high rendering fidelity during dynamic interactions.
Ablation Study: Removing view-direction restoration causes the largest performance drop in edited scenes. Using Euclidean KNN instead of RT-GPS reintroduces artifacts similar to point-based baselines. Removing feature distillation leads to visible holes and floating structures, though the quantitative PSNR impact is limited due to the sparsity of these artifacts.

Significance

The paper claims that EKS provides a promising foundation for physically interactive and editable neural scene representations. By decoupling the latent space from rigid grid structures and point-based discretization, EKS addresses the fundamental limitation of transformation-aware encoding in NeRFs. The method enables stable, localized, and deformation-aware editing while maintaining the high reconstruction quality of modern NeRFs, facilitating applications in robotics, virtual environments, and content creation that require both high fidelity and intuitive manipulation.