🤖 machine learning

Neural Field Tokenizations with Hierarchy and Spatial Locality Priors

The paper introduces LH-NeF, a feed-forward framework that leverages locality and hierarchy priors to generate general-purpose tokenized representations of continuous signals, achieving significant memory and batch size efficiency while matching or exceeding the performance of existing modality-agnostic and specialized neural field baselines across diverse data types.

Original authors: Alonso Urbano, David W. Romero, Max Zimmer, Sebastian Pokutta

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Alonso Urbano, David W. Romero, Max Zimmer, Sebastian Pokutta

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive library of different types of data: photos, 3D models of chairs, and global weather maps. Traditionally, if you wanted to teach a computer to understand all of these, you'd need a different "translator" for each type. A translator for photos wouldn't work on weather maps, and a translator for 3D chairs would struggle with images.

The paper introduces a new method called LH-NeF (Locality-preserving Hierarchical Neural Fields). Think of this as a universal translator that can handle any kind of data—images, shapes, or weather—using the same set of rules.

Here is how it works, broken down with simple analogies:

1. The Problem: The "Slow and Expensive" Way

Before this paper, the best way to teach a computer to understand these continuous data fields was like trying to learn a new language by memorizing every single sentence one by one.

The Old Method (Meta-Learning): For every single image or 3D shape, the computer had to stop, think, and "optimize" its understanding from scratch. It was like asking a student to re-learn the alphabet every time they wanted to read a new word.
The Cost: This was incredibly slow and required a massive amount of computer memory (RAM). It was like trying to carry the entire library in your backpack just to read one book.

2. The Solution: The "Smart Tokenizer"

The authors propose a new way to look at data. Instead of memorizing every single point, they turn the data into a set of structured "tokens" (like puzzle pieces) that the computer can process instantly.

They use two main "superpowers" to make this work:

A. Locality (The "Neighborhood" Rule)

Imagine you are organizing a huge crowd of people.

The Bad Way: You line them up randomly. The person from the far left of the room is standing next to someone from the far right. They have nothing in common, but they are grouped together. This confuses the computer.
The LH-NeF Way: You use a "locality-preserving" rule. You group people who are standing near each other in the room. If you are looking at a photo, you group pixels that are neighbors. If you are looking at a 3D chair, you group parts of the chair that are physically close.
The Analogy: It's like organizing a library not by random order, but by keeping books about "Cooking" next to other "Cooking" books, and "History" next to "History." This makes it much easier to find what you need.

B. Hierarchy (The "Zoom-Out" Rule)

Imagine looking at a map.

The Bad Way: You only look at the street level. You see every single house, but you can't see the whole city.
The LH-NeF Way: It builds a hierarchy. First, it groups small neighborhoods. Then, it groups those neighborhoods into districts. Then, it groups districts into the whole city.
The Analogy: It's like a set of Russian nesting dolls. You start with the tiny details (the smallest doll), then you step back to see the medium group, and finally the big picture. This helps the computer understand both the fine details (like the texture of a chair leg) and the big structure (the whole chair) at the same time.

3. How It Reads the Data (The "Renderer")

Once the data is turned into these smart, grouped tokens, the computer needs to read it back to create the image or shape.

The Old Way: It had to do complex math for every single point, over and over again.
The New Way: When the computer wants to know what a specific point looks like, it asks: "Which neighborhood (group) is this point in?" It then looks at the nearest few neighborhoods, blends their information together smoothly (like mixing paints), and instantly knows the answer.
The Result: It's like asking a local guide for directions. Instead of checking a map for every single step, the guide knows the whole neighborhood and gives you a smooth, continuous path.

4. Why This Matters (The Results)

The paper claims three major wins:

Speed and Memory: Because they stopped doing the "re-learn from scratch" step, the new method uses 42 times less memory and can process 133 times more data at once than the previous best methods. It's like switching from a bicycle to a high-speed train.
Quality: Despite being faster, it creates images, 3D shapes, and weather maps that are just as good (or better) than the slow, old methods.
Universality: It works on everything. Whether it's a 2D photo, a 3D object, or a global climate map, the same "translator" works. You don't need to build a new engine for every new type of data.

Summary

The paper presents a new way to teach computers to understand continuous data (like images and shapes) by organizing it into neighborhoods and levels of detail (hierarchy). This allows the computer to process data instantly instead of slowly re-learning it every time, saving massive amounts of computer power while maintaining high quality. It's a universal, efficient, and smart way to handle data that was previously too heavy to manage easily.

Technical Summary: Neural Field Tokenizations with Hierarchy and Spatial Locality Priors

Problem Statement

Neural fields parameterize data as continuous functions from coordinates to values (e.g., $f_\theta: [0, 1]^2 \to \mathbb{R}^3$ ), offering a unified framework for representation learning across modalities. However, existing modality-agnostic approaches, such as Functa and its descendants, rely heavily on per-sample meta-learning (e.g., MAML or auto-decoding) to infer latent representations. This paradigm presents three critical limitations:

Scalability Bottleneck: Meta-learning requires storing the full computation graph for inner-loop optimization to compute second-order gradients. This severely limits batch sizes and memory efficiency, hindering scaling to high resolutions or large datasets.
Lack of Structure: Optimization-derived latent spaces often lack explicit structural priors, making learning inefficient and the resulting representations difficult to use for downstream tasks like classification.
Trade-off: There is a fundamental tension between introducing useful structural priors (like locality and hierarchy) and maintaining modality-agnosticism. Most methods that introduce structure (e.g., 2D grids, point clouds) sacrifice the ability to handle arbitrary coordinate domains.

Methodology: LH-NeF

The authors propose LH-NeF (Locality-preserving Hierarchical Neural Fields), a framework that learns general-purpose tokenized representations of continuous signals without compromising modality-agnosticism. The method replaces per-sample optimization with a single forward pass through a learned encoder-decoder architecture.

1. The LH-NeF Tokenizer (Encoder)
The tokenizer builds upon Hierarchical Perceivers (HiP) but modifies them to respect coordinate-space locality:

Locality-Preserving Ordering: Unlike standard HiP, which uses a fixed raster scan (violating spatial locality in coordinate space), LH-NeF reorders input observations $\{(x_i, v_i)\}$ using a locality-preserving permutation $\pi$ . This is derived from space-filling curves (e.g., Morton/Z-order for Euclidean domains, S2 Hilbert curves for spherical data). This ensures that contiguous groups in the token sequence correspond to compact spatial regions in the coordinate domain.
Hierarchical Grouped Attention: The reordered sequence is processed through $L$ blocks of grouped attention. Tokens are progressively merged from fine-grained groups to coarser groups.
Spatial Metadata: The final output is a set of grouped tokens $Y^{(L)}$ accompanied by routing metadata: the centroid $\mu_g$ (Fréchet mean of coordinates in the group) and extent $\lambda_g$ (spatial radius or bounding box). This allows the representation to adapt to the sampling density and geometry of the input.

2. The LH-NeF Renderer (Decoder)
Given the tokenization and a query coordinate $x$ , the renderer reconstructs the field value $f_\theta(x)$ via three steps:

k-Group Soft Routing: The query $x$ is routed to the $k$ nearest groups based on the distance between $x$ and group centroids $\mu_g$ . A learnable Gaussian kernel assigns soft weights $w_g(x)$ to these groups, ensuring continuity and avoiding abrupt switching between groups.
Cross-Attention Aggregation: For each routed group, cross-attention is performed between the query embedding and the group's tokens. The outputs are aggregated via a weighted sum.
Geometry-Conditioned Modulation: The aggregated feature vector is modulated using a FiLM-style mechanism conditioned on the query's relative position $\tilde{x}$ within the nearest group (normalized by the group's extent). This provides invariance to translation and scaling, allowing the network to learn local patterns that generalize across different groups.

Key Contributions

Identification of a Trade-off: The paper identifies the tension between scalability, structural priors, and modality-agnosticism in neural field learning, arguing that hierarchy and locality can be introduced without modality-specific assumptions.
LH-NeF Framework: A novel architecture that produces spatially structured grouped tokenizations. It conditions a renderer that decodes fields at arbitrary coordinates via soft group routing and cross-attention, eliminating the need for per-sample optimization.
Efficiency and Performance: The method replaces the memory-intensive inner loop of meta-learning with a single forward pass, achieving significant gains in training efficiency while matching or exceeding state-of-the-art performance.

Experimental Results

The authors evaluated LH-NeF on images (CIFAR-10, CelebA-HQ, ImageNet), 3D shapes (ShapeNet), and climate fields (ERA5).

Reconstruction: LH-NeF achieves state-of-the-art performance among modality-agnostic methods. For example, on CelebA-HQ (64²), it achieves 44.4 dB PSNR with only 198K parameters, outperforming Functa (38.1 dB, 2.6M params) and ENF (42.2 dB, 522K params).
Downstream Tasks:
- Generation: On CelebA-HQ, LH-NeF achieves a FID of 9.7, outperforming specialized generative neural field methods and modality-agnostic baselines.
- Classification: On ShapeNet16, a lightweight classifier on frozen tokens reaches 96.8% accuracy, surpassing ENF (96.6%) and Functa (90.3%).
- Forecasting: On ERA5, LH-NeF improves over Functa in temperature forecasting, though it trails ENF (which uses symmetry-specific priors).
Efficiency: By removing the meta-learning inner loop, LH-NeF uses ~42× less memory and supports 133× larger batch sizes (266 vs. 2) compared to the strongest modality-agnostic baseline (ENF) on the same hardware.

Significance and Claims

The paper claims that LH-NeF serves as a scalable, modality-agnostic foundational method for neural field representation learning. By leveraging hierarchy and locality as inductive biases, the framework successfully bridges the gap between the efficiency of forward-pass inference and the flexibility of neural fields.

The authors note that while the method is highly effective for uniformly sampled data (images, grids) and adapts well to irregular sampling (point clouds), locality preservation on highly irregular data (e.g., LiDAR) holds only in expectation. They also suggest future work could explore learning which biases generalize across tasks rather than hand-designing them, and extending the conditioning mechanism to exploit information from all encoder levels rather than just the final one.

The work does not claim to solve all representation learning problems but demonstrates that structured tokenization can replace optimization-based latent fitting without sacrificing the generality required for multi-modal applications.