💻 computer science

Neural Texture Compression using Hypernetworks

This paper introduces a hypernetwork-based approach for neural texture compression that eliminates the need for per-material gradient-descent optimization by training a single network to directly output both latent features and decoder weights, achieving quality comparable to existing methods while enabling multi-decoder inference and super-resolution capabilities.

Original authors: Belcour Laurent

Published 2026-06-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Belcour Laurent

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Too Much Stuff to Carry

Imagine you are a video game developer trying to build a realistic world. To make a wooden table look real, you don't just need one picture; you need a whole "suitcase" of textures: a picture of the wood grain, a picture of how rough it is, a picture of how shiny it is, and a picture of how it reflects light.

In the past, these "suitcases" were huge. They took up so much space on a console's hard drive that developers literally couldn't fit the whole game on the machine. To fix this, they used "compression" (like zipping a file) to shrink the images. But standard zipping loses detail, making the wood look blurry or fake.

The Old Solution: The Slow, Personalized Tailor

Recently, scientists invented a smarter way to compress these textures using "Neural Networks" (AI). Instead of just shrinking the image, the AI learns the rules of how that specific wood looks.

However, there was a major catch with the old method: It was a slow, one-on-one process.
Think of it like a master tailor who makes a custom suit for every single piece of furniture in your house.

To make a suit for the table, the tailor spends hours measuring and stitching just for that table.
To make a suit for the chair, they have to start over from scratch and spend hours again.
If you have 1,000 objects, the tailor needs 1,000 separate, long sessions. This is too slow for making games.

The New Solution: The "Magic Recipe Generator" (Hypernetwork)

This paper introduces a new invention called a Hypernetwork.

Instead of a tailor making one suit at a time, imagine a Magic Recipe Generator.

The Input: You hand the generator a photo of a wooden table.
The Magic: The generator instantly spits out two things:
- The Ingredients (Latents): A tiny, compressed list of the wood's "flavor" (the grain, the color, the roughness).
- The Chef (The MLP): A tiny, pre-written recipe card that tells a computer exactly how to cook those ingredients back into a full-size, high-quality image.
The Result: The computer can now "cook" (decode) the texture instantly, right while the game is running, without needing to do any slow measuring or stitching.

The Key Innovation: The authors trained one single Magic Recipe Generator to handle any texture. Whether it's wood, metal, skin, or fabric, the same generator learns to create the specific "ingredients" and "recipe" for that material instantly. It doesn't need to spend hours optimizing each one individually.

How It Works (The Assembly Line)

The paper describes the generator as having three main stations:

The Scanner (t-Encoder): It looks at the whole texture set and creates a "summary token" (a high-level understanding of what the material is).
The Brain (DiT Blocks): This is the heavy lifter. It uses a special type of AI (a Transformer) to mix the summary with the details of the image. It figures out exactly what the "ingredients" and "recipe" need to be.
The Printer (BC1 Conversion): It takes the brain's ideas and prints them out as:
- Latents: Compressed data blocks (like a tiny zip file).
- Weights: The specific numbers for the decoder recipe.

What Else Can It Do?

The paper shows this "Magic Recipe Generator" is very flexible. Because it's so smart, you can tweak it to do other cool tricks without retraining it from scratch:

One Recipe for All: You can force the generator to use the same recipe (decoder) for every single object, only changing the ingredients. This makes the game run even faster because the computer doesn't have to switch recipes constantly.
Super-Resolution (The "Upscaler"): You can feed the generator a blurry, low-resolution image (like a small thumbnail). The generator learns to "guess" the missing details and output a recipe that creates a crisp, high-definition image. It's like giving a chef a blurry photo of a cake and having them bake a perfect, detailed version of it based on their knowledge of what cakes look like.

The Results

The authors tested this on thousands of real-world textures (wood, metal, fabric).

Quality: The images it produces look almost identical to the original high-quality photos. You can still see tiny scratches and wood grain.
Speed: It is much faster to set up than the old "tailor" method because it doesn't need to optimize each material individually.
Generalization: Even when they showed it a texture it had never seen before (like a weird alien material), it still did a great job compressing and decoding it.

Summary

In short, this paper replaces a slow, custom-made process with a fast, universal "AI machine." This machine can look at any material, instantly figure out how to shrink it down to a tiny size, and write a recipe to make it look perfect again. This saves massive amounts of storage space and makes it possible to put more realistic details into video games without slowing them down.

Technical Summary: Neural Texture Compression using Hypernetworks

Problem Statement

Physically Based Rendering (PBR) relies on multiple texture maps (e.g., diffuse, normal, roughness) to simulate realistic material interactions. These maps consume significant storage and bandwidth, particularly in real-time applications like video games. While recent neural texture compression (NTC) methods have demonstrated the ability to learn compact, per-material representations using Multi-Layer Perceptrons (MLPs) and latent textures, they suffer from a critical bottleneck: optimization time. Existing approaches require a separate, computationally expensive gradient-descent optimization process for every individual material to find the optimal latent features and MLP weights. This per-material optimization is slow and hinders scalability.

Methodology

The paper proposes a novel approach that replaces per-material optimization with a single, trained hypernetwork. Instead of optimizing parameters for each texture set individually, the hypernetwork learns to map any input texture set directly to its compressed representation.

Core Architecture

The system consists of a monolithic hypernetwork that takes a normalized texture set $T$ as input and outputs two components:

Latent Features ( $l_0, ..., l_N$ ): A set of block-compressed textures (specifically using the BCF1 variant, which utilizes BC1 compression).
Decoder Weights ($wb$): The weights and biases for a small MLP decoder.

The reconstruction is performed by the formula:
$\text{mlp}([l(i,j)_0, ..., l(i,j)_N]; wb) \simeq T_{i,j}$

Network Design

The authors utilize a Diffusion Transformer (DiT) architecture with cross-attention mechanisms, structured into three main blocks:

t-Encoder: A mixed convolutional and attention module that downscales the input texture set and extracts a set of global tokens ( $t$ ). These tokens represent the global characteristics of the material.
DiT Blocks: The core processing unit applies cross-attention between image tokens (derived from $4\times4$ texel blocks of the input) and the latent tokens ( $t$ ). This allows the network to correlate local texture details with global material properties.
BC1 Conversion Module: The final stage decodes the tokens into the specific outputs:
- A linear layer converts latent tokens into the MLP weights ($wb$).
- A specialized module converts image tokens into BC1 latent textures ( $l_i$ ). This involves reshaping tokens, applying convolutions to generate 22 channels (representing RGB endpoints and blending parameters), and applying sigmoid functions with offsets to prevent endpoint collapse during quantization.

Training Strategy

The hypernetwork is trained on a dataset of diverse texture sets (from Polyhaven and Sketchfab) using a blend of $L_1$ norm and DISTS (a perceptual loss metric). The training objective minimizes the difference between the original texture set and the set reconstructed by the MLP using the hypernetwork's predicted weights and latents. The model is trained to generalize across different materials without requiring per-material gradient descent.

Key Contributions

Hypernetwork for NTC: The introduction of a single hypernetwork that outputs both latent features and decoder weights for any given material, eliminating the need for per-material optimization.
Quality and Efficiency: The method achieves compression quality comparable to current reference neural texture compressors (which rely on gradient descent) but significantly reduces the time required to generate compressed representations.
Versatility and Repurposing: The paper demonstrates that the trained hypernetwork can be adapted for tasks beyond standard compression:
- Shared MLPs: The network can be fine-tuned to output a single shared MLP for all texture sets, with only the latent textures varying. This simplifies runtime execution by removing the need for tile-based classification to homogenize workgroups.
- Super-Resolution: By training the network with downscaled inputs ( $T^l$ ) and full-resolution targets ( $T^h$ ), the hypernetwork learns to infer high-frequency details, effectively performing super-resolution for SVBRDFs even when high-resolution ground truth is unavailable during inference.

Results

Qualitative: Visual comparisons show that the decompressed outputs preserve high-frequency details (such as scratches and wood grain) and are visually indistinguishable from the input across various scenarios, including materials seen during training, materials in the test set, and materials entirely outside the training dataset. This confirms the model's strong generalization capabilities.
Quantitative/Performance: The authors report that their unquantized model weights occupy 294MB on disk. The approach successfully compresses $4096 \times 4096$ texture sets into a neural representation (BCF1 format) that can be decoded in real-time.
Generalization: The system produces high-quality results for texture sets not present in the training data, validating that the hypernetwork learns a generalizable mapping rather than overfitting to specific assets.

Significance

The paper claims that this approach represents a significant step forward in making neural texture compression practical for real-time applications. By shifting the computational burden from a per-material optimization phase (which is slow and non-scalable) to a one-time training of a hypernetwork, the method enables the rapid generation of compressed neural materials. Furthermore, the ability to repurpose the same architecture for super-resolution and shared-decoder scenarios highlights the flexibility of the hypernetwork approach in addressing diverse challenges in material representation and rendering.