🤖 AI

Render, Don't Decode: Weight-Space World Models with Latent Structural Disentanglement

NOVA introduces a novel world modeling framework that represents system states as the weights of coordinate-based implicit neural representations, enabling efficient, decoder-free rendering and zero-shot super-resolution while achieving unsupervised disentanglement of scene structure and motion for controllable video forecasting.

Original authors: Roussel Desmond Nzoyem, Mauro Comi

Published 2026-05-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Roussel Desmond Nzoyem, Mauro Comi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a computer to predict what happens next in a video, like a ball bouncing or a weather pattern shifting. Most current AI models do this by taking a video, compressing it into a tiny, invisible "secret code" (a latent space), and then using a massive, complex machine to decode that code back into a picture.

The authors of this paper argue that this "decode" step is the problem. It's slow, it's hard to understand, and it makes the AI rigid (if you want a higher-resolution picture, you have to retrain the whole thing).

Instead, they propose a new framework called NOVA (Neural Ontology for Visual Abstraction). Their philosophy is simple: Render, Don't Decode.

Here is how NOVA works, using some everyday analogies:

1. The "Recipe" vs. The "Cake"

Most AI models treat a video frame like a cake. They try to memorize the exact arrangement of every crumb (pixel) and then try to bake a new cake that looks exactly the same. This requires a huge oven (decoder) and is very specific to that one cake size.

NOVA treats a video frame like a recipe. Instead of memorizing the pixels, it memorizes the instructions (the weights and biases) needed to bake the cake.

The Analogy: Imagine you have a master baker (the AI). Instead of giving the baker a photo of a cake and asking them to copy it, you give them a specific set of instructions: "Use 2 cups of flour, 1 egg, and bake at 350 degrees."
The Benefit: If you want a tiny cake or a giant cake, you don't need a new baker or a new photo. You just change the size of the pan (the coordinate grid) and ask the baker to follow the same instructions. The "recipe" (the weights) is portable, compact, and can be used to make the cake at any resolution instantly.

2. The Three-Layer Cake of Reality

The paper claims NOVA naturally separates a video into three distinct parts without needing special training tricks. Think of a video scene like a stage play:

The Background (The Set): This is the static part of the room that never changes. NOVA learns this once and stores it as a "Base Recipe." It's like the permanent walls of the theater.
The Foreground (The Actors): This is the moving stuff, like a person walking or a ball bouncing. NOVA treats this as a small "adjustment" to the Base Recipe. It's like telling the baker, "Keep the base recipe, but add a cherry on top."
The Motion (The Script): This is the instruction on how the actors move. It's the direction the ball is thrown or the speed the person walks.

Because NOVA keeps these three things separate (the set, the actor, and the script), you can do something magical: Edit the video without breaking physics.

3. The "Magic Swap" (Disentanglement)

The paper demonstrates that because the "Actor" (content) and the "Script" (motion) are stored separately, you can swap them around freely.

The Experiment: Imagine a video of a red ball rolling across the screen.
The Swap: You can take the "Script" (the rolling motion) from the red ball and apply it to a blue square.
The Result: The blue square will roll exactly like the red ball did, but it will still look like a blue square.
Why it matters: In other AI models, if you try to change the motion, the object often changes shape or color too, or the whole video turns into a blurry mess. NOVA keeps the "identity" of the object safe while letting you change how it moves.

4. Seeing the Invisible (Super-Resolution)

Because NOVA uses "recipes" (mathematical functions) rather than fixed grids of pixels, it can "render" the video at any size.

The Analogy: If you have a digital photo, zooming in usually makes it blocky (pixelated). If you have a vector drawing (like a logo), you can zoom in forever, and it stays crisp.
NOVA's Power: NOVA is like the vector drawing. The paper shows it can take a low-resolution weather map and "render" it at 32 times higher resolution instantly, revealing fine details without the blurry artifacts you get from standard AI upscaling.

5. Why This is a Big Deal

The authors tested this on three very different types of videos:

Moving Digits: Numbers bouncing around.
Physics Collisions: Balls hitting each other (testing if the AI understands real-world physics like momentum).
Weather Maps: Predicting global temperature changes.

They found that NOVA could predict the future of these scenes accurately, edit the content and motion independently, and run on a single standard computer graphics card (a consumer GPU) with about 40 million parameters.

In summary: The paper argues that instead of building a giant, opaque machine to guess what a video looks like, we should build a system that learns the rules for generating the video. By storing the "rules" (weights) directly, the AI becomes smaller, faster, easier to edit, and capable of seeing the world at any level of detail.

Technical Summary: Render, Don't Decode: Weight-Space World Models with Latent Structural Disentanglement

Problem Statement

The prevailing paradigm for training world models and video generation involves encoding raw pixels into opaque, abstract latent spaces and relying on heavy, learned decoders to reconstruct frames. This approach presents three critical limitations:

Computational and Interpretability Bottlenecks: Decoders consume a disproportionate number of parameters (up to 87% in some tokenizers) and introduce uninterpretable parameters, hindering safety-critical applications.
Resolution Rigidity: Changing output resolution requires expensive retraining or post-hoc upsampling, as latent spaces are tethered to specific decoder architectures.
Lack of Structural Disentanglement: Abstract latents lack semantic or geometric interpretations. Separating background, foreground, and motion typically requires auxiliary losses, adversarial objectives, or complex architectural decompositions, which add training complexity without structural guarantees.

Furthermore, while Weight-Space Learning (WSL)—treating neural network weights as data points—has shown promise in time-series modeling, it remains untested on video modalities due to acute stability issues, often requiring aggressive clipping to prevent numerical explosion.

Methodology: The NOVA Framework

The authors introduce NOVA (Neural Ontology for Visual Abstraction), a world model that operates entirely within the weight space of an Implicit Neural Representation (INR). Instead of decoding abstract vectors into pixels, NOVA represents the system state as the weights and biases of a coordinate-based MLP.

Core Architecture

NOVA evolves dynamics in the weight space using a three-level semantic ontology that mirrors object-centric video structure:

Base Weights ( $\bar{z}$ ): A single set of parameters shared across the entire dataset, learned to encode static elements (e.g., the background). This acts as a dataset-level prior and stabilizes training by framing predictions as residuals.
Frame Offsets ( $z_t$ ): Per-frame weight offsets encoding dynamic foreground content (object identity) present in the current frame but not shared globally.
Latent Actions ( $u_t$ ): Inter-frame signals dictating motion transitions.

The architecture consists of:

Encoder: A strided convolutional backbone mapping frames $o_t$ to weight-space offsets $z_t$ .
Inverse Dynamics Model (IDM): Infers latent actions $u_t$ from consecutive encoded states $(z_t, z_{t+1})$ .
Forward Dynamics Model (FDM): Predicts the next state using an additive formulation: $z_{t+1} = A(z_t) + B(u_t)$ $z_{t + 1} = A (z_{t}) + B (u_{t})$ .
- $A$ processes the current state (content/identity).
- $B$ processes the action (motion/dynamics).
- This decoupling is the mechanism enabling disentanglement without auxiliary losses.
Renderer: An analytical, fixed MLP that maps spatial coordinates $(x, y)$ to pixel values using the combined weights $\theta_t = \text{unflat}(\bar{z} + z_t)$ . Crucially, this renderer is not trained; it is a mathematical function.

Training Procedure

Training proceeds in three phases:

Pre-training: The encoder and base latent $\bar{z}$ are trained to minimize per-frame reconstruction loss (MSE + SSIM).
Dynamics Learning: With the encoder and $\bar{z}$ frozen, the FDM and IDM are trained to minimize the latent transition objective $\|z_{t+1} - A(z_t) - B(u_t)\|^2$ .
Generative Control: A Generative Control Model (GCM), typically a causal sequence model (LSTM, GRU, or Transformer), is trained to autoregressively predict actions $u_t$ from $z_t$ and previous actions, enabling autonomous video generation without ground-truth actions.

Key Technical Innovations

Analytical Rendering: By using INRs, the model achieves zero-shot super-resolution. The renderer can query any coordinate grid density at test time. To prevent aliasing at high resolutions, the authors implement a post-training Nyquist-Shannon frequency masking procedure that zeros out Fourier basis frequencies exceeding the training grid's Nyquist limit.
Additive Disentanglement: The separation of $A(z_t)$ and $B(u_t)$ allows for "zero-shot content (or motion) retargeting." Users can swap the content state $z_t$ or the action $u_t$ from an arbitrary sequence to edit the generation process without retraining.

Results

The framework was validated on three datasets: Moving MNIST (deterministic rigid bodies), PhyWorld (rigid-body collisions with OOD generalization), and WeatherBench (global atmospheric dynamics). All experiments were conducted on a single consumer GPU (RTX 4080) with approximately 40M parameters.

Long-Horizon Stability: NOVA maintains structural integrity and object identity over 1,000-step rollouts, whereas baselines like LAPO (pixel-space) collapse into dark figures and standard latent-space models degrade into blurry averages.
Disentanglement and Editing:
- Content Editing: Interpolating $z_t$ towards zero morphs digit identities (e.g., from 5 to 8) while preserving the original motion trajectory.
- Motion Editing: Interpolating $u_t$ towards zero causes objects to decelerate and converge to the center while preserving their visual identity.
- Retargeting: Swapping actions from one sequence to another forces objects to adopt new, physically consistent (or intentionally unphysical) trajectories without altering their shape or color.
Super-Resolution: On WeatherBench, NOVA achieves zero-shot super-resolution (up to $\times32$ ) without hallucinating high-frequency textures or suffering from the aliasing artifacts common in standard INRs, thanks to the frequency masking strategy.
Physical Consistency: In PhyWorld, NOVA preserves physical laws (momentum, kinetic energy) in in-distribution settings and demonstrates robust OOD generalization regarding ball radii and velocities.

Significance and Claims

The paper claims that NOVA demonstrates that world models can be constructed entirely from structured weight-space representations rather than raw pixels.

Interpretability: By separating the system state into background ( $\bar{z}$ ), content ( $z_t$ ), and motion ( $u_t$ ), the model offers a transparent, interpretable alternative to opaque latent spaces. This facilitates auditing and safe interventions in safety-critical scenarios.
Efficiency and Portability: The elimination of the decoder bottleneck and the use of compact weight vectors make the models highly portable and capable of running on consumer hardware.
Resolution Independence: The analytical rendering capability allows for arbitrary resolution querying, a critical feature for domains like extreme weather forecasting where pixel granularity impacts decision-making.
General Applicability: The authors posit that the disentanglement mechanism (additive decomposition in weight space) is applicable to standard latent action models, potentially offering a new paradigm for controllable video generation and immersive virtual experiences.

The authors acknowledge limitations, including the potential for spurious correlations in latent actions (e.g., encoding brightness or vertical position as part of identity) and challenges in OOD generalization for complex physics without explicit differential equation constraints. They emphasize that while NOVA achieves strong controllability, it currently trades off some raw pixel-level reconstruction fidelity compared to supervised spatio-temporal models.