💻 computer science

NAE-VC: Neural Arithmetic Encoding as a Learned Replacement for CABAC in Modern Video Codecs

NAE-VC is a modular, learned entropy coding framework that replaces the standard CABAC engine in H.264, H.265, and H.266 codecs with a neural architecture combining context aggregation, Gaussian mixture modeling, and hyperpriors to achieve significant bitrate savings while maintaining full compatibility with existing video coding standards.

Original authors: Reka Sandaruwan Gallena Watthage, Anil Fernando

Published 2026-07-17

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Reka Sandaruwan Gallena Watthage, Anil Fernando

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to send a massive, high-definition movie to a friend across the world. To make this possible without clogging the internet, you have to shrink the file down as much as you can. This process is called video compression. Think of it like packing a suitcase for a trip: you want to fit everything in, but you can't just throw things in randomly; you need a smart system to fold clothes tightly and arrange them so nothing is wasted.

In the world of video, this "smart packing system" has a final, crucial step called entropy coding. This is the very last stage where the computer decides exactly how to write the data into a stream of 1s and 0s. For decades, the industry has used a specific, hand-crafted method called CABAC (Context-based Adaptive Binary Arithmetic Coding) to do this. You can think of CABAC as a very experienced, but slightly rigid, librarian who has memorized a huge rulebook on how to organize books. It's good, but it follows fixed rules and can't easily adapt to weird or new patterns in the data. Meanwhile, a new wave of neural networks (computer systems that learn like brains) has shown they can be much better at predicting patterns, but they usually require replacing the entire video system from scratch, which is like rebuilding the whole library just to change the librarian.

This paper introduces a clever middle ground called NAE-VC. Instead of tearing down the whole library, the researchers built a "neural librarian" that can swap out the old, rule-bound librarian (CABAC) while keeping all the other parts of the video system exactly the same. They tested this new system on three generations of video standards (H.264, H.265, and H.266) and found that it consistently shrinks the file size further than the old method, proving that even the best hand-made rules have room for improvement if you let a learning computer take the wheel.

The Problem: The Rigid Librarian

For years, video codecs (the software that compresses video) have relied on CABAC to do the final packing. CABAC is like a librarian who has memorized a massive, hand-written rulebook. When a new piece of data arrives, the librarian checks a specific index in the book to guess how likely that piece is to appear next. If the data is common, they write it down with a short code; if it's rare, they use a longer code.

The problem is that this rulebook is fixed. It was designed by humans years ago and can't change based on the specific movie being watched. It also breaks complex data down into simple "yes/no" (binary) questions, which loses some of the rich, messy patterns found in real video. It's like trying to describe a complex painting by only asking "is it red?" or "is it blue?" one pixel at a time, rather than understanding the whole brushstroke.

The Solution: A Learning Librarian

The researchers proposed NAE-VC (Neural Arithmetic Encoding for Video Compression). Imagine replacing that rigid librarian with a super-smart, learning AI assistant. This assistant doesn't just look up a rule; it looks at the whole picture.

The AI assistant has three special ways of gathering clues before it decides how to pack the data:

Spatial Context: It looks at the neighbors. Just like you know what's in the next room by looking at the current one, the AI looks at the pixels around the current one to guess what comes next.
Channel Context: It looks at the frequency. Video data has different "layers" of detail (like bass and treble in music). The AI understands how these layers relate to each other.
Temporal Context: It looks at the past. If you are watching a video of a ball moving, the AI knows the ball will likely be in a similar spot in the next frame. It uses this motion to make better guesses.

The AI combines all these clues using a technique called multi-head cross-attention (think of it as having four different experts in a meeting, each focusing on a different type of clue, and then voting on the best prediction). Instead of guessing a simple "yes or no," the AI predicts a Gaussian Mixture Model. In plain English, this means it doesn't just guess one number; it guesses a whole cloud of possibilities, understanding that the data might be clustered in a few different ways. This allows it to pack the data much more tightly.

The Results: Shaving Off Bits

The researchers tested this new "neural librarian" by swapping it into the reference software for three different video standards: H.264, H.265, and H.266. They kept everything else exactly the same so they could see if the new librarian was truly better.

The results were consistent across the board:

H.264 (The older standard): The new system saved about 4.82% of the file size in "All-Intra" mode (where every frame is independent) and 6.15% in "Low-Delay" mode (where frames depend on each other).
H.265 (The middle standard): It saved 3.67% (All-Intra) and 4.93% (Low-Delay).
H.266 (The newest, most advanced standard): Even though this standard already has a very sophisticated system, the new AI still managed to save 2.41% (All-Intra) and 3.28% (Low-Delay).

The paper suggests a clear pattern: the more advanced the original system was, the less room there was for improvement. H.264, being the oldest and simplest, had the most "headroom" for the AI to improve upon. However, the fact that the AI still improved the brand-new H.266 by over 2% is a big deal. It suggests that no matter how well humans design a rulebook, a learning system can find patterns they missed.

What This Means

The study shows that you don't need to throw away the entire video system to get better compression. You can surgically replace just the "packing" part with a smart, learning AI.

The researchers also checked if this made the video look better. They found that the savings weren't just a trick of the math; the video actually looked better to human eyes (measured by metrics like VMAF and MS-SSIM). The AI was better at preserving the important details that make a video look sharp and natural.

One interesting finding is that the new system works best when there is motion (like in a sports game or a movie) because it can use the "temporal context" (looking at the past) to make smarter guesses. In static scenes, the improvement is smaller, but still present.

The Catch

There is a cost to this extra smarts. The new system requires more computer power to run. For the older H.264 standard, the encoding time (the time it takes to compress the video) went up by about 4 times. For the newer, already complex H.266, the time only went up by about 15%. This suggests that this technology is most practical for modern, high-end video systems where the computer is already working hard, or for cloud servers where speed is less critical than saving storage space.

In short, the paper proves that by swapping out the final step of video compression with a learning AI, we can squeeze out more efficiency from our existing video standards, making our videos smaller and clearer without needing to wait for the next generation of video technology to arrive.

Technical Summary: NAE-VC – Neural Arithmetic Encoding as a Learned Replacement for CABAC

1. Problem Statement

Entropy coding represents the final, lossless stage of hybrid video codecs (H.264/AVC, H.265/HEVC, H.266/VVC), directly determining the bitrate required to achieve a specific reconstruction quality. The current standard, Context-based Adaptive Binary Arithmetic Coding (CABAC), relies on three fundamental architectural constraints that limit its probability estimation accuracy:

Binary Probability Modelling: CABAC decomposes multi-valued syntax elements into binary decisions, estimating separate Bernoulli probabilities for each bit. This discards the joint statistical structure and multimodal nature of transform coefficients.
Fixed Context Derivation: The mapping from coding state to context model index is hand-designed and frozen at standardization. It cannot adapt to the specific statistical properties of individual video sequences or local content regions.
Single-Rate Adaptation: Probability updates use a fixed exponential moving average (EMA) rule, which cannot distinguish between stationary contexts (requiring slow adaptation) and non-stationary contexts (requiring rapid adaptation).

While end-to-end neural compression frameworks have demonstrated superior probability modelling using learned hyperpriors, autoregressive contexts, and Gaussian Mixture Models (GMM), these approaches replace the entire codec pipeline. They are not standards-compliant and lack backward compatibility. There is a significant gap in transferring these learned modelling advances to the entropy coding stage of existing standardized codecs without altering the prediction, transform, or filtering tools.

2. Methodology: The NAE-VC Framework

The authors propose NAE-VC (Neural Arithmetic Encoding for Video Compression), a modular framework designed to surgically replace only the CABAC module in existing codecs (JM-19.0, HM-16.25, VTM-17.0) while preserving all other coding tools. The framework consists of three primary components:

A. Context Aggregation Network

To overcome the limited representational capacity of fixed CABAC context indices, NAE-VC employs a learned network that fuses three complementary context sources into a unified conditioning vector ( $D=192$ ):

Spatial Context: Extracted via masked $3\times3$ convolutions over previously coded coefficients within the current block, augmented with features from reconstructed neighboring blocks (left/above) to handle boundaries.
Channel-wise Autoregressive Context: Coefficients are partitioned into four groups and processed sequentially via a Multi-Layer Perceptron (MLP) to capture inter-channel dependencies (e.g., between DC and AC coefficients).
Temporal Context: For inter-coded frames, features are extracted from motion-compensated reference blocks. For intra frames, a learned default embedding is used.
Fusion: These paths are fused using a Multi-Head Cross-Attention mechanism ( $H=4$ heads, $d_k=48$ ), allowing the network to adaptively weight the most informative context source for each syntax element.

B. Neural Probability Estimation (GMM)

Instead of single Bernoulli probabilities, NAE-VC models the distribution of quantized transform coefficients using a Gaussian Mixture Model (GMM) with $K=3$ components.

Prediction Heads: Three parallel linear heads predict the mean ( $\mu$ ), scale ( $\sigma$ , via Softplus), and mixing weights ( $\pi$ , via Softmax) for the GMM components based on the fused context vector.
Discretization: The continuous GMM density is integrated over unit quantization bins to generate a discrete Probability Mass Function (PMF), which is then converted to a Cumulative Distribution Function (CDF) for entropy coding.
Advantage: This approach captures multimodal, heavy-tailed, and asymmetric distributions that CABAC's single-mode models cannot represent.

C. Hierarchical Context Modelling & rANS Encoding

Hyperprior: A lightweight encoder network (1.8M parameters) compresses global coefficient statistics into a latent side information vector ( $\hat{z}$ ), transmitted as side information to provide spatially adaptive initial estimates.
Checkerboard Parallelism: To enable partial parallelism without violating causal dependencies, coefficient positions are split into even and odd sets (checkerboard pattern). Even positions are coded first using hyperprior and temporal context; odd positions are coded second, additionally conditioned on the decoded even positions.
Encoding Engine: The framework utilizes rANS (Range Asymmetric Numeral Systems) with 16-bit CDF precision, replacing the binary arithmetic coder. This ensures near-optimal compression efficiency (within 0.01 bits/symbol of theoretical entropy) and bit-exact synchronization between encoder and decoder.

3. Key Contributions

First Unified Neural Replacement: NAE-VC is presented as the first neural replacement for the transform-coefficient entropy coding pipeline evaluated across three generations of standardized hybrid codecs (H.264, H.265, H.266) under a single architecture.
Multi-Head Cross-Attention Fusion: The introduction of a learned fusion mechanism that adaptively combines spatial, channel-wise, and temporal contexts, surpassing the fixed mapping of CABAC.
Comprehensive Cross-Codec Evaluation: The study provides extensive evaluation across six benchmark datasets (UVG, MCL-JCV, HEVC Classes B–E) and two configurations (All-Intra, Low-Delay P), demonstrating consistent gains across PSNR, MS-SSIM, and VMAF metrics.
Modular Integration Strategy: The work demonstrates that learned entropy coding can be integrated as a "drop-in" replacement for CABAC, isolating the contribution of probability estimation from other coding tools and preserving the standard codec's prediction and transform semantics.

4. Experimental Results

Experiments were conducted on six benchmark datasets using QP values {22, 27, 32, 37}.

BD-Rate Savings (vs. Native CABAC)

NAE-VC consistently achieved bitrate savings across all codecs, with the magnitude of improvement inversely related to the sophistication of the native CABAC implementation:

H.264/AVC: $-4.82\%$ (All-Intra), $-6.15\%$ (Low-Delay P).
H.265/HEVC: $-3.67\%$ (All-Intra), $-4.93\%$ (Low-Delay P).
H.266/VVC: $-2.41\%$ (All-Intra), $-3.28\%$ (Low-Delay P).

Key Observations

Inverse Relationship: The results confirm the hypothesis that as native CABAC becomes more sophisticated (from H.264 to H.266), the "headroom" for neural replacement decreases, yet measurable gains persist even in the highly optimized VVC.
Temporal Context: The Low-Delay P configuration yields larger gains than All-Intra (e.g., an additional $1.33$ percentage points for H.264), validating the contribution of the temporal context path which models cross-frame coefficient correlations unavailable to CABAC.
Perceptual Quality: Gains were consistent across metrics, with VMAF improvements ( $-5.47\%$ for H.264) slightly exceeding PSNR gains, suggesting the GMM better preserves structurally important coefficients.
Component Ablation: The largest individual gain came from replacing fixed tables with neural probability estimation ($-1.18$ pp), followed by GMM modelling ($-0.64$ pp), hyperprior ($-0.56$ pp), and spatial context ($-0.49$ pp).
Cross-Generational Impact: H.265+NAE-VC closes approximately $25\%$ of the compression gap to native H.266, demonstrating that learned entropy coding can provide a meaningful fraction of generational gains without a full codec redesign.

Computational Overhead

Encoding: The overhead is highest for H.264 (approx. $3.9\times$ baseline) due to its fast native encoding, but is more modest for H.266 ( $1.15\times$ ) where the baseline pipeline is already computationally intensive.
Model Size: The total trainable parameters are $8.6$ M, with an ONNX model size of $\approx 34$ MB.
Deployment: The authors note that for real-time H.264/H.265, hardware acceleration or model compression would be necessary, while cloud-based transcoding is a more immediately viable scenario.

5. Significance and Claims

The paper claims that NAE-VC provides concrete evidence that learned entropy coding addresses fundamental architectural limitations (binary modelling, fixed context derivation, single-rate adaptation) that persist across all CABAC variants.

Modularity: The approach offers a path to improve compression efficiency without replacing the entire codec ecosystem, preserving backward compatibility at the coding-tool level (though the bitstream format changes, requiring a matching decoder).
Headroom Quantification: The results quantify the residual entropy coding headroom available even in the most advanced standards (VVC), suggesting that further gains are possible through learned probability modelling.
Practicality: By isolating the entropy stage, the work bridges the gap between the high performance of end-to-end neural codecs and the deployment reality of standardized video coding, offering a "surgical" upgrade path for existing infrastructure.

The authors conclude that while the inverse relationship between CABAC sophistication and neural gains is directionally consistent, it is influenced by other codec-level differences (partitioning, transforms, etc.). Nevertheless, the consistent improvements across all generations validate the potential of modular neural entropy coding as a viable enhancement for future video compression standards.