💻 computer science

Hierarchical Image Tokenization for Multi-Scale Image Super Resolution

This paper proposes a novel multi-scale Image Super Resolution method that combines Hierarchical Image Tokenization (HIT) and Direct Preference Optimization (DPO) within a Visual Auto-Regressive framework to achieve state-of-the-art performance with a compact 300M-parameter model, eliminating the need for external training data while enabling flexible, single-pass multi-scale output generation.

Original authors: Isma Hadji, Enrique Sanchez, Adrian Bulat, Brais Martinez, Georgios Tzimiropoulos

Published 2026-05-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Isma Hadji, Enrique Sanchez, Adrian Bulat, Brais Martinez, Georgios Tzimiropoulos

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a blurry, low-resolution photo of a cat, and you want to turn it into a crystal-clear, high-definition masterpiece. This is the job of Image Super-Resolution (ISR).

For a long time, computers did this by either guessing the missing details (which sometimes created fake features, like a cat with six eyes) or by slowly "denoising" the image step-by-step, which was very slow.

Recently, a new method called Visual Auto-Regressive (VAR) models became popular. Think of these models like a painter who doesn't paint the whole canvas at once. Instead, they paint the image in layers: first a rough sketch, then a medium-detail version, and finally the high-definition masterpiece. Each layer builds upon the previous one.

However, the paper you shared points out two big problems with how these "painters" were currently working, and introduces a clever new way to fix them.

The Two Problems with the Old Way

1. The "All-or-Nothing" Ladder
Imagine the painter has a ladder with rungs representing different sizes: a small sketch, a medium sketch, and the final big painting.

The Problem: In previous methods, the ladder was broken. You could only climb to the very top rung (the final 4x size). If you tried to stop at the middle rung (a 2x size), the image would look like garbage or a distorted mess. The "rungs" in between didn't actually exist as valid images; they were just math steps needed to get to the top.
The Paper's Fix (Hierarchical Image Tokenization): The authors built a real, sturdy ladder. They taught the model to paint the small sketch, then the medium sketch, and then the big one, ensuring that every single step looks like a real, coherent image. Now, you can stop the process at any size (1x, 2x, or 4x) and get a perfect picture. They call this Hierarchical Image Tokenization (HIT).

2. The Need for a Giant Library

The Problem: To get good results, previous models needed to be massive (like a giant encyclopedia with billions of pages) or required a huge library of specially labeled "good vs. bad" examples to learn what a sharp image looks like.
The Paper's Fix (DPO Regularization): The authors introduced a new "coach" for the model called Direct Preference Optimization (DPO). Instead of needing a massive library of examples, this coach simply tells the model: "Hey, the high-resolution version is better than the blurry one. Make sure you prefer the sharp details."
- This allows a much smaller model (only 310 million parameters, compared to the 1 billion used by others) to learn effectively using standard, everyday training data.

The Result: A Smarter, Smaller Painter

By combining these two ideas, the authors created a new system that:

Works at any size: It can take a small image and make it 2x bigger, or 4x bigger, or even 8x bigger, all in one go, without the image falling apart at the intermediate steps.
Is efficient: It is much smaller and faster than its competitors. While other models are like heavy, slow-moving tanks, this one is a nimble sports car that still wins the race.
Needs less data: It doesn't need secret, massive datasets to perform well; it works great with standard public data.

The Analogy in a Nutshell

Think of the old methods as trying to build a skyscraper by only being allowed to lay the foundation and then instantly jump to the roof. If you tried to stop halfway up, you'd just have a pile of rubble.

This paper introduces a method where you build the first floor, then the second, then the third. Each floor is a complete, usable apartment. You can stop at the 2nd floor and have a nice place to live, or go all the way to the top. And they did it by hiring a smaller, smarter foreman who knows exactly which floor plans to prefer, without needing a library of blueprints the size of a city.

In short: They made image upscaling flexible (working at any size), efficient (using a smaller model), and smarter (using a new training trick), all while keeping the quality top-tier.

Technical Summary: Hierarchical Image Tokenization for Multi-Scale Image Super Resolution

Problem Statement

Image Super-Resolution (ISR) aims to generate high-resolution (HR) images from low-resolution (LR) inputs. While recent Visual Auto-Regressive (VAR) models have shown promise by aligning well with the "next-scale prediction" paradigm, their application to ISR faces two critical limitations:

Inability to Generate Intermediate Scales: Existing VAR-based ISR methods (e.g., VARSR) rely on Residual Quantization (RQ) where residuals are accumulated to reconstruct a final, fixed-resolution image. The intermediate residuals do not guarantee semantically meaningful reconstructions at lower scales. Consequently, these models cannot produce valid intermediate outputs (e.g., a 2× upsampled image) during a single forward pass; they are restricted to a predefined, fixed scale factor.
Dependency on Large Resources: Current state-of-the-art approaches often require massive backbone models (e.g., 1B+ parameters) or extensive, carefully annotated datasets (including negative samples for classifier-free guidance) to achieve competitive performance.

Methodology

The authors propose a novel framework that integrates two key components into the VAR training pipeline to address the aforementioned shortcomings: Hierarchical Image Tokenization (HIT) and Direct Preference Optimization (DPO) regularization.

1. Hierarchical Image Tokenization (HIT)

Standard RQVAE tokenizes an image by decomposing latent features into a sequence of residuals, but these residuals are not inherently aligned with specific image scales. The proposed HIT approach modifies this process:

Sequential Encoding: Instead of tokenizing the full resolution and then downsampling, the input image is progressively downsampled to target scales (e.g., 1×, 2×, 4×).
Token Overlap: The tokenization is applied sequentially to these downsampled versions. Crucially, tokens generated for lower scales are reused and shared when computing residuals for higher scales.
Fine-tuning: The vocabulary and decoder of a pre-trained RQVAE are fine-tuned using a scale-specific decoder and reconstruction losses for each target scale. This ensures the vocabulary is semantically consistent across different resolutions.
Result: This creates a "strong inductive bias" where the model learns that early tokens represent coarse semantic structures valid at multiple scales, allowing the decoder to reconstruct valid images at intermediate resolutions (e.g., 128×128, 256×256, 512×512) from the same token sequence.

2. DPO Regularization

To improve image quality without relying on external reward models or large-scale negative datasets, the authors introduce a regularization term based on Direct Preference Optimization (DPO):

Mechanism: The model is trained to maximize the log-likelihood of the ground-truth HR tokens while simultaneously penalizing the likelihood of tokens that resemble a simple bilinear upsampling of the LR input.
Formulation: The loss function is defined as $\mathcal{L}_{DPO} = - \log \sigma \left ( \beta \log \frac {p(z_{HR})}{p(z_{LR})} \right )$ , where $p(z_{HR})$ and $p(z_{LR})$ are the probabilities of the HR and LR token sequences, respectively.
Advantage: This approach leverages the model's own capacity to distinguish between HR and LR sequences, eliminating the need for external VLMs or negative sample collection.

Architecture and Training

Model Size: The authors utilize a relatively small transformer (310M parameters, 16 blocks) compared to the 1B+ parameter models used in prior works.
Training Data: The model is trained on standard ISR datasets (DIV2K, DIV8K, Flickr2k, OST, and a subset of FFHQ) without proprietary or massive external corpora.
Conditioning: Unlike VARSR which uses ControlNet, this method directly uses RQVAE encoder features of the upsampled LR image as conditioning signals.

Key Contributions

First Multi-Scale VAR-based ISR: The paper introduces the first VAR-based approach capable of decoding intermediate scales to the image space while maintaining semantic consistency across scales, enabling multi-scale output in a single forward pass.
Strong Inductive Bias via HIT: The proposed Hierarchical Image Tokenization acts as a strong inductive bias, allowing a small 310M parameter model to achieve state-of-the-art results without the need for large backbones.
Simplified Training via DPO: The introduction of a DPO-based regularization term simplifies the training process by removing the dependency on large-scale annotated negative samples or external reward models, yet yields superior sharpness and quality.

Experimental Results

The proposed method (H-VAR) was evaluated on synthetic (DIV2K-Val) and real-world (RealSR, DRealSR) datasets against GAN-based, diffusion-based, and AR-based baselines.

Multi-Scale Capability: Unlike baselines (VARSR, Baseline) which fail to produce coherent images at intermediate scales (e.g., 256×256), H-VAR successfully reconstructs valid images at 1×, 2×, and 4× scales from a single pass.
Performance:
- On RealSR, H-VAR achieves a PSNR of 25.55 and LPIPS of 0.256, outperforming VARSR (PSNR 24.61, LPIPS 0.350) and other diffusion/GAN methods.
- On DRealSR, H-VAR achieves a PSNR of 28.73, surpassing VARSR (28.16) and StableSR (27.87).
Efficiency: The 310M parameter model significantly outperforms the 1B parameter VARSR model while requiring less computational resources (0.25s inference time vs. 0.93s for VARSR on DIV2K-Val).
Ablation: Removing the HIT component results in a significant drop in performance, confirming its role as a critical inductive bias. Similarly, removing DPO regularization leads to blurrier results and lower PSNR/LPIPS scores.

Significance and Claims

The paper claims to advance the paradigm of applying VAR to ISR by solving the "fixed-scale" limitation inherent in standard RQVAE formulations. By introducing Hierarchical Image Tokenization, the authors demonstrate that it is possible to achieve state-of-the-art super-resolution results with a compact model (310M params) trained on standard datasets, without the need for massive proprietary data or complex external guidance mechanisms. The work highlights that the structure of the tokenization itself (HIT) is a powerful driver of performance, acting as a strong inductive bias that guides the model toward semantically meaningful multi-scale representations.