💻 computer science

Unified Pix Token And Word Token Generative Language Model

This paper proposes a novel generative language model that unifies pixel and word tokens with specific architectural innovations like color folding and global conditional attention to overcome the limitations of current ViT-based vision encoders in recognizing fine visual details, demonstrating strong performance even with small models and limited data.

Original authors: Haun Leung, ZiNan Wang

Published 2026-05-15

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Haun Leung, ZiNan Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a computer to "see" and "speak" at the same time. Currently, the best computers use a method called ViT (Vision Transformer) to look at pictures. The authors of this paper argue that this current method is like trying to describe a complex painting by only looking at a blurry, summarized sketch. It misses the tiny details, like the license plate number on a car or small text on a sign.

Here is a simple breakdown of their new idea, the Unified Pix Token and Word Token Model, using everyday analogies.

1. The Problem: The "Blurry Sketch" vs. The "Pixel-by-Pixel" Reality

Current Method (ViT/CLIP):
Imagine you have a giant mosaic made of 1 million tiny tiles. The current AI doesn't look at every single tile. Instead, it groups them into big chunks (patches) and asks, "What does this whole chunk look like?" It turns that chunk into a single "word" or summary.

The Flaw: If you change just one tiny tile (like a license plate number from 1 to 6), the summary of the whole chunk changes completely. It's like if you changed one letter in a sentence, and the AI thought the entire paragraph had a totally different meaning. This makes it bad at spotting small details.

The New Method (Pix Token):
The authors say, "Let's stop summarizing. Let's give every single pixel (the tiniest dot of color in an image) its own unique name tag and its own dictionary entry."

The Analogy: Instead of grouping tiles into chunks, we give every single tile in the mosaic its own unique ID card. If you change one tile, only that one ID card changes. The rest of the picture stays exactly the same. This makes the AI much more sensitive to tiny details.

2. The Challenge: Too Many Names to Remember

The Problem:
If you give every pixel its own name, and a pixel can be any of 16.7 million colors, you end up with a dictionary that is impossibly huge. It would take too much computer power to look up every single color.

The Solution: "Color Folding"
The authors propose a clever trick called Color Folding.

The Analogy: Imagine you have a box of 16.7 million different shades of blue. To the human eye, the difference between "Sky Blue" and "Sky Blue + a tiny speck of white" is invisible.
The Fix: The model groups these tiny, indistinguishable shades together. It says, "We don't need 16 million names; let's just use 4,000 names that cover all the colors humans can actually see."
Result: This shrinks the dictionary size massively (like folding a giant map into a pocket size) without making the picture look blurry to the human eye. It saves huge amounts of computing power.

3. The Magic Trick: "Global to Local" Attention

The Problem:
Even with a smaller dictionary, looking at a whole image pixel-by-pixel is still a lot of work. If you try to compare every pixel to every other pixel at once (Global Attention), the computer gets overwhelmed, like trying to listen to 10,000 people talking all at once.

The Solution: "Windowed Attention"
The model breaks the image into small windows (like looking through a small square frame).

Step 1: It looks at a small window of pixels and figures out how they relate to each other.
Step 2: It takes the "summary" of that window and moves to the next one.
Step 3: It combines these window summaries to understand the whole picture.

The Analogy: Instead of trying to understand the whole crowd at a concert at once, you listen to small groups of friends talking, then listen to the next group, and finally piece together what the whole crowd is saying. The authors claim this is a "smart approximation" that works almost as well as listening to everyone at once, but is much faster.

4. The Big Goal: Unifying Words and Pictures

Currently, AI models treat text and images as two different things. They read text with one brain and look at images with a separate "translator" (the ViT encoder).

The New Model:
This model treats pixels exactly the same way it treats words.

The Analogy: Imagine a language where "Apple" is a word, and a specific shade of red is also a word. The AI can read a sentence like "The [Red Pixel] is on the [Green Pixel]" just as easily as it reads "The apple is on the table."
The Benefit: Because it treats pixels like words, it can learn from unsupervised data. This means it can learn by just looking at millions of raw images on the internet without needing humans to write labels like "This is a cat." It's like a child learning to see by just looking at the world, rather than being taught with flashcards.

5. What Did They Actually Do?

The authors built a small version of this model (120 million parameters) and trained it only on images (no text mixed in yet).

The Result: The model learned successfully. The "loss" (a measure of how confused the AI was) went down, meaning it started to understand the patterns of pixels.
The Claim: They believe that if they give this model more computing power and more data, it will get even better, following the same "scaling law" that made text-based AI (like GPT-3) so successful.

Summary

The paper proposes a new way for computers to see:

Stop summarizing images; give every pixel its own unique identity.
Simplify the colors (Color Folding) so the computer doesn't get overwhelmed.
Look in windows instead of all at once to save energy.
Treat pixels like words, allowing the AI to learn from raw images without needing human teachers.

The authors believe this is a better foundation for the future of AI vision than the current methods, though they admit they need more computing power to prove it on a massive scale.

Technical Summary: Unified Pix Token and Word Token Generative Language Model

1. Problem Statement

The authors identify critical limitations in current state-of-the-art open-source multimodal models (e.g., LLaVA, Kimi-2.5, Qwen-2.5-VL) that rely on Vision Transformers (ViT) pre-trained via contrastive methods like CLIP or SigLIP as vision encoders. The paper argues that this architecture suffers from four primary issues:

Loss of Detail: ViT encodes images into feature vectors similar to text, which may fail to represent fine-grained details such as small text or numbers within an image.
Instability (The "Chain Reaction"): Minor changes in an image (e.g., a license plate number changing from 1 to 6) cause all floating-point numbers in the encoded features to change, leading to unstable representations.
"Fake" Token Embeddings: ViT generates token embeddings by patching pixels. These embeddings lack truly adjustable parameters, limiting their expressive power compared to word tokens in Large Language Models (LLMs).
Supervised Training Limitations: CLIP and SigLIP rely on supervised training with limited datasets, leading to potential deficiencies in out-of-distribution scenarios.

2. Methodology

The paper proposes a Unified Pix Token and Word Token Generative Language Model that treats image pixels as first-class tokens, similar to words, eliminating the need for a separate ViT encoder. The methodology consists of three core components:

2.1 Pix Token Embedding

Instead of patching, the model assigns a unique, learnable token embedding to every single pixel.

Token Calculation: A pixel with RGB values $(r, g, b)$ is mapped to a unique token value using the formula: $TokenValue = r + g \times L + b \times L^2$ , where $L$ is the color channel length (default 256).
Embedding Space: This creates a vocabulary of $256^3$ (approx. 16.7 million) tokens. Each token has its own independent, trainable embedding vector of dimension $d$ , allowing for adjustable parameters similar to word tokens.
Lossless Preprocessing: Unlike traditional methods that normalize RGB values to floating-point numbers (introducing truncation errors), this method uses integer-based tokenization, ensuring lossless data preprocessing.

2.2 Color Folding

To address the computational infeasibility of a vocabulary size of 16.7 million, the authors introduce Color Folding.

Concept: A "Folding Factor" ( $f$ ) reduces the color channel length from $L$ to $L/f$ . The new color value becomes $(R//f, G//f, B//f)$ .
Rationale: The authors argue that human eyes cannot distinguish minute color differences (e.g., changes of 8 units in an 8-bit channel). Reducing the color space (e.g., using a folding factor of 16) drastically reduces the token vocabulary size (to 4,096 for $f=16$ ) and computational load while maintaining visual fidelity.

2.3 Unified Model Architecture

The model integrates word tokens and pix tokens into a single generative sequence.

Global-to-Local Window Approximation: To avoid the $O(N^2)$ complexity of global self-attention on long pixel sequences, the model converts the global pixel sequence into a batch of local windows.
Local Window Attention: Within each window, a masked multi-head self-attention is performed.
Global Conditional Attention Approximation: The model extracts the last token embedding from each local window. These embeddings are then combined with word token embeddings and passed through subsequent transformer decoders. The authors claim this approximates global conditional attention, as the probability of the output $Y$ given the input is approximated by the probability given the aggregated local window representations.
Prediction Strategy: The model predicts the next token, which can be either a word token or the first pixel token of the next local window.

3. Key Contributions

True Per-Pixel Tokenization: The proposal of "Pix Token Embeddings" where every pixel has an independent, trainable embedding, contrasting with the "fake" patch-based embeddings of ViT.
Color Folding: A novel technique to reduce the computational complexity of pixel tokenization by quantizing color space based on human visual perception limits, making unsupervised pixel-level training feasible.
Unified Architecture: A generative model that unifies text and image tokens into a single sequence, enabling the model to perform self-attention and token prediction on both modalities simultaneously.
Image Unsupervised Pretraining: A framework enabling models to learn patterns directly from massive image data without supervised labels, mirroring the success of GPT-3's text pretraining.

4. Experimental Results

The authors conducted experiments focusing on image unsupervised pretraining due to limited computing resources.

Setup: A small model (120 million parameters) was trained on 100,000 samples from the llava-cc3m-pretrain-595k dataset (approx. 5 billion pixel tokens). The image size was 224x224 with a color folding factor of 16.
Findings:
- The training and validation loss curves converged similarly to GPT-3's text pretraining, but with faster convergence relative to the number of pixel tokens processed.
- The model demonstrated the ability to learn patterns of pixel token sequences even with a small parameter count and limited data.
Limitations: The authors explicitly state they could not test large-scale models or mixed text-image pretraining due to resource constraints.

5. Significance and Claims

The paper claims that this architecture represents a fundamental shift in multimodal learning:

Scalability: The authors believe the model conforms to the scaling law; as model parameters and training data increase, performance will continue to improve, potentially surpassing current ViT-based approaches.
Robustness: By isolating pixel changes to specific token embeddings rather than altering an entire feature vector, the model is more robust to minor image perturbations.
Future Potential: The authors posit that this method could replace the CLIP/SigLP pipeline in production, offering a more detailed, stable, and scalable approach to visual understanding. They express hope that with sufficient computing resources, the model's potential could benefit humanity broadly.

The paper concludes that the proposed model successfully unifies pixel and word tokens, enabling lossless preprocessing and unsupervised pretraining, thereby addressing the inherent limitations of current contrastive learning-based vision encoders.