Xray-Visual Models: Scaling Vision models on Industry Scale Data

Xray-Visual is a unified, efficient vision model trained on over 25 billion industry-scale social media data pairs using a novel three-stage pipeline and LLM-enhanced text encoders to achieve state-of-the-art performance in image and video understanding across diverse benchmarks.

The Gist

Imagine you are trying to teach a child how to understand the world. Most schools (academic researchers) teach this child using a small, perfect library of textbooks with clear pictures and neat labels. But the real world is messy, loud, and chaotic.

Xray-Visual is a new kind of "super-student" built by Meta AI. Instead of just reading textbooks, this student spent years scrolling through billions of real posts on Facebook and Instagram. It didn't just look at the photos; it learned to understand the messy captions, the hashtags, and the context of real life.

Here is a simple breakdown of how they built this super-student and why it's a game-changer:

1. The Massive Library (The Data)

Most AI models are trained on a few million images. Xray-Visual was trained on 15 billion image-text pairs and 10 billion video clips.

• The Analogy: Imagine trying to learn a language. Other models read a dictionary and a few novels. Xray-Visual read every single tweet, text message, and comment on the internet for a decade.
• The Cleanup: The internet is full of spam, emojis, and nonsense. The team built a "smart janitor" system that swept away the garbage (URLs, random symbols) and organized the good stuff. They even made sure the student didn't just learn about "cats" (which are everywhere) but also learned about rare things like "a specific type of mushroom" by balancing the lessons.

2. The Three-Stage Training Camp

You can't just throw a student into a final exam. Xray-Visual went through three specific training phases:

• Stage 1: The Blindfold Game (MAE): The model was shown images with big chunks covered up and had to guess what was missing. This taught it the basic structure of the world (e.g., "if I see two eyes and a nose, there's probably a face underneath").
• Stage 2: The Hashtag Quiz: The model was shown videos and had to guess the correct hashtag (like #sunset or #dogpark). This taught it to recognize specific objects and actions.
• Stage 3: The Match-Up Game (CLIP): Finally, it learned to match pictures with their descriptions. If you showed it a picture of a dog, it learned to say, "This matches the text 'a golden retriever playing fetch'."

3. The "Super-Brain" Text Encoder (LLM2CLIP)

This is the secret sauce. Usually, AI models use a small, simple brain to read text. Xray-Visual swapped that out for a Large Language Model (LLM)—the same kind of brain that powers advanced chatbots.

• The Analogy: Imagine a librarian who only knows the Dewey Decimal System (standard AI) versus a librarian who has read every book in the world and understands sarcasm, jokes, and complex stories (Xray-Visual).
• The Result: Because it uses a "super-brain" for text, it understands the nuance of what people are saying, not just the keywords. This makes it incredibly good at finding the right video for a specific search query in the real world.

4. Efficiency: The "Smart Filter"

Usually, high-resolution video requires massive computing power, like trying to run a marathon while carrying a heavy backpack. Xray-Visual uses a technique called EViT (Efficient Vision Transformer).

• The Analogy: Imagine watching a movie. A normal AI watches every single frame and every single pixel. Xray-Visual is like a smart viewer who realizes, "The background is just a blurry wall; I don't need to focus on that." It ignores the boring parts and focuses only on the important action.
• The Benefit: It runs 4 times faster and uses 75% less computing power than its competitors, yet it still sees everything clearly.

5. Why It Matters (The Real-World Test)

Here is the most important part: Academic tests vs. Real life.

• The Problem: Many AI models are like athletes who win gold medals in the gym (academic benchmarks) but trip over their own shoelaces when they go outside (real-world data). They fail when the lighting is weird, the image is blurry, or the content is from a different culture.
• The Xray-Visual Solution: Because it was trained on messy, real social media data, it doesn't get confused by the real world. It handles "domain shifts" (sudden changes in style or quality) like a pro.
• The Proof: When tested on internal Meta tasks (like matching ads to user videos), Xray-Visual crushed the competition, beating previous "champions" by huge margins.

Summary

Xray-Visual is a vision model that stopped studying from perfect textbooks and started learning from the messy, chaotic, beautiful reality of the internet. By combining a massive amount of real data, a "super-brain" for reading text, and a smart way to ignore unnecessary details, it has become the most efficient and robust visual AI to date. It's not just smart in a lab; it's smart in the real world.

Technical Summary

1. Problem Statement

Despite significant advances in Large Language Models (LLMs), computer vision models lag behind in terms of data scale and generalization capabilities. This disparity stems from the difficulty of curating visual data at the scale of human-generated text. Existing vision models often struggle with:

• Data Scarcity: Lack of massive, high-quality, diverse image-text pairs compared to text corpora.
• Domain Shift: Poor performance when moving from academic benchmarks (e.g., ImageNet) to real-world, noisy social media data.
• Efficiency vs. Performance: High-resolution models often require excessive computational resources (tokens), creating a bottleneck for large-scale deployment.
• Modality Separation: Traditional approaches often train separate models for images and videos, increasing deployment overhead.

2. Methodology

A. Data Curation (The "ViSE" and "URU" Datasets)

The authors constructed the largest publicly available vision encoder pre-training dataset to date, leveraging over 100 billion raw social media assets from Facebook and Instagram.

• ViSE (Image Dataset): Curated 15 billion image-text pairs.
  • Cleaning: Removed URLs, emojis, and non-semantic content.
  • Semantic Balancing: Used WordNet synsets to resample data, balancing long-tail concepts (oversampling rare concepts, undersampling common ones) to ensure diverse semantic coverage.
  • Noise Suppression: Applied similarity-based filtering using a pre-trained MetaCLIP model and label smoothing.
• URU (Video Dataset): Curated 10 billion video-hashtag pairs from 24 billion Instagram posts.
  • Hashtag Canonicalization: Mapped noisy user hashtags to canonical representations to reduce vocabulary size and improve consistency.
  • Action vs. Object: Created specific subsets for object recognition (Noun-5B) and action understanding (Action-5B).
• Synthetic Data & Refinement: Generated synthetic captions using an internal Multimodal LLM (MMLLM). To address repetition and hallucinations, they employed LLM-based rewriting (using LLaMA) to refine captions, followed by a Siamese loss strategy to jointly learn from both synthetic captions and original hashtags, preserving action-centric information.

B. Model Architecture

• Unified Backbone: A single Vision Transformer (ViT) architecture with 3D tokenization handles both images and videos. Images are processed by repeating them along the temporal dimension to match 3D convolution kernels.
• Efficiency (EViT): Utilizes Efficient Vision Transformer (EViT) with token reorganization. The model identifies and drops "inattentive" tokens (low attention scores from the CLS token), allowing for higher input resolutions (336px) with significantly fewer tokens (288 tokens) compared to baselines (1024 tokens).
• Registers: Appended "register" tokens to the sequence to capture outlier features and improve video performance.
• Text Encoder (LLM2CLIP): Replaced the standard small CLIP text encoder with LLaMA-1B.
  • Alignment: Used LoRA fine-tuning and SimCSE to align the LLM output space with the visual encoder, addressing the issue of feature homogeneity in vanilla LLMs.
  • Benefit: Enables processing of long, complex captions and improves real-world retrieval.

C. Three-Stage Training Pipeline

1. Stage 1: Self-Supervised MAE: Masked Autoencoding on 75% (images) or 90% (videos) of patches to learn robust visual representations.
2. Stage 2: Semi-Supervised Hashtag Classification: Fine-tuning on the curated hashtag datasets to learn object-centric and action-centric representations.
3. Stage 3: Contrastive Learning (CLIP-style): Joint training on image/video-caption pairs.
   • Loss Functions: Includes standard contrastive loss, Denoising Loss (injecting noise into embeddings to improve robustness), and SLIP (combining self-supervised and supervised objectives).
   • Progressive Resolution: Trains from 224px up to 336px/448px to adapt the model to higher resolutions efficiently.
   • Batch Sampling: Uses a probabilistic sampling strategy to ensure uniform exposure to both image and video modalities within an epoch, preventing modality bias.

3. Key Contributions

1. Scale: Training on 26 billion total samples (15B images + 10B videos), representing the largest scale for vision encoder pre-training reported.
2. Unified Architecture: A single model that achieves SOTA on both image and video tasks, eliminating the need for separate models.
3. Efficiency: Achieves state-of-the-art accuracy with 71.9% fewer tokens and 43.75% less pixel area than leading models (e.g., PE, SigLIP), resulting in a 4x increase in queries per second (QPS).
4. LLM Integration: Demonstrates that using LLaMA-1B as a text encoder significantly boosts retrieval performance and generalization in real-world scenarios, outperforming standard CLIP encoders.
5. Robustness: The model exhibits superior robustness to domain shifts, adversarial perturbations, and out-of-distribution (OOD) data compared to academic-benchmark-trained models.

4. Results

The Xray-Visual model establishes new State-of-the-Art (SOTA) across multiple benchmarks:

• Image Classification: 89.3% Top-1 accuracy on ImageNet (using 336px resolution and 288 tokens).
• Video Understanding: 78.1% Top-1 accuracy on Kinetics-700 and 74.69% on HMDB51.
• Cross-Modal Retrieval: New SOTA on MS-COCO and internal Meta benchmarks (Reels-to-Ads, FB Search).
  • Internal Metrics: Outperformed Perception Encoder (PE) by +10.8% on Instagram Reels-to-Ads retrieval and +10.2% on Facebook Reels-to-Ads retrieval.
• Robustness: Significant gains on adversarial benchmarks (e.g., +15.5% on ImageNet-Adversarial) and OOD datasets (ObjectNet, ImageNet-Sketch).
• Efficiency: Achieves these results with only 25% of the token count used by Perception Encoder (PE) and DiNO.

5. Significance

• Bridging the Gap: This work proves that scaling vision models on industry-scale, noisy social media data can surpass models trained on curated academic datasets, particularly in real-world applications.
• Real-World Applicability: The findings highlight that academic benchmarks do not always correlate with production performance. The integration of LLMs as text encoders and the use of massive, diverse datasets are critical for handling the complexity of real-world user content.
• Deployment Viability: By combining high accuracy with extreme computational efficiency (via token pruning and progressive resolution), the model offers a practical solution for large-scale deployment in recommendation systems, search, and content moderation.
• Future Direction: The paper sets a new precedent for "Unified Multimodal" training, suggesting that joint image-video learning yields better representations for both modalities than training them separately.