🤖 AI

Fake & Square: Training Self-Supervised Vision Transformers with Synthetic Data and Synthetic Hard Negatives

This paper introduces Syn2Co, a framework that enhances self-supervised learning for vision transformers by leveraging synthetic data for sample diversity and synthetic hard negatives for challenging contrasts, thereby exploring the potential and limitations of "faking it" to reduce reliance on vast real-world datasets.

Original authors: Nikos Giakoumoglou, Andreas Floros, Kleanthis Marios Papadopoulos, Tania Stathaki

Published 2026-07-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nikos Giakoumoglou, Andreas Floros, Kleanthis Marios Papadopoulos, Tania Stathaki

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Art of Learning Without a Teacher

Imagine you are trying to teach a robot to recognize a cat. In the old days, you would need a human to look at thousands of photos, point at the cat, and say, "This is a cat." This is called "supervised learning," and it works great, but it's slow, expensive, and requires a lot of human babysitting. To get around this, scientists invented "self-supervised learning." Think of this as giving the robot a pile of photos and saying, "Figure out what makes these pictures similar or different on your own." The robot learns by comparing pictures, trying to spot that two photos of the same dog are "friends" (positives) while a photo of a dog and a photo of a toaster are "strangers" (negatives).

However, there's a catch. To get really good at this, the robot needs a massive library of real-world photos, and it needs to be challenged with "hard negatives"—tricky comparisons that are almost the same but slightly different, like a golden retriever versus a labrador. If the robot only sees easy examples, it gets lazy and doesn't learn the subtle details. But what if we don't have enough real photos, or finding those tricky examples is too hard? This is where the idea of "faking it" comes in. What if we could generate fake photos or fake tricky examples using computers to help the robot learn? This is the playground of computer vision, a field where machines try to see and understand the world, and it's the stage for the story we are about to tell.

Fake It Till You Make It: A New Way to Train AI Eyes

In a paper titled "Fake & Square: Training Self-Supervised Vision Transformers with Synthetic Data and Synthetic Hard Negatives," a team of researchers from Imperial College London decided to test a bold idea: Can we teach AI to see just as well using "fake" stuff as we can using "real" stuff? Their approach is inspired by the old saying, "Fake it till you make it." Instead of waiting for the perfect real-world data, they decided to synthesize (or create) their own training materials to see if it would work.

The researchers focused on two specific ways to "fake it." First, they looked at Synthetic Data. Imagine you have a photo album of 100 different animals. Instead of taking more photos of real animals, they used a special computer program called a "diffusion model" to paint 130,000 new, fake pictures of those same animals. These aren't just blurry copies; they are brand new images that look like the real thing. They wanted to see if an AI could learn to recognize animals just by studying this fake album, or if it needed a mix of real and fake.

Second, they tackled the problem of Hard Negatives. In the robot's learning game, a "hard negative" is a tricky comparison, like asking the robot to tell the difference between two very similar shades of blue. Usually, finding these tricky pairs in real life is hard. So, the researchers invented a way to "fake" these tricky comparisons inside the robot's brain (the mathematical space where the AI stores what it knows). They took the AI's current understanding and mathematically blended it to create new, super-challenging examples that the robot had to figure out. They called their combined method Syn2Co.

What They Found

The team tested their "fake" methods on two popular AI brain designs: DeiT-S and Swin-T. They ran experiments on a dataset called ImageNet-100, which contains 100 categories of images.

Here is what the numbers tell us:

The Mix Matters: When they trained the AI using only the fake images, it did surprisingly well, but it wasn't perfect. However, when they mixed the fake images with real ones, the AI got even better. The results suggest that synthetic images are like a powerful supplement; they boost the learning, but they don't completely replace the need for real-world data just yet.
Different Brains, Different Needs: The two AI models reacted differently to the "faking." The DeiT-S model loved the combination of fake images and fake tricky comparisons, reaching a top score of 82.12% accuracy when trained for 300 epochs (cycles of learning). The Swin-T model, on the other hand, seemed to prefer the fake tricky comparisons (synthetic negatives) more than the fake images.
The "Hard" Lesson: The researchers found that adding these synthetic hard negatives helped the AI learn sharper, more detailed features. For instance, in one test, they varied the percentage of these hard negatives from 0% to 40%. The AI's performance generally improved as they added more of these challenging examples, showing that the "fake" struggles were actually helping the AI get stronger.

The Verdict

The paper doesn't claim that fake data is a magic wand that solves everything. Instead, it suggests that synthetic data is a promising tool. The authors found that while you can't just throw away all your real photos and rely 100% on computer-generated ones, using synthetic data to augment (add to) your real data works very well. It helps the AI learn faster and become more robust, especially when real data is hard to get or when you need more challenging examples to test the AI's limits.

In short, the researchers showed that you can indeed "fake it" to a certain extent. By generating synthetic images and synthetic challenges, they created a training environment that helped AI models learn better, proving that sometimes, a little bit of artificial help can make the learning process feel very real.

Technical Summary: Fake & Square: Training Self-Supervised Vision Transformers with Synthetic Data and Synthetic Hard Negatives

Problem Statement

Contrastive self-supervised learning has achieved remarkable success in computer vision but faces two primary bottlenecks:

Data Scarcity and Diversity: These methods typically require vast amounts of real-world labeled or unlabeled data to achieve competitive performance, creating a scalability issue similar to supervised learning.
Hard Negative Limitations: The quality of learned representations is heavily dependent on "hard negatives" (dissimilar examples that are semantically close). Existing methods often struggle with the limited diversity of these hard negatives, which are crucial for learning discriminative features.

The authors propose exploring alternatives to these requirements by investigating the adage "fake it till you make it." They ask whether synthetic data and synthetic hard negatives can effectively augment or replace real-world components in training Vision Transformers (ViTs).

Methodology: Syn2Co

The paper introduces Syn2Co, a framework that integrates two forms of "faking" into the self-supervised learning pipeline:

1. Synthetic Data Generation (Image Space)

Approach: The authors utilize a generative diffusion model to create a synthetic clone of the ImageNet-100 dataset.
Scale: Approximately 130,000 synthetic images are generated, maintaining the same class distribution as the original dataset.
Integration: During training, images are sampled from a mixed pool of real data ( $X$ ) and synthetic data ( $X_s$ ). This allows for scenarios ranging from training exclusively on real data to exclusively on synthetic data, or various mixing ratios.

2. Synthetic Hard Negatives (Representation Space)

Approach: Instead of relying solely on real negative samples from a memory queue, the framework generates synthetic hard negatives directly in the feature representation space ( $Z$ ).
Selection: Given a query representation $q$ , the $N$ hardest negatives ( $Q_N$ ) are selected from the existing queue based on cosine similarity.
Synthesis: A synthesis function $F$ operates on the query and selected hard negatives to create new synthetic negatives ( $s$ ). The authors implement six strategies for $F$ , including interpolation, extrapolation, mixing, noise jittering, perturbation, and adversarial methods.
Integration: These synthetic negatives ( $Q_s$ ) are combined with the real negative queue ( $Q$ ) to expand the diversity of contrasts during training without requiring additional real data collection.

3. Learning Framework

The method employs a contrastive learning objective (InfoNCE loss) similar to MoCo and BYOL.

Architectures: The framework is evaluated on DeiT-S (Data-efficient Image Transformer) and Swin-T (Swin Transformer Tiny).
Training: The target encoder is updated via a momentum mechanism, while the online encoder is optimized via backpropagation.

Key Contributions

Synthetic Data Clone: The creation of a synthetic clone of ImageNet-100 and a demonstration of how synthetic data impacts contrastive self-supervised learning.
Synthetic Negatives for Transformers: The adaptation and extension of synthetic negative generation techniques (previously explored in other contexts) to Vision Transformers, showing that DeiT and Swin can leverage these synthetic contrasts to learn more discriminative features.
Comprehensive Evaluation: A systematic investigation into the trade-offs between real and synthetic data, and the impact of varying the proportion and hardness of synthetic negatives.

Experimental Results

The experiments were conducted on ImageNet-100 using linear probing to evaluate the quality of learned representations.

Performance Comparison:
- DeiT-S: The model effectively leveraged both synthetic data and synthetic negatives. The best performance (82.12% Top-1 accuracy) was achieved using the full Syn2Co approach (both synthetic components) over 300 epochs, outperforming the supervised baseline (80.50%) and standard MoBY (79.36%).
- Swin-T: This architecture benefited primarily from synthetic negatives. It achieved optimal performance with synthetic negatives alone (84.04% Top-1), showing that it could leverage synthetic contrasts effectively even without synthetic image data.
Data Mixtures:
- Models trained exclusively on synthetic data achieved reasonable performance, but accuracy improved as the proportion of real data increased.
- The results suggest synthetic images serve as valuable complements rather than complete substitutes for real data, though the performance gap between fully synthetic and fully real training is relatively modest.
Hard Negative Sensitivity:
- DeiT: Exhibited sensitivity to both the hardness level and the proportion of synthetic negatives.
- Swin: Maintained consistent performance gains across different configurations of synthetic negatives.

Significance and Claims

The paper modestly claims that it does not introduce a fundamentally new method per se, but rather builds on existing self-supervised approaches to explore the limits of synthetic augmentation.

Promise: The findings highlight the potential of synthetic data and synthetic hard negatives to create robust and transferable visual representations, particularly in reducing the reliance on vast amounts of curated real-world data.
Limitations: The authors note that synthetic data follows a pattern of diminishing returns and that diffusion models introduce distribution shifts that limit their effectiveness as complete substitutes for real data.
Architectural Nuance: A key insight is that different transformer architectures (DeiT vs. Swin) respond differently to synthetic augmentation strategies, suggesting that the choice of augmentation must be tailored to the specific model design.

In summary, Syn2Co demonstrates that "faking it" through synthetic data and synthetic hard negatives is a viable strategy to enhance self-supervised learning, offering a path toward more scalable and resource-efficient training of Vision Transformers.