EA-Swin: An Embedding-Agnostic Swin Transformer for AI-Generated Video Detection

The paper proposes EA-Swin, an embedding-agnostic Swin Transformer that achieves state-of-the-art accuracy and generalization in detecting AI-generated videos by modeling spatiotemporal dependencies on pretrained embeddings, supported by a new large-scale benchmark dataset.

The Gist

Imagine you are a detective trying to spot a fake painting. In the past, you could easily tell a fake because the brushstrokes were messy, the colors were wrong, or the perspective was off. You were looking at the pixels (the tiny dots of color).

But today, AI artists (like Sora, Veo, and Kling) have become so good that their paintings look perfect. The brushstrokes are smooth, the colors are vibrant, and the perspective is flawless. If you look at the painting up close, you can't tell it's fake.

This is the problem the paper "EA-Swin" solves.

The Old Way: Looking at the Canvas

Previous detectors tried to find fakes by looking for tiny errors in the image (like a weird shadow or a glitchy texture). But because AI has gotten so good at fixing those errors, these detectors are failing. It's like trying to catch a master thief who wears a perfect disguise; looking at their face (the pixels) doesn't help anymore.

The New Way: Watching the Dancer

The authors of this paper realized that while AI can make a single frame look perfect, it struggles to keep the story consistent over time.

Imagine a real human dancer and a robot dancer.

• The Real Human: Their movements flow naturally. If they spin, their hair follows a specific physics-based path. If they jump, their landing has a specific weight. Their "trajectory" (the path their body takes through time) is complex and slightly messy in a natural way.
• The Robot (AI): Even if the robot looks exactly like a human, its movement might be too smooth, or it might drift slightly in a way that feels "mathematically perfect" but physically impossible.

EA-Swin doesn't look at the dancer's face (the pixels). Instead, it looks at the dancer's movement pattern (the "embedding trajectory").

How EA-Swin Works (The "Embedding-Agnostic" Part)

The paper introduces a new tool called EA-Swin. Here is the simple breakdown:

1. The "Embedding" (The Secret Code): Instead of looking at the raw video, the system first asks a smart AI (called a "pretrained encoder") to translate the video into a secret code. This code doesn't describe the colors; it describes the meaning and the motion of the video.
   • Analogy: Imagine translating a song into sheet music. You aren't listening to the singer's voice anymore; you are looking at the notes.
2. The "Agnostic" Part (The Universal Adapter): This tool is "agnostic," meaning it doesn't care which AI made the video. Whether the video was made by OpenAI, Google, or a random open-source tool, EA-Swin can read the secret code from any of them. It's like a universal translator that works with any language.
3. The "Swin" Part (The Window Watcher): The system uses a special method called "Swin Transformer." Imagine looking at a video through a small window.
   • Old way: You look at the whole room at once (too much data, too slow).
   • EA-Swin way: You look at small windows of the video, shifting them slightly to see how the movement flows from one window to the next. It checks: "Does the movement in this 1-second window connect naturally to the next window?"

The Big Dataset: "EA-Video"

To train this detective, the authors built a massive library called EA-Video.

• It contains 130,000 videos.
• It includes videos from almost every major AI generator (Sora 2, Veo 3, Kling, etc.).
• Crucially: They tested the detective on "unseen" generators. Imagine training a dog to find a specific type of fake coin, and then testing it on a brand new type of fake coin it has never seen before. EA-Swin passed this test with flying colors, proving it learned the concept of fakeness, not just the specific look of one AI.

The Results: A Super Detective

When they tested EA-Swin:

• Old Detectors: Got about 80-90% accuracy. They were confused by the new, high-quality AI videos.
• EA-Swin: Got 97% to 99% accuracy. It could spot the fake even when the video looked perfect to the human eye.

Why This Matters

We are entering an era where we can't trust our eyes anymore. A video of a politician saying something they never said, or a celebrity doing something they never did, could look 100% real.

EA-Swin is a new kind of truth detector. It doesn't rely on finding "glitches" because glitches are disappearing. Instead, it relies on the laws of physics and time. It knows that real life has a certain rhythm and flow that AI, no matter how advanced, still struggles to perfectly mimic.

In short: EA-Swin is a smart, adaptable detective that ignores the "face" of the video and instead studies the "dance" of the video to tell us what is real and what is fake.

Technical Summary

1. Problem Statement

The rapid advancement of foundation video generators (e.g., Sora-2, Veo-3, Kling, Hunyuan) has produced synthetic videos that are increasingly indistinguishable from real footage. This evolution poses significant security and societal risks, including misinformation and deepfake proliferation.

Existing detection methods face three critical limitations:

• Pixel-Level Reliance: Early detectors relied on visible artifacts (e.g., frequency anomalies, pixel inconsistencies). Modern generators explicitly minimize these artifacts through diffusion models and post-processing, rendering pixel-space detection ineffective.
• Inadequate Temporal Modeling: Many approaches adapt image-based detectors to video or use shallow embedding trajectories (e.g., simple velocity/acceleration statistics). These fail to capture the complex, long-range spatiotemporal dependencies inherent in high-fidelity video synthesis.
• Data and Generalization Gaps: Existing benchmarks often rely on outdated generators or lack diversity. Consequently, models trained on specific generators fail to generalize to unseen, emerging models (the "unseen-generator" problem).
• Computational Cost: Methods based on Multimodal Large Language Models (MLLMs) offer semantic reasoning but are computationally prohibitive for large-scale deployment.

2. Methodology: EA-Swin

The authors propose EA-Swin, an Embedding-Agnostic Spatiotemporal Detection Framework. The core philosophy is to shift detection from the pixel space to the representation space of pretrained video encoders.

Key Architectural Components:

• Embedding-Agnostic Design: Instead of processing raw pixels, EA-Swin operates on frozen embeddings extracted from a pretrained video encoder (e.g., V-JEPA2). This decouples the detector from the specific visual features of the generator, focusing instead on the dynamics of the representation.
• Factorized Spatiotemporal Attention:
  • Unlike standard Video Swin Transformers that apply joint attention over all tokens (computationally expensive), EA-Swin uses a factorized windowed attention mechanism.
  • Temporal Attention: Models dependencies across time for each spatial token independently using shifted windows.
  • Spatial Attention: Models dependencies across space within each frame using shifted windows.
  • This design reduces computational complexity from quadratic to linear relative to sequence length while preserving long-range modeling capabilities.
• Window Shifting Mechanism: To enable cross-window interaction and global context modeling without incurring quadratic complexity, the architecture employs spatial and temporal window shifts between transformer blocks.
• Encoder Compatibility: The framework is designed to be compatible with any generic ViT-style patch-based encoder, allowing it to leverage the strongest available representation models (e.g., V-JEPA2, CLIP, DINOv2).

3. Key Contributions

A. The EA-Video Dataset

The authors constructed EA-Video, a large-scale benchmark dataset comprising 130,000 videos (approx. 65k AI-generated and 62k real).

• Diversity: It integrates data from diverse commercial (Sora-2, Veo-3, Kling) and open-source (CogVideoX, Wan2) generators.
• Unseen-Generator Protocol: The dataset is split such that the test set contains generators completely unseen during training. This rigorously evaluates cross-distribution generalization, a critical metric for real-world deployment.
• Real-World Artifacts: The real video subset includes diverse sources, including game physics and bug-induced artifacts, to prevent the model from confusing game glitches with AI artifacts.

B. Representation-Level Paradigm

The paper demonstrates that temporal evolution in embedding space provides a stronger forensic signal than static frame analysis. Visualizations (t-SNE) show that while real and AI videos overlap in early frames, their embedding trajectories diverge significantly over time, with AI videos exhibiting smoother, more constrained transitions compared to the irregular dynamics of real videos.

C. Scalable and Robust Architecture

EA-Swin achieves state-of-the-art performance while remaining computationally efficient by avoiding raw pixel processing and heavy MLLMs.

4. Experimental Results

Performance on Seen Generators

On the "seen" generator test set, EA-Swin achieved:

• Accuracy: 0.9866 (vs. ~0.95 for the next best, DeMamba).
• F1-Score: 0.9869.
• AUC: 0.9991.
• It consistently outperformed prior SoTA methods (ResTraV, NPR, STIL, Forgelens) by a margin of 5–20%.

Performance on Unseen Generators (Generalization)

This is the most critical result. When tested on generators not present in the training set (e.g., Gen4, PixVerse, SkyReels):

• EA-Swin Accuracy: 0.974 (Average).
• EA-Swin AUC: 0.997.
• Comparison: Many baselines collapsed (e.g., WaveRep dropped to ~0.5 accuracy on specific unseen generators; TALL showed significant instability). EA-Swin maintained high recall and precision across nearly all unseen distributions.

Robustness and Ablation

• Post-Processing: The model remains robust against common real-world degradations like H.264 compression, Gaussian blur, and noise, with accuracy dropping only moderately (e.g., from 0.974 to 0.916 under heavy noise).
• Frame Reduction: Performance degrades gracefully as input frames are reduced from 16 to 2, though 16 frames yield the most stable results.
• Backbone Analysis: V-JEPA2 was identified as the optimal vision encoder, outperforming CLIP, DINOv2, and DINOv3, likely due to its superior self-supervised spatiotemporal pretraining.

5. Significance and Conclusion

• Paradigm Shift: The paper establishes that the future of AI-generated video detection lies in representation-space modeling rather than pixel-space artifact hunting. As generators improve, pixel-level cues vanish, but the "temporal consistency" and "representation dynamics" of real videos remain difficult for AI to perfectly replicate.
• Scalability: By operating on frozen embeddings, EA-Swin offers a scalable solution that can be easily adapted to new encoders without retraining the entire pipeline from scratch.
• Benchmarking: The introduction of the EA-Video dataset with an unseen-generator protocol sets a new standard for evaluating the generalization capabilities of forensic tools, moving the field beyond overfitting to specific training distributions.

In summary, EA-Swin provides a robust, efficient, and generalizable solution to the growing challenge of detecting high-fidelity synthetic video, effectively bridging the gap between rapidly evolving generative models and reliable detection mechanisms.