🤖 machine learning

LoRA-based Adaptation Alone Is Not Enough: Understanding the Limits of Foundation Models for Face Presentation Attack Detection

This study systematically evaluates 32 foundation models for face presentation attack detection and finds that while LoRA-based adaptation achieves strong intra-dataset performance, it fails to solve cross-dataset generalization issues, indicating that pretrained representations and adaptation data are more critical than lightweight fine-tuning strategies.

Original authors: Peter Lorenz, Anjith George, Marcel Sébastien

Published 2026-08-11

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Peter Lorenz, Anjith George, Marcel Sébastien

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to spot a fake ID card. You show it a thousand photos of real people and a thousand photos of people holding up printed pictures or playing videos of faces on their phones. The robot gets really good at spotting the fakes in that specific room, under that specific light, with those specific cameras. But the moment you take the robot to a different room, change the lighting, or swap the camera, the robot suddenly forgets everything and starts guessing randomly. This is the frustrating reality of "Face Presentation Attack Detection" (PAD). It's a branch of computer vision dedicated to keeping face recognition systems safe from tricks like photos, masks, or replayed videos. The big dream is to build a detector that works everywhere, no matter the camera or the lighting. Recently, scientists have been trying to use "Foundation Models"—massive, super-smart AI brains trained on billions of internet images and text—to solve this. The hope was that these giant brains already know so much about the world that they could easily learn to spot fakes with just a tiny bit of extra training.

This paper puts that hope to the test with a massive experiment. The researchers took 32 different types of these giant AI brains (including famous ones like CLIP and DINO) and tried to teach them to spot face fakes using a clever, lightweight trick called "LoRA." Think of LoRA like putting a small, adjustable training collar on a giant dog; you don't retrain the whole dog, you just tweak a tiny fraction of its brain (less than 1%) to learn a new trick. The results were a mix of amazing success and a harsh reality check. When the AI was tested on the same data it was trained on, it became a superstar, spotting fakes with near-perfect accuracy (less than 2% error). However, when the researchers asked the AI to spot fakes in a new environment it hadn't seen before, the performance crashed. The error rate jumped to between 20% and 43%, which is barely better than flipping a coin. The paper concludes that while LoRA is great at polishing a model for a specific job, it isn't enough to fix the deeper problem of making AI work across different worlds. The "brain" of the model matters more than the size of the model, and the data it was trained on matters even more.

The Story of the Smart Robot and the Shifty Fakes

Let's dive into the adventure. The scientists started by asking a simple question: Is a tiny bit of extra training (LoRA) enough to make these giant AI brains robust enough to spot face fakes anywhere?

To find out, they set up a massive tournament. They gathered 32 different "Vision Encoders"—these are the eyes of the AI, the parts that actually look at the pictures. Some of these eyes came from models trained just on images, while others were the "vision towers" pulled out of huge "Vision-Language Models" (VLMs), which are AI brains that can see images and read text at the same time. They also tested 9 of these giant VLMs in their "zero-shot" mode. Zero-shot is like handing a student a test without letting them study the subject first; you just ask, "Is this a real face or a fake?" and see what they say.

The Zero-Shot Surprise
First, they tried the zero-shot approach. They asked the giant VLMs to spot fakes using a simple prompt, like a teacher asking a question. The result? The AI was completely lost. It performed at about 50% accuracy, which is the same as guessing "heads or tails" by flipping a coin. Even the smartest, most expensive models with billions of parameters couldn't figure it out without specific training. This told the researchers that just having a smart brain isn't enough; the brain needs to be specifically tuned for the job of spotting fakes.

The LoRA Magic (and its Limits)
Next, they tried the LoRA method. They took the frozen eyes of these 32 models and added those tiny, adjustable training collars. They trained them on four different datasets (collections of face photos taken with different cameras and lighting).

The Good News: When they tested the AI on the same dataset it learned from, it was incredible. Most models dropped their error rate to below 2%. Some, like the CLIP ViT-B/32, got as low as 0.3%. It was as if the AI had suddenly become a master detective for that specific room.
The Bad News: When they moved the AI to a different dataset (a different room, different camera), the magic vanished. The error rates skyrocketed to between 19.8% and 42.6%.

The researchers realized that LoRA was acting like a very good tailor. It could perfectly adjust a suit to fit one person (one dataset) by tweaking a few buttons. But it couldn't make that suit fit a completely different person (a different dataset) just by adjusting the buttons. The suit was still shaped for the first person.

Size Doesn't Matter (Much)
A common belief in AI is that "bigger is better." The researchers tested this by comparing tiny models to massive ones. They found that a relatively small model called CLIP ViT-B/32 (with only 0.09 billion parameters) actually performed better than some massive models with over 8 billion parameters. In fact, the tiny CLIP model was one of the best at spotting fakes across different datasets, while a huge model called InternViT-6B (with 5.54 billion parameters) was one of the worst at crossing the gap between datasets.

This suggests that the type of training the model had originally received matters way more than how big it is. Models trained with "contrastive" methods (learning to match images with their text descriptions) or "self-distillation" (learning to be consistent with itself) worked better than models trained just to fill in missing parts of a picture.

The "Data Diet" Matters
The team also looked at how much data the models had eaten during their original training. They found that eating a "web-scale" diet (billions of images from the internet) was necessary to get good results, but eating more than that didn't help much. Once a model had seen enough of the internet, adding more data didn't make it better at spotting fakes in new places. It was like a student who has read every book in the library; reading a few more won't suddenly make them a genius at a specific new subject.

The Hidden Clue: The "Room" vs. The "Face"
The most fascinating discovery came from looking at how the AI "saw" the pictures. The researchers used a technique called t-SNE to visualize the AI's internal thoughts. They found that even after the LoRA training, the AI was still grouping pictures based on where they were taken (the dataset) rather than what they were (real or fake).
Imagine you are trying to sort a pile of photos into "Real People" and "Fake People." But the AI keeps sorting them into "Photos taken in the Lab" and "Photos taken on the Street." The LoRA training helped it sort the "Lab" photos perfectly, but when it saw a "Street" photo, it got confused because it was still looking for "Lab" clues. The AI hadn't learned the universal concept of a "fake face"; it had just learned to recognize the specific lighting and camera quirks of the training room.

The Verdict
The paper concludes that LoRA adaptation alone is not enough to solve the problem of cross-dataset face detection. While it makes models incredibly accurate in the environment they were trained in, it fails to make them robust enough to handle new cameras, new lights, or new attack methods. The "generalization score" (a new metric the authors invented to measure how well a model works in both scenarios) showed that even the best models were still struggling to bridge the gap.

The authors suggest that the problem isn't the size of the model or the cleverness of the LoRA trick. The problem is that the models are still too dependent on the specific "flavor" of the data they were trained on. To truly solve this, we might need to change how we train these models from the ground up, perhaps by teaching them to ignore the "room" and focus only on the "face," or by using different training strategies that force the AI to learn universal rules rather than specific habits.

In short, we have built AI detectives that are brilliant in their home precinct but get lost the moment they step out the door. The tiny LoRA collar helps them walk a little better, but it doesn't give them a map for the whole world.

Technical Summary: LoRA-based Adaptation Alone Is Not Enough: Understanding the Limits of Foundation Models for Face Presentation Attack Detection

Problem Statement
Face Presentation Attack Detection (PAD) aims to distinguish genuine users from spoofed inputs (e.g., printed photos, replayed videos). While deep learning methods achieve high accuracy within specific datasets, they suffer from a significant performance gap when evaluated across different datasets (cross-dataset evaluation). This degradation occurs due to variations in sensors, lighting conditions, and attack types. Although Foundation Models (FMs) pretrained on web-scale data offer richer visual priors than limited PAD datasets, existing research has primarily focused on CLIP-based models. It remains unclear whether parameter-efficient adaptation strategies, such as Low-Rank Adaptation (LoRA), can effectively bridge the cross-dataset gap across diverse foundation model architectures and training paradigms.

Methodology
The authors present a systematic benchmark evaluating 32 pretrained vision encoders and 9 Vision-Language Models (VLMs) under a unified protocol. The study utilizes the four standard MCIO datasets: MSU-MFSD, CASIA-FASD, Replay-Attack, and OULU-NPU.

Adaptation Strategy: The primary method employed is LoRA, applied to the query and value projections of attention blocks (or equivalent layers in ConvNeXt) with a rank of 8 and scaling factor of 16. This updates fewer than 1% of the model's parameters while keeping the backbone weights frozen. A linear head is trained jointly with the adapters.
Model Diversity: The evaluation covers five pretraining categories:
1. Vision-language contrastive learning (e.g., CLIP, SigLIP, EVA-CLIP).
2. Self-distillation (e.g., DINOv2, DINOv3).
3. Masked-image modeling (e.g., BEiT, ViT-MAE, ViT-MSN).
4. Masked-image self-supervised convolutional models (e.g., ConvNeXt-V2).
5. Vision towers extracted from Multimodal Large Language Models (e.g., Qwen-VL, LLaVA, DeepSeek-VL2).
Baselines: Results are compared against zero-shot VLM prompting and four specialist PAD baselines (DeepPixBiS, FSFM-FAS, FLIP variants, and FoundPAD) that utilize full fine-tuning or specialized architectures.
Metrics: Performance is measured using the Average Classification Error Rate (ACER) for intra-dataset (ID) and cross-dataset (CD) scenarios. A new "bias-free generalization score" ( $G$ ) is proposed, defined as the geometric mean of the skill (distance from 50% chance) in both ID and CD settings.

Key Results

Zero-Shot Limitations: Zero-shot prompting of VLMs without task-specific training yields performance near chance (average ACER between 35% and 50%), indicating that semantic reasoning alone cannot solve PAD.
Intra-Dataset Success: LoRA adaptation is highly effective within the training domain. Most adapted backbones achieve an intra-dataset ACER below 2%, with top performers like CLIP ViT-B/32 reaching 0.3%. This demonstrates that lightweight adaptation is sufficient for strong performance when the test distribution matches the training distribution.
Cross-Dataset Failure: Despite strong intra-dataset results, cross-dataset ACER remains high, ranging from 19.8% to 42.6% across all models. LoRA adaptation fails to overcome dataset bias. For instance, InternViT-6B achieves a low 0.7% intra-dataset ACER but a poor 39.8% cross-dataset ACER.
Role of Pretraining and Scale:
- Objective over Scale: The pretraining objective matters more than model size or data scale. Contrastive vision-language models (CLIP, EVA-CLIP) and self-distillation models (DINOv2-R) consistently outperform masked-image modeling models (ViT-MAE, ViT-MSN) and large VLM vision towers.
- Data Scale: While web-scale pretraining (beyond 100M images) is necessary for better cross-dataset performance, simply increasing data scale beyond a certain threshold (e.g., SigLIP's ~10B images vs. CLIP's 400M pairs) does not guarantee further improvements. Curation and the pretraining objective are dominant factors.
Feature Geometry: t-SNE analysis reveals that even after LoRA adaptation, the feature space remains structured by the acquisition domain (source dataset) rather than the presentation attack class. This explains why thresholds calibrated on one dataset fail to generalize to others.

Significance and Claims
The paper claims that LoRA adaptation alone is insufficient for robust cross-dataset PAD. The study challenges the assumption that parameter-efficient tuning of large foundation models automatically solves the generalization problem. Instead, the authors argue that the primary limitations lie in the selection of the backbone architecture, the pretraining objective, and the specific training dataset used for adaptation, rather than the adaptation strategy itself.

The authors propose that future work should focus on mechanisms beyond lightweight adaptation, such as domain-invariant losses or alignment with target-domain data, to address the persistent cross-dataset gap. They also introduce the generalization score $G$ as a metric to jointly evaluate intra- and cross-dataset capabilities, preventing models that excel in one area but fail in the other from being overvalued. The study concludes that while foundation models offer powerful representations, their direct application to PAD via LoRA does not yet resolve the fundamental issue of dataset bias.

The Story of the Smart Robot and the Shifty Fakes

More like this