💻 computer science

On the Use of Synthetic Data for Threshold Calibration in Face Recognition: Performance and Security Implications for Border Control Systems

This paper investigates the use of synthetic face data for calibrating verification thresholds in European border control systems, finding that while synthetic data is useful for preliminary development, it fails to reliably generalize to real-world, low false-match rate scenarios due to distribution mismatches that degrade performance and increase security vulnerabilities.

Original authors: Arto Apila

Published 2026-07-29

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Arto Apila

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the head of security for a massive, high-tech castle. Every day, thousands of people try to enter, and your job is to make sure only the right ones get in. You have a magical mirror that can recognize faces, but it's not perfect. Sometimes it thinks a stranger looks like a resident (a "false match"), and sometimes it thinks a resident looks like a stranger (a "false non-match"). To keep the castle safe, you have to set a very strict rule: "Only let people in if the mirror is almost 100% sure they belong." This rule is called a "threshold." If you set the bar too low, impostors get in; if you set it too high, your own friends get locked out.

Now, imagine you want to test your mirror before the castle opens. You can't just wait for real people to show up and make mistakes; that's too dangerous. So, you decide to build a bunch of perfect, computer-generated "fake" people to test your mirror. This is the world of synthetic data: digital faces created by AI that look real but aren't. The big question scientists are asking is: "Can we trust these fake faces to tell us exactly where to set the security bar?" If the fake faces act differently than real people, your security system might be a disaster waiting to happen. This paper dives into that exact problem, specifically for the new European border control systems that need to be incredibly precise.

The Digital Dress Rehearsal That Went Wrong

In this study, Arto Apila from the University of Oulu decided to play the role of a border guard's test pilot. The goal was to see if using synthetic face data (computer-generated faces) to set the security "threshold" for border control systems would work in the real world. Think of it like a flight simulator. Pilots train in simulators to learn how to fly, but if the simulator's physics are slightly off, the pilot might crash when they finally get into a real plane. The researchers wanted to know: Is the "simulator" (synthetic data) good enough to set the rules for the "real plane" (actual border control)?

They used a specific type of AI model called EdgeFace, which is like a lightweight, fast-running version of a super-smart face detector, designed to work on small devices like the ones you might find at an airport gate. They tested this model using two different sets of "fake" faces (synthetic datasets) and compared them against sets of "real" faces (photos taken under controlled and messy, real-world conditions).

The "Tail" of the Story

Here is the tricky part that the paper uncovers. When you look at how a face recognition system scores people, it's like a graph. Most people get scores in the middle (clearly a match or clearly not a match). But the danger zone is at the very end of the graph, in the "tails." This is where the system gets confused and might mistake a stranger for a friend.

The researchers found that while synthetic data looks great in the middle of the graph, it gets the tails completely wrong. It's like trying to predict how a crowd will behave during a panic by watching a calm rehearsal. The fake faces didn't have the same "worst-case scenarios" as real people.

When the researchers tried to use the rules (thresholds) they learned from the fake faces on real people, things went sideways:

The "Too Strict" Problem: Sometimes, the system set the bar so high that it started rejecting real travelers. In one test, when they tried to be super strict (aiming for a 0.001% error rate), the system rejected nearly 26% of the real people it should have let in!
The "Too Loose" Problem: In other cases, the system was too trusting. It let in impostors at a rate that was way higher than the safety limit.
The "Morph" Attack: The researchers also tested what happens if someone tries to sneak in by blending two faces together (a "morph" attack). They found that the settings derived from fake data made the system much more vulnerable to these tricks. If you set the rules based on the fake data, the system might let a blended face slip right through the door.

The Verdict: Don't Trust the Simulator Alone for the Final Call

The paper's main discovery is a bit of a bummer for those hoping for a quick, easy fix that relies only on fake data. It suggests that synthetic data cannot be trusted to set the final security rules for high-stakes systems like border control, especially when you need to be incredibly precise (low False Match Rates).

The authors found that even if you use two different types of fake data, they give you different answers. It's like asking two different weather forecasters to predict a hurricane; one says "bring an umbrella," and the other says "build a bunker." You can't know which one is right until you see the real storm.

The study explicitly argues against the idea that you can just generate a million fake faces, run your tests, and then set your border security rules based solely on that. The "mismatch" in the tails of the data means the rules will be wrong when applied to real humans. However, the paper does not say synthetic data is useless. Instead, it recommends using it for the early stages of development and preliminary calibration, but insists that the final, critical threshold selection must be validated with real-world data.

What Should We Do Instead?

So, is synthetic data useless? Not at all! The paper suggests it's still a great tool for the early stages of building the system. You can use it to train the AI, to see if the code works, and to do a rough check. But when it comes time to set the final, life-or-death security threshold, you must use real-world data.

Think of it like baking a cake. You can use a recipe book (synthetic data) to learn how to mix ingredients and bake a test batch. But before you serve the cake to the Queen, you have to taste the actual cake you made with real ingredients. If you skip the tasting and just trust the book, you might serve something that's too salty or too sweet.

The researchers conclude that for systems like the European Entry/Exit System (EES), which needs to be incredibly secure, relying solely on computer-generated faces is a risky gamble. The "fake" faces just don't capture the messy, unpredictable reality of human faces well enough to set the rules for the real world. To keep the borders safe, we need to test our systems on real people, not just their digital twins, especially for the final decision.

Technical Summary: On the Use of Synthetic Data for Threshold Calibration in Face Recognition

Problem Statement

The deployment of the European Entry/Exit System (EES) mandates large-scale biometric verification at border control points, requiring face recognition systems to operate at extremely low False Match Rates (FMR). While regulatory frameworks define performance targets for the central system, they do not specify how verification thresholds should be calibrated at the Member State level. In operational environments, obtaining representative real-world data for calibration is often hindered by legal, logistical, and privacy constraints. Consequently, there is a critical need to determine whether synthetic face data can reliably substitute for real-world data in calibrating decision thresholds, particularly under strict low-FMR constraints where system security and usability are most sensitive.

Methodology

The authors designed an experimental framework to evaluate the transferability of thresholds derived from synthetic datasets to real-world document-to-live verification scenarios. The study utilized the following components:

Datasets:
- Synthetic: FLUXSyn-ID (a diffusion-based dataset modeling document-to-live domain shifts) and Syn-Multi-PIE (a StyleGAN2-based dataset with controlled variations).
- Real-World: Color FERET (controlled conditions, low intra-class variability) and CFPW (unconstrained conditions, high variability).
- Adversarial: AMSL morph dataset to evaluate security against morph-based attacks.
Model: The EdgeFace-S model (a lightweight architecture with $\gamma=0.5$ ) was used to reflect realistic deployment constraints in border control systems.
Protocols:
- Calibration: Thresholds were estimated from impostor score distributions at target FMR levels ranging from 0.1% to 0.001%.
- Evaluation: Calibrated thresholds were applied to separate evaluation datasets to measure generalization. Metrics included False Non-Match Rate (FNMR), observed FMR, and morph attack passing rates (NMPR and FMPR).
- Scope: The study isolated the effects of dataset-dependent score distributions on calibration behavior, focusing on the tails of impostor distributions that dictate low-FMR performance.

Key Contributions

Distribution Alignment Analysis: A systematic comparison of genuine and impostor score distributions between synthetic and real datasets, revealing significant misalignments in the distribution tails.
Cross-Dataset Generalization: An evaluation of threshold transferability under low-FMR constraints, demonstrating that thresholds calibrated on one dataset (synthetic or real) often fail to generalize to others.
Security Assessment: An analysis of system robustness against morph-based attacks, highlighting the trade-off between recognition performance and security vulnerability induced by calibration choices.
Operational Insight: The first systematic analysis of threshold calibration transfer using synthetic data specifically under the low-FMR constraints relevant to operational biometric systems like EES.

Results

The experiments yielded three primary findings:

Dataset Dependency: Threshold calibration is highly dataset-dependent. Synthetic datasets exhibited fundamentally different score distributions; for instance, Syn-Multi-PIE produced systematically higher thresholds due to tight identity clustering, while FLUXSyn-ID yielded lower thresholds closer to real-world variability.
Poor Generalization: Thresholds derived from synthetic data did not reliably generalize to real-world scenarios.
- When applied to Color FERET (controlled), FLUXSyn-ID thresholds maintained low FNMR and FMR, whereas Syn-Multi-PIE thresholds resulted in high FNMR.
- When applied to CFPW (unconstrained), all synthetic calibration strategies suffered significant performance degradation, with FNMR rates rising sharply (e.g., up to 67.62% for Syn-Multi-PIE at 0.001% target FMR).
Amplified Sensitivity at Low-FMR: Small discrepancies in impostor score distribution tails led to disproportionately large increases in FNMR at stricter operating points (e.g., 0.001%).
Security Vulnerabilities: Calibration strategies that achieved low FNMR often resulted in higher morph acceptance rates (NMPR/FMPR), indicating that synthetic-based calibration can inadvertently increase vulnerability to morph-based attacks.

Significance and Claims

The paper claims that while synthetic data is a valuable resource for system development and preliminary calibration, it is insufficient for reliable threshold selection in high-security deployments without validation against representative real-world data.

The authors emphasize that the primary limitation lies in the inability of current synthetic datasets to accurately reproduce the extreme quantiles of impostor score distributions required for low-FMR operation. Consequently, relying solely on synthetic data can lead to:

Overly permissive thresholds: Increasing the risk of false matches and security breaches.
Overly strict thresholds: Degrading system usability by rejecting genuine users.
Increased vulnerability: Creating exploitable gaps in defense against morph-based attacks.

The study concludes that reliable deployment in systems like EES requires validation and adjustment using real-world data that captures operational variability, particularly in the impostor tails. Synthetic data should be viewed as a tool for initial development rather than a substitute for final operational calibration.