💻 computer science

Pipeline Optimisation for Real-World Masked Face Recognition: YOLOv8 Detection, Landmark-Guided Cropping, and Multi-Model Embedding Benchmarking

This paper presents a systematic pipeline for robust masked face recognition that combines a fine-tuned YOLOv8 detector with landmark-guided upper-face cropping, demonstrating through extensive benchmarking that this multi-stage optimization significantly outperforms single-model approaches, particularly when using FaceNet with cosine similarity.

Original authors: Hamza Zidoum, Shihab Al Busaidi

Published 2026-06-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hamza Zidoum, Shihab Al Busaidi

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Masked Face" Glitch

Imagine you have a very smart security guard (a computer system) who has spent years memorizing the faces of everyone in a building. This guard is excellent at recognizing people when they are wearing their normal clothes and have their whole face visible.

However, when the pandemic hit, everyone started wearing face masks. Suddenly, the guard got confused. The mask covered the nose, mouth, and chin—parts of the face the guard relied on heavily to say, "Yes, that's Bob!" The guard started failing, either missing people entirely or guessing the wrong person.

This paper asks: How do we fix the guard so they can recognize people even when they are wearing masks?

The Solution: A Three-Step Pipeline

The authors didn't just try to "retrain" the guard's brain from scratch. Instead, they built a three-step assembly line (a pipeline) to process the image before the guard looks at it. Think of it like preparing a meal before serving it to a picky eater.

Step 1: Finding the Face (The "Spotter")

The Problem: Old security cameras often use outdated software to find a face in a crowd. When a face is covered by a mask, these old programs get lost. They are like a child trying to find a specific toy in a messy room but only looking for the whole toy; if half is covered, they give up.

The Fix: The authors used a modern, high-tech "spotter" called YOLOv8.

The Analogy: Imagine a highly trained eagle that doesn't need to see the whole bird to know it's a bird. It can spot a bird just by seeing its head and eyes.
The Result: They taught this eagle specifically to look for masked faces. It became incredibly good at it, finding masked faces 91.9% of the time. In comparison, the old "child" (older software) only found them about 47% of the time.

Step 2: Cutting Out the Clutter (The "Chef")

The Problem: Even if the system finds the face, the mask is still there, blocking the view. It's like trying to read a book where someone has put a sticky note over the bottom half of the page. The text is still there, but it's messy and confusing.

The Fix: The authors used a tool called MediaPipe to act like a precise chef with a knife.

The Analogy: Instead of trying to read the whole messy page, the chef cuts out only the top half of the page (the eyes, forehead, and cheeks) where the text is clear and unobstructed. They throw away the sticky note (the mask) and the bottom half of the page.
The Result: By feeding the system only the clean, unmasked top part of the face, the recognition accuracy jumped massively. For their best system, accuracy went from a failing grade (46.5%) to a solid B+ (70.1%).

Step 3: The Comparison (The "Librarian")

The Problem: Once the system has a clean picture of the upper face, it needs to compare it to a database of known faces. There are different ways to do this comparison, like different ways to measure distance.

The Analogy: Imagine you are trying to match two puzzle pieces.
- Method A (Euclidean): You measure the straight-line distance between the edges.
- Method B (Cosine): You look at the angle and shape of the pieces, ignoring how big or small they are.
The Result: The authors tested three different "Librarians" (FaceNet, ArcFace, and VGG-Face). They found that the Cosine method (looking at the angle/shape) worked best for all of them. Interestingly, the "Librarian" with the smallest memory (FaceNet) actually did the best job, likely because it wasn't overthinking the details that were missing.

What They Discovered (The Takeaways)

Old software fails: If you use old detection tools on masked faces, they will fail miserably. You need a modern tool specifically tuned for masks.
Cutting is better than guessing: The biggest improvement didn't come from making the computer smarter; it came from cropping the image. Removing the blocked part of the face made the system much more accurate without needing to retrain it.
Less is more: A simpler, more compact computer model (FaceNet) handled the missing information better than the complex, heavy models.
Real masks are hard: The paper emphasizes that using real photos of people wearing real masks is crucial. Previous studies used fake, digital masks, which made the problem look easier than it actually is.

The Bottom Line

The paper concludes that you don't need to invent a completely new "super-brain" to recognize masked faces. Instead, you just need to optimize the process:

Use a sharp eye to find the face.
Cut out the messy, covered parts.
Compare the clean parts using the right measuring tool.

By doing these three simple steps, the system becomes reliable again, even when everyone is wearing a mask.

Technical Summary: Pipeline Optimisation for Real-World Masked Face Recognition

Problem Statement
The widespread adoption of face masks in post-pandemic environments has exposed critical vulnerabilities in standard face recognition systems. Models trained exclusively on unoccluded faces suffer significant performance degradation when the nose, mouth, and chin—regions heavily relied upon for detection and feature extraction—are covered. Existing literature often relies on synthetically overlaid masks, which fail to capture real-world variances in fabric, fit, and wearing style, leading to overestimated performance metrics. Furthermore, there is a lack of systematic, end-to-end benchmarking that integrates modern one-stage detectors, multiple embedding architectures, and preprocessing strategies specifically on real-world masked datasets.

Methodology
The authors propose a systematic, end-to-end evaluation pipeline tested on the Real-World Masked Face Recognition Dataset (RMFRD), which contains genuine masked images rather than synthetic overlays. The study utilizes a balanced subset of 10,000 images (5,000 masked, 5,000 unmasked) partitioned 80:20 by identity. The pipeline consists of five sequential stages:

Dataset Preparation: The RMFRD subset was standardized to 640×640 pixels. Annotations for masked and unmasked faces were created using LabelImg.
Face Detection (YOLOv8): A YOLOv8 detector was fine-tuned using transfer learning from COCO-pretrained weights. It was benchmarked against three baselines: Dlib (HOG+SVM), MTCNN, and RetinaFace. Detection success was measured by Image-level Success Rate at IoU ≥ 0.5 (ISR@0.5).
Face Recognition (Embedding Models): Three pre-trained embedding models were evaluated in a zero-shot protocol (no fine-tuning on RMFRD):
- FaceNet: 128-dimensional triplet-loss embeddings (Inception-ResNet backbone).
- ArcFace: 512-dimensional additive angular margin embeddings (ResNet backbone).
- VGG-Face: 512-dimensional softmax-trained embeddings (VGG-16 backbone).
Similarity Metrics & Augmentation: Recognition performance was tested using Cosine similarity, Euclidean distance, and Manhattan distance. The impact of photometric augmentation (rotation, brightness/contrast jitter, blur, translation, scaling) was also assessed.
Landmark-Guided Cropping: Using MediaPipe FaceMesh, the pipeline dynamically extracted 468 facial landmarks. Only the upper facial region (eyes, forehead, upper cheeks) was retained, while occluded lower-face regions (mouth, chin) were excluded. This cropped image was then fed into the recognition models.

Key Contributions
The paper addresses five specific gaps in the literature:

Real-World Validation: Moving beyond synthetic masks to evaluate on the genuine RMFRD dataset.
Detector Benchmarking: Providing the first systematic comparison of YOLOv8 against MTCNN, RetinaFace, and Dlib specifically for masked face detection.
Cross-Model Analysis: Benchmarking three distinct embedding architectures (FaceNet, ArcFace, VGG-Face) under identical occlusion conditions.
Metric Evaluation: Systematically comparing Cosine, Euclidean, and Manhattan distances across all models.
Quantified Preprocessing: Measuring the specific impact of landmark-guided cropping within an integrated pipeline.

Results

Detection Performance: The fine-tuned YOLOv8 achieved an overall ISR@0.5 of 95.76% and 91.9% specifically on masked faces. This significantly outperformed MTCNN (47.3%), RetinaFace (14.7%), and Dlib (4.7%). Statistical validation via McNemar's test confirmed these differences were significant ( $p < 0.001$ ).
Recognition Baseline: Under full mask occlusion without preprocessing, all models degraded significantly from their standard benchmarks. FaceNet achieved the highest Rank-1 accuracy at 46.5%, followed closely by ArcFace (45.8%) and VGG-Face (34.2%).
Similarity Metrics: Cosine similarity consistently outperformed Euclidean and Manhattan distances across all models.
Augmentation: Photometric augmentation yielded negligible benefits (changes within ±2 percentage points) and slightly degraded VGG-Face performance, suggesting that the inherent diversity of the real-world dataset acts as sufficient implicit augmentation.
Landmark-Guided Cropping: This technique produced the most substantial improvement. By isolating the unoccluded upper face, FaceNet's Rank-1 accuracy rose from 46.5% to 70.1% (+23.6 percentage points). ArcFace improved by 21.6 points (to 67.4%), and VGG-Face by 8.1 points (to 42.3%).

Significance and Claims
The paper concludes that robust masked face recognition is best achieved through multi-stage pipeline optimization rather than relying on single-model tuning or replacement. The authors claim that:

Domain-specific fine-tuning of modern detectors (like YOLOv8) is essential for handling occlusion, as models trained on full-face data (MTCNN, RetinaFace) fail significantly on masked inputs.
Compact embeddings (FaceNet's 128-dim) may be more resilient to occlusion than higher-dimensional angular embeddings (ArcFace's 512-dim) when lower-face features are missing.
Spatial preprocessing (landmark-guided cropping) is the single most effective enhancement, offering greater gains than metric selection or data augmentation, and can be implemented without retraining the recognition models.
The proposed pipeline (YOLOv8 detection → MediaPipe cropping → FaceNet → Cosine similarity) offers a practical, deployable solution for access control and surveillance in environments where mask occlusion is persistent.

The authors note limitations regarding the demographic narrowness of the RMFRD dataset (predominantly East Asian subjects), the exclusion of compound occlusions (e.g., masks plus eyewear), and the lack of large-scale inference speed testing. They suggest future work should focus on cross-dataset validation and compound occlusion scenarios.