💻 computer science

OccFace: Unified Occlusion-Aware Facial Landmark Detection with Per-Point Visibility

OccFace is a unified occlusion-aware framework for diverse human-like faces that jointly predicts facial landmark coordinates and per-point visibility by integrating local evidence with cross-landmark context.

Original authors: Xinhao Xiang, Zhengxin Li, Saurav Dhakad, Theo Bancroft, Jiawei Zhang, Weiyang Li

Published 2026-02-12

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Xinhao Xiang, Zhengxin Li, Saurav Dhakad, Theo Bancroft, Jiawei Zhang, Weiyang Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are playing a high-tech game of "Simon Says" with a digital character. To make the character move realistically—to wink, smile, or look surprised—the computer needs to know exactly where its "facial landmarks" are (the corners of the eyes, the tip of the nose, the edges of the mouth).

But there’s a problem: life isn't a perfect studio photo. Sometimes a hand covers the mouth, a strand of hair falls over an eye, or the character turns their head so far that one ear disappears from view.

Most current AI "gets confused" in these moments. It tries to guess where the hidden parts are, often resulting in "glitchy" movements—like a character’s mouth stretching toward a hand or an eye floating in mid-air.

The researchers created "OccFace" to solve this. Here is how it works, explained through three simple ideas:

1. The "Detailed Map" (The 100-Point Layout)

Most AI uses a basic map of the face, like a simple sketch with only a few dots. OccFace uses a much more detailed map with 100 points.

The Analogy: Imagine trying to navigate a city using only four major highways (the old way) versus using a detailed GPS that shows every side street, alleyway, and park (the OccFace way). Because the map is so detailed, the AI understands the "geometry" of the face much better, even if it's a cartoon, an animal, or a robot.

2. The "Detective" (The Occlusion Module)

This is the "brain" of OccFace. Instead of just guessing where a point is, the AI now asks itself two questions: "Where is this point?" and "Can I actually see it?"

The Local Detective: This part looks closely at a specific spot. If it sees a blurry mess (like hair), it says, "Hey, something is blocking this eye!"
The Context Detective: This part looks at the "big picture." If the character turns their head sharply to the left, the AI knows, "Based on the angle of the nose, the right ear must be hidden behind the head."
The Analogy: It’s like a detective at a crime scene. One detective looks through a magnifying glass at a single footprint (Local), while another looks at the whole room to see how the furniture is arranged (Context). By combining their notes, they get the full story.

3. The "Safety Filter" (Downstream Benefits)

Because OccFace explicitly says, "I think this point is hidden," it provides a "safety signal" to other programs.

The Analogy: Imagine you are a puppeteer. If a string on your puppet gets tangled or hidden, you don't just pull it blindly and risk breaking the puppet; you stop and realize, "I can't see that string right now."
Because OccFace tells the computer which points are "invisible," the computer can ignore those "glitchy" points. This results in much smoother animations—no more jittery mouths or floating eyes.

Summary

In short, OccFace isn't just trying to find facial points; it's learning to understand what is visible and what is hidden. This makes digital characters—from your favorite video game avatars to friendly robots—move with much more grace and realism, even when they are being partially covered up.

Technical Summary: OccFace: Unified Occlusion-Aware Facial Landmark Detection

OccFace is a novel framework designed for universal human-like facial landmark detection. Unlike traditional detectors that focus on realistic human faces, OccFace is built to handle a diverse spectrum of "human-like" entities, including stylized avatars, cartoon characters, anthropomorphic animals, and robots.

1. The Problem: Limitations of Current Detectors

The authors identify two critical gaps in existing facial landmark detection research:

Domain Limitation (Layout): Most current models use human-centric layouts (e.g., 68 or 98 points) that are insufficient for stylized characters. These layouts often lack expressive structures critical for non-human faces, such as ear contours, inner-mouth details, and fine-grained eye structures.
Occlusion Complexity: Existing methods handle occlusion implicitly. They struggle with two distinct types: external occlusion (hair, hands, accessories) and rotation-driven self-occlusion (where large head poses cause far-side facial features to disappear). Current models often fail to distinguish between a landmark being "hard to see" and a landmark being "genuinely occluded," leading to unstable downstream applications like animation.

2. Methodology: The OccFace Framework

OccFace adopts a heatmap-based backbone and introduces several architectural and supervisory innovations:

Unified 100-Point Layout: To ensure consistency across domains, the authors propose a denser 100-point schema. This includes standard facial features plus fine-grained eye anchors (pupils/irises), inner-mouth structures, and explicit ear contours.
Geometric Evidence Maps: To prevent landmark "drifting" in ambiguous or occluded regions, the model predicts auxiliary point maps (Gaussian kernels) and edge maps (semantic boundary curves). These maps act as geometric guides to reweight the landmark heatmaps, ensuring stable localization.
Occlusion Prediction Module: The model explicitly predicts a per-point visibility score ( $v \in [0, 1]$ $v \in [0, 1]$ ) using a dual-branch approach:
- Local Branch: Uses local neighborhood features to detect independent external occluders.
- Cross-Landmark Context Branch: Shares information across landmarks to capture correlated patterns (e.g., when a head turns, the entire far side of the face becomes occluded).
- Gated Fusion: A learnable gate ( $\alpha$ ) dynamically decides how much to rely on global context versus local evidence.
Landmark-Aware Masking (Training Strategy): To overcome the scarcity of manual occlusion labels, the authors use a synthetic augmentation technique. They apply random masks to images and derive "pseudo-visibility" labels based on the overlap between the mask and the ground-truth landmark heatmaps.

3. Key Contributions

OccFace Model: A unified framework that jointly predicts dense 100-point coordinates and per-point visibility.
Genie-Face Dataset: A large-scale, diverse dataset containing 15,475 images across multiple domains (avatars, mammals, robots, etc.) annotated with the 100-point layout and visibility.
Occlusion-Aware Evaluation Suite: A new set of metrics that separates performance into NME-vis (visible landmarks) and NME-occ (occluded landmarks), alongside classification metrics (Occ AP, F1, ROC-AUC) for visibility accuracy.

4. Results and Significance

Superior Robustness: Experiments on the Genie-Face dataset show that OccFace significantly outperforms state-of-the-art models (like ORFormer) in both localization accuracy and visibility prediction. It specifically excels in reducing error on occluded regions (NME-occ) while maintaining high precision on visible ones.
Ablation Insights: The study confirms that auxiliary geometric maps are vital for stability, and the dual-branch visibility head is essential for capturing the structured nature of self-occlusion.
Downstream Utility: The authors demonstrated that the explicit visibility signal directly improves avatar animation. By using visibility scores to filter or weight landmarks, they achieved smoother, more stable facial movements with significantly less jitter compared to traditional methods.

Significance: OccFace moves facial landmark detection from a "human-only" task to a "universal human-like" task, providing a robust foundation for the growing field of digital humans, gaming, and character animation.