💻 computer science

FaceLiVTv2: An Improved Hybrid Architecture for Efficient Mobile Face Recognition

FaceLiVTv2 introduces an improved hybrid architecture featuring a lightweight global token interaction module and a unified RepMix block to significantly enhance the accuracy-efficiency trade-off for mobile face recognition, achieving up to 30.8% faster inference than state-of-the-art lightweight methods while maintaining superior accuracy.

Original authors: Novendra Setyawan, Chi-Chia Sun, Mao-Hsiu Hsu, Wen-Kai Kuo, Jun-Wei Hsieh

Published 2026-04-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Novendra Setyawan, Chi-Chia Sun, Mao-Hsiu Hsu, Wen-Kai Kuo, Jun-Wei Hsieh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart security guard who can recognize anyone's face instantly. In the past, this guard needed a massive office, a huge library of books, and a team of assistants to do the job. This worked great in a data center (the cloud), but trying to put this guard in your pocket (on a mobile phone) was impossible—it was too heavy, too slow, and drained the battery instantly.

This paper introduces FaceLiVTv2, a new version of a "smart guard" designed specifically to fit in your pocket while still being incredibly accurate.

Here is the breakdown of how they made this magic happen, using some everyday analogies:

1. The Problem: The "Over-Engineered" Guard

Previous attempts to make face recognition work on phones had two main issues:

The "Heavy" Models: Traditional deep learning models were like a tank. They were accurate but too heavy for a phone to carry around.
The "Myopic" Models: To make them light, engineers built "Compact" models. But these were like guards with very short attention spans. They could see the nose and mouth clearly (local details) but couldn't understand how the whole face fits together (global context), especially if the person was wearing sunglasses, had a weird angle, or was aging.

2. The Solution: The "Hybrid" Guard

The authors created FaceLiVTv2, a hybrid model that combines the best of two worlds:

The Local Expert (CNN): Good at seeing small details like the shape of an eye or a scar.
The Global Thinker (Transformer): Good at understanding the whole picture and how features relate to each other.

The challenge was making the "Global Thinker" part light enough for a phone.

3. The Three Magic Tricks (Key Innovations)

A. The "Lite MHLA": The Efficient Translator

In the old version, the "Global Thinker" was like a translator who spoke to every single person in a crowd individually, one by one, using a complex script. It was accurate but took forever.

The Fix: They invented Lite MHLA. Imagine instead of a complex script, the translator uses a shortcut code. They project the information into a simpler format and use a "linear" (straight-line) connection to talk to everyone at once.
The Result: It's like switching from writing a novel to sending a quick text message. It loses almost no meaning but is 22% faster and uses much less memory.

B. The "RepMix": The Modular Toolbox

The old model had separate tools for "local details" and "global thinking" that didn't talk to each other well.

The Fix: They created a RepMix block. Think of this as a Swiss Army knife where the different tools (scissors, screwdriver, knife) are fused into one single, compact unit.
The Result: When the phone is actually running the app, it doesn't need to carry all the separate tools; it just uses the one fused unit. This makes the app run smoother and faster without losing any sharpness.

C. The "GDConv Head": The Smart Spotlight

Usually, when a model finishes looking at a face, it takes an "average" of everything it saw (like taking a photo of the whole room and blurring it to get the average color). This often washes out important details.

The Fix: They replaced the "average" with a Smart Spotlight (GDConv). Instead of blurring everything, the model learns to shine a light specifically on the most important parts of the face (like the eyes or jawline) and ignore the background noise.
The Result: The model becomes much better at recognizing faces even in bad lighting or when the person is looking away.

4. The Results: Faster, Smarter, Lighter

The authors tested this new guard on many different "obstacle courses" (datasets) involving:

People of different ages.
People with different poses (looking left, right, up).
Low-quality, blurry photos (like from a security camera).

The Verdict:

Speed: On an iPhone 15 Pro, FaceLiVTv2 is 22% faster than the previous version and up to 30% faster than its closest competitors.
Accuracy: It is more accurate than almost every other lightweight model, even beating some massive models that are 10 times bigger.
Efficiency: It does all this while using less battery and memory, making it perfect for real-time use (like unlocking your phone instantly).

Summary Analogy

If previous face recognition models were like a heavy truck (too slow for the city) or a scooter with no engine (fast but can't carry a heavy load), FaceLiVTv2 is like a high-performance electric motorcycle. It's light enough to zip through traffic (mobile devices), has a powerful engine to carry a heavy load (high accuracy), and uses a smart navigation system (Lite MHLA) to get there faster than anyone else.

This paper proves that you don't need a supercomputer to recognize faces; you just need a smarter, more efficient design.

1. Problem Statement

Face recognition (FR) on mobile and edge devices faces a critical trade-off between recognition accuracy and computational efficiency (latency, memory, and energy).

Limitations of CNNs: Traditional lightweight CNNs (e.g., MobileFaceNet, GhostFaceNet) are efficient but suffer from limited receptive fields, hindering their ability to model long-range dependencies (e.g., global facial structure) necessary for robust recognition under pose variations or occlusions.
Limitations of Transformers: Vision Transformers (ViTs) offer superior global context modeling but are computationally expensive due to the quadratic complexity of self-attention mechanisms, making them impractical for real-time mobile deployment.
Limitations of Previous Hybrid Models: Existing hybrid CNN-Transformer models (including the authors' previous work, FaceLiVTv1) still face inefficiencies. Specifically, FaceLiVTv1's Multi-Head Linear Attention (MHLA) used a two-layer MLP-style design with non-linear activations that introduced redundancy, and its embedding head relied on uniform Global Average Pooling (GAP), which failed to adaptively weight discriminative spatial regions.

2. Methodology: FaceLiVTv2 Architecture

The authors propose FaceLiVTv2, a co-designed hybrid architecture that optimizes the balance between local feature extraction and global context modeling. The architecture consists of three core innovations:

A. Lite Multi-Head Linear Attention (Lite MHLA)

This module replaces the original MHLA to reduce redundancy while preserving representational diversity.

Linear Token Projections: Instead of the two-layer MLP (Linear-GELU-Linear) used in v1, Lite MHLA employs a single linear mapping per head.
Affine Rescaling: It replaces heavy Layer Normalization with a lightweight Affine Rescale Transformation ( $\alpha \odot x + \beta$ ). Since the input is already normalized by Batch Normalization in the preceding RepMix block, full re-normalization is redundant. This affine transformation stabilizes training and recalibrates feature distributions with minimal memory overhead.
Complexity Reduction: By eliminating Q/K/V projections, softmax, and intermediate non-linearities, the complexity is reduced to approximately $O(N^2C)$ per block (where $N$ is token count and $C$ is channels), operating efficiently at low spatial resolutions (e.g., $4\times4$ or $7\times7$ ).

B. Unified RepMix Block

To strengthen the integration of local and global features:

Reparameterization: The RepMix block utilizes structural reparameterization. During training, it uses a parallel $3\times3$ depthwise convolution and a $1\times1$ depthwise convolution. At inference, these are merged into a single equivalent $3\times3$ convolution.
Unified Design: Unlike v1, where local and global branches were separate, FaceLiVTv2 unifies RepMix with Lite MHLA in the later stages (Stages 3 and 4). This allows the model to capture local structural patterns first, followed immediately by global token interaction, improving architectural coherence.

C. Global Depthwise Convolution (GDConv) Head

The embedding head is redesigned to replace uniform Global Average Pooling (GAP).

Adaptive Aggregation: A GDConv layer (with a kernel size matching the feature map, e.g., $4\times4$ ) is applied after the final encoding stage.
Mechanism: This layer learns channel-wise spatial weights, allowing the model to emphasize discriminative regions (e.g., eyes, nose) rather than treating all spatial locations equally. This results in more robust feature embeddings for open-set recognition.

D. Architectural Staging

The model follows a feature pyramid design:

Stages 1 & 2: Use RepMix and Feed-Forward Networks (FFN) to capture low-level spatial patterns.
Stages 3 & 4: Stack residual blocks combining RepMix, Lite MHLA, and FFN to model high-level global dependencies.

3. Key Contributions

Lite MHLA Module: A streamlined, activation-free linear attention mechanism that reduces parameter redundancy and memory overhead while maintaining global modeling capabilities.
Co-Designed Architecture: The integration of RepMix, Lite MHLA, and a GDConv-based embedding head creates a unified framework that outperforms predecessors in both accuracy and efficiency.
Comprehensive Benchmarking: Extensive evaluation across eight challenging benchmarks (LFW, CA-LFW, CP-LFW, CFP-FP, AgeDB-30, IJB-B, IJB-C, and TinyFace) demonstrating superior performance.
Real-World Validation: Profiling on diverse hardware (iPhone 15 Pro, Jetson AGX Orin, CPU, GPU) proving the model's suitability for real-time mobile and edge deployment.

4. Experimental Results

The paper reports significant improvements over state-of-the-art (SOTA) lightweight models:

Accuracy vs. Efficiency:
- FaceLiVTv2-S (4.62M params, 179 MFLOPs) achieves 96.69% mean accuracy across five benchmarks, outperforming GhostFaceNetV2-2 by 0.21% while being 30.8% faster on mobile devices.
- FaceLiVTv2-L (8.52M params) achieves 97.03% mean accuracy, narrowing the gap with heavy large-scale models (like ResNet200-TopoFR) to less than 0.7% while using 13.9x fewer parameters and 76.1x fewer FLOPs.
Latency Improvements:
- FaceLiVTv2 reduces mobile inference latency by 22% compared to FaceLiVTv1.
- It achieves speedups of up to 30.8% over GhostFaceNets and 20–41% over EdgeFace and KANFace on mobile platforms.
Low-Resolution Performance: On the TinyFace benchmark (surveillance-grade low-res images), FaceLiVTv2-XS (90 MFLOPs) outperforms MobileFaceNet and ShuffleFaceNet, demonstrating robustness without needing distillation or image restoration.
Ablation Insights:
- Replacing MHLA with Lite MHLA improved mean accuracy by +1.14% and reduced latency by 10%.
- Using 4 heads in Lite MHLA was found to be the optimal balance between diversity and computational cost.
- CosFace loss function yielded slightly better generalization than ArcFace for this architecture.

5. Significance

FaceLiVTv2 represents a significant step forward in mobile biometrics. It successfully addresses the "efficiency-accuracy" bottleneck by:

Democratizing Transformer Benefits: It brings the global context modeling capabilities of Transformers to resource-constrained devices without the heavy computational cost of standard attention mechanisms.
Practical Deployability: The use of structural reparameterization and affine rescaling ensures the model is not only accurate but also highly optimized for the memory-bound nature of mobile hardware.
Robustness: The model maintains high performance across diverse conditions, including age variation, pose changes, and low-resolution inputs, making it suitable for real-world security and authentication applications.

In conclusion, FaceLiVTv2 provides a practical, deployable solution for real-time face recognition, setting a new standard for lightweight hybrid architectures in computer vision.