A Lightweight Ensemble-Based Face Image Quality Assessment Method with Correlation-Aware Loss

Technical Summary: A Lightweight Ensemble-Based Face Image Quality Assessment Method with Correlation-Aware Loss

Problem Statement
Face Image Quality Assessment (FIQA) is a critical gatekeeper for face recognition and verification systems, particularly in uncontrolled, real-world environments ("in the wild"). While general-purpose no-reference image quality assessment (IQA) metrics (e.g., BRISQUE, NIQE) exist, they often fail to capture face-specific degradations such as pose, illumination, and occlusion, correlating weakly with actual recognition success. Conversely, state-of-the-art (SOTA) perceptual FIQA models typically rely on heavy deep learning architectures (e.g., large transformers or iterative diffusion loops) that are computationally intensive. This high inference cost (multi-GFLOP range) creates a bottleneck for deployment in resource-constrained environments like mobile devices or embedded systems, limiting their practical applicability despite high accuracy.

Methodology
The authors propose a lightweight, efficient FIQA framework designed to balance high accuracy with low computational cost. The methodology consists of three core components:

1. Lightweight Ensemble Architecture:
   The model integrates an ensemble of two compact Convolutional Neural Networks (CNNs): MobileNetV3-Small and ShuffleNetV2. Both networks are pre-trained on ImageNet and fine-tuned for the FIQA task by replacing their final classification layers with Multi-Layer Perceptrons (MLPs) to output a single scalar quality score.

   • MobileNetV3-Small: Uses a global average pooled feature vector of dimension 576, processed through a two-layer MLP.
   • ShuffleNetV2: Uses a global average pooled feature vector of dimension 1024, processed through a three-layer MLP.
   • Fusion Strategy: The final prediction is generated via a dual-level averaging scheme. First, predictions are averaged across multiple augmented views of the input image (Test-Time Augmentation). Second, the outputs of the two distinct models are averaged. This approach leverages the complementary representational strengths of the two architectures while maintaining a total parameter count of approximately 2 million.

2. Correlation-Aware Loss Function (MSECorrLoss):
   To align model predictions with human perceptual judgments, the authors introduce a joint loss function that combines Mean Squared Error (MSE) with a Pearson correlation regularizer.

   • The loss is defined as $\mathcal{L} = \mathcal{L}_{MSE} + \alpha \mathcal{L}_{Corr}$.
   • $\mathcal{L}_{MSE}$ minimizes the direct prediction error.
   • $\mathcal{L}_{Corr}$ is defined as $1 - \text{Pearson}(q, \hat{q})$, encouraging the model to preserve the monotonic ranking of quality scores relative to ground truth. This is crucial for perceptual tasks where relative ordering is often more important than absolute score precision.

3. Test-Time Augmentation (TTA):
   During inference, the input image is transformed into multiple augmented views (specifically random horizontal and vertical flips). The ensemble processes these views, and the final quality score is obtained by averaging predictions across both the augmentations and the models. This strategy enhances robustness and stability without requiring additional training data.

Key Contributions
The paper outlines three primary contributions:

• Compact Ensemble Design: A two-branch ensemble fusing MobileNetV3-Small and ShuffleNetV2, achieving high accuracy at a sub-million-parameter scale (approx. 2M parameters total) and low inference cost (~0.5 GFLOPs).
• MSECorrLoss: A novel objective function that simultaneously minimizes MSE and maximizes Pearson correlation with human ratings, tightening the alignment with perceptual ground truth.
• Robust TTA Integration: An effective inference-time strategy that averages predictions across augmented views, improving stability and reducing variance in quality assessment.

Experimental Results
The method was evaluated on the VQualA FIQA benchmark, a large-scale, in-the-wild dataset containing approximately 30,000 training images and 1,000 validation images. The evaluation utilized Spearman Rank Correlation Coefficient (SRCC) and Pearson Linear Correlation Coefficient (PLCC).

• Performance: The proposed method achieved an SRCC of 0.9829 and a PLCC of 0.9894, resulting in a Final Score of 0.9862.
• Comparison: These results significantly outperform existing SOTA approaches. For instance, the previous top-performing face-specific method, TOPIQ Swin Face, achieved a Final Score of 0.9286 (SRCC 0.9156, PLCC 0.9416). The proposed method surpasses this by a margin of 0.058 in the final score.
• Efficiency: Despite the performance gains, the model remains lightweight, suitable for real-world deployment where computational resources are limited.

Ablation Studies
The authors conducted ablation studies to validate the contribution of each component:

• Ensemble: Combining MobileNet and ShuffleNet improved the Final Score from ~0.97 (individual models) to 0.9792, demonstrating that ensemble learning reduces prediction variance.
• Loss Function: Switching from standard MSE to MSECorrLoss further improved the score to 0.9821, confirming the benefit of optimizing for correlation.
• TTA: Adding Test-Time Augmentation yielded the final optimal performance (0.9862), highlighting its role in reducing prediction variance and improving rank consistency.

Significance and Claims
The paper claims that its approach successfully addresses the trade-off between accuracy and computational efficiency in FIQA. By utilizing a lightweight ensemble and a correlation-aware loss, the method achieves state-of-the-art performance on the VQualA benchmark while remaining within the efficiency constraints required for mobile and embedded deployment. The authors position this work as a practical solution for "in-the-wild" scenarios where reliable, automatic, and face-aware quality judgment is essential for downstream tasks such as identity verification, access control, and forensic analysis. The work suggests that simple, well-orchestrated architectural and training strategies can advance robust FIQA systems without relying on computationally expensive backbones.