Non-frontal face recognition using GANs and memristor-based classifiers

Technical Summary: Non-frontal Face Recognition Using GANs and Memristor-based Classifiers

Problem Statement
Deep learning-based face recognition systems have achieved high performance but incur substantial computational overhead, rendering them unsuitable for resource-constrained edge platforms (e.g., drones) operating under strict Size, Weight, and Power (SWaP) constraints. A critical limitation in real-world deployments is the degradation of recognition accuracy due to non-frontal pose variations. While individuals often provide frontal biometric images for reference, surveillance and mobile scenarios frequently capture subjects from arbitrary angles, leading to self-occlusion and significant accuracy drops. Conventional solutions often rely on heavy generative models or require high-bandwidth connectivity for cloud processing, which is impractical for mission-critical, offline edge applications.

Methodology
The authors propose a hybrid framework that integrates a lightweight Generative Adversarial Network (GAN) for pose frontalisation with a memristor-based neuromorphic classifier for efficient recognition. The system operates in two primary stages:

1. Face Reconstruction Unit (FRU): To mitigate pose variability, the system employs a convolutional GAN to synthesize frontal facial images from non-frontal inputs. The generator utilizes an encoder-decoder architecture with U-Net-style skip connections to reconstruct frontal views from profile inputs without relying on random noise vectors. The discriminator, based on a DCGAN-inspired architecture, evaluates the authenticity of the generated images against ground-truth frontal images. This adversarial process enables the synthesis of high-resolution frontal representations that conform to the target domain distribution.
2. Face Classification Unit (FCU): The synthetic frontal images are processed by a memristor-based Spiking Neural Network (SNN). The FCU extracts features via a conventional Convolutional Neural Network (CNN), converts these feature maps into spike trains, and maps them onto a memristor crossbar array. The core computation relies on a Winner-Take-All (WTA) mechanism where the neuron with the highest membrane potential determines the classification. The system is simulated using an empirical model of TiN/HfON/TiN memristor devices, derived from experimental measurements to ensure hardware-representative validation.
3. Open-Set and Incremental Learning: To address open-set scenarios where unknown identities appear, the framework incorporates a prototype-based incremental learning scheme. Class prototypes are initialized from training data and updated online using an Exponential Moving Average (EMA) strategy. This allows the system to adapt to new identities without retraining the entire network or suffering from catastrophic forgetting. Unknown identities are detected via distance thresholds and, upon sufficient consistency, registered as new classes.

Key Contributions

• Hybrid Architecture: The integration of GAN-based frontalisation with memristor-based neuromorphic computing to address the dual challenges of pose variation and computational constraints.
• Hardware-Empirical Validation: The development of an empirical memristor model based on experimental data (TiN/HfON/TiN devices) to simulate the FCU, bridging the gap between algorithmic design and hardware implementation.
• Open-Set Adaptability: Implementation of an auxiliary mechanism for unknown rejection and online learning, enabling the system to handle dynamic, evolving identity sets typical of real-world surveillance.

Experimental Results
The framework was evaluated on two datasets: the controlled CMU Multi-PIE dataset and the real-world DroneFace dataset.

• CMU Multi-PIE Dataset:
  • Baseline Performance: Direct classification of non-frontal images by the memristor FCU showed severe accuracy degradation as pose angles increased, dropping from 35.13% at ±15° to 7.44% at ±90°.
  • Proposed Framework: After frontalisation by the FRU, identification accuracy remained stable and high across all angles, achieving an average of 96.31% at ±15° and maintaining approximately 96.16% even at ±90°.
• DroneFace Dataset:
  • Baseline Performance: Recognition accuracy decreased with increasing altitude (and thus steeper angles of depression), falling from 69.85% at 3m to 40.47% at 5m.
  • Proposed Framework: The reconstructed images improved accuracy significantly, achieving averages of 73.30% (3m), 73.35% (4m), and 73.60% (5m). This represents a substantial improvement over the baseline, particularly at higher altitudes where pose variation is most severe.
• Incremental Learning: The system successfully demonstrated the ability to dynamically register new identities and adapt prototypes in an open-set environment without catastrophic forgetting.

Significance and Claims
The paper claims that the proposed framework offers a scalable and efficient solution for face recognition in dynamic, resource-constrained environments. By combining adversarial learning with memristive technology, the approach alleviates the computational bottlenecks of conventional AI while maintaining robustness against non-frontal pose variations. The authors assert that this method is particularly suitable for edge AI applications, such as drone-based surveillance, where low power consumption and the ability to operate without reliable connectivity are paramount. The results suggest that compact generative reconstruction combined with memristor-based classifiers can significantly enhance recognition performance under both constrained and dynamic imaging conditions.