People Analytics Framework

Technical Summary: People Analytics Framework (PAF)

Problem Statement
Automated surveillance systems face significant challenges in identifying individuals when facial features are obscured, occluded, or absent from the camera frame (e.g., when a person's back is turned). Existing systems often struggle with low-resolution images, atmospheric turbulence, and varying illumination conditions common in long-range or indoor surveillance. Furthermore, current literature indicates a gap in handling images where the face is not visible, necessitating a robust solution that can identify individuals based on appearance alone. The primary objective of this research is to develop a comprehensive People Analytics Framework (PAF) capable of detecting, tracking, and verifying individuals in closed environments (such as universities, hospitals, and schools) regardless of whether their faces are visible.

Methodology
The proposed PAF is an end-to-end deep learning pipeline designed to process video streams from multiple cameras. The system operates through four integrated phases:

1. Detection and Tracking Module (DTM):

   • Video frames are captured at 30 fps and resized to 1000x600.
   • The system employs object detection and tracking algorithms to extract individual persons.
   • A "cleaning" process filters the extracted images, removing frames with a person-to-image ratio less than 2:1 or frames containing multiple people.
   • Extracted images are organized into folders based on tracker IDs, creating a database of person trajectories.
   • Model Selection: The authors compared combinations of YOLOv5 and YOLOv8 with DeepSort and ByteTrack. YOLOv5 combined with ByteTrack was selected as the optimal configuration, offering a balance of high detection accuracy (160 boxes detected) and acceptable processing time (2m 52.7s).

2. Face Detection and Recognition Module (FDRM):

   • Face Detection: Various detectors (Haar Cascade, SSD, MTCNN, RetinaFace) were evaluated. MTCNN was chosen as the primary detector due to its balance of accuracy and computational efficiency, particularly in handling low-resolution images where other models failed or produced excessive false positives.
   • Face Clustering & Verification: The system utilizes deep learning models (VGG16, FaceNet, DeepID, SFace, ArcFace) to extract feature vectors.
   • Loss Functions: The framework employs Triplet Loss and Cluster Loss to group similar faces and separate dissimilar ones. For verification, the ArcFace loss function is utilized to increase inter-class variation and reduce intra-class variation.
   • Model Selection: Among recognition models, SFace demonstrated the highest efficiency and accuracy (99.80% success rate with Euclidean distance and 0.47s average time), followed by ArcFace and VGG-Face.

3. Person Re-Identification Module (PRM):

   • This module addresses scenarios where faces are not visible by analyzing full-body appearance (clothing, body shape, gait).
   • It utilizes Deep Metric Learning (DML) approaches, specifically Omni-Scale Feature Learning (OsNet).
   • Model Selection: OsNet with Triplet Loss was selected, achieving a Rank-1 accuracy of 99.97% and a mean Average Precision (mAP) of 99.92% on the test dataset.

4. System Integration:

   • The modules are integrated into a unified pipeline. When a user queries a specific person (via image or name), the system searches the database.
   • If the face is visible, Face Verification is used. If the face is not visible, Person Re-Identification is employed.
   • The system merges data from different cameras and timeframes using tracker IDs to provide a complete trajectory of the individual.

Key Contributions

• Hybrid Framework: The paper proposes a unified framework that seamlessly combines face-based recognition with appearance-based Person Re-Identification (ReID), ensuring identification even when facial features are occluded.
• Operational Pipeline: A complete, tested pipeline is established for closed-area surveillance, moving from raw video capture to database registration and user query response.
• Empirical Model Selection: The study provides a comparative analysis of state-of-the-art models (YOLO variants, DeepSort/ByteTrack, MTCNN, ArcFace, SFace, OsNet) specifically for this surveillance context, identifying the most effective combinations for accuracy and speed.
• Database Architecture: The creation of a structured database that links person IDs, timestamps, camera locations, and trajectory data to answer "who, where, and when" queries.

Results
The system was tested using data collected from the authors' campus environment involving four cameras.

• Overall Accuracy: The integrated system achieved an overall success rate of 98.7%.
• Face Verification: The system achieved a 97.6% success rate in face verification.
• ReID Performance: The selected ReID model (OsNet with Triplet Loss) demonstrated high efficacy, with a Rank-1 accuracy of 99.97% and an mAP of 99.92%.
• Handling Occlusion: The framework successfully identified individuals whose faces were not visible in the image frames, relying on the ReID module with a matching accuracy of up to 97.6%.

Significance and Claims
The paper claims that the PAF successfully addresses the limitations of current surveillance systems that fail when faces are not visible. By integrating detection, tracking, face recognition, and ReID into a single operational framework, the system provides a robust solution for monitoring closed areas like campuses and hospitals. The authors emphasize that the system operates independently to perform monitoring and alerting.

The authors modestly acknowledge that dataset collection and labeling remain significant challenges, particularly regarding variations in camera distance, resolution, and angles. They note that while the current system performs well on their specific dataset, future work may involve unsupervised domain adaptation or semi-supervised learning to improve generalization to unseen environments with minimal manual labeling. The paper does not claim universal applicability to all outdoor or long-range scenarios but focuses on its effectiveness within controlled, private network environments.