Towards Emotion Recognition with 3D Pointclouds Obtained from Facial Expression Images

1. Problem Statement

Facial Expression-based Emotion Recognition (FER) is vital for Human-Computer Interaction (HCI) and mental health monitoring. However, current state-of-the-art methods rely heavily on 2D image data, which presents two critical challenges:

• Privacy Concerns: High-resolution 2D images contain identifiable biometric data (texture, skin tone), raising significant compliance issues with regulations like the EU AI Act and GDPR.
• Data Scarcity for 3D: While 3D pointclouds offer a privacy-preserving alternative by removing texture and retaining only geometric structure, there is a severe lack of large-scale, labeled 3D FER datasets. Existing 3D datasets (e.g., BU-3DFE) are small, often captured in controlled studio environments, and lack the "in-the-wild" diversity of 2D datasets like AffectNet.

The paper addresses the feasibility of using High-Frequency Wireless Sensing (HFWS) (e.g., Terahertz sensors in wearables) to generate continuous, privacy-aware 3D facial data, and proposes a method to overcome the data scarcity bottleneck by generating 3D training data from existing 2D datasets.

2. Methodology

The proposed framework consists of a data generation pipeline, a refinement process, and a deep learning evaluation strategy.

A. Data Generation (AffectNet3D)

To solve the scarcity of 3D data, the authors developed a pipeline to convert 2D images from the AffectNet dataset (400k+ images) into 3D pointclouds:

• Reconstruction Model: They utilize the FLAME (Faces Learned with an Articulated Model and Expressions) model, a statistical 3D face model.
• Depth Estimation: To enhance geometric accuracy, they combine ZoeDepth and Boost Your Own Depth algorithms to generate high-resolution depth maps from 2D RGB images.
• Process: 2D landmarks are regressed, back-projected to 3D, and fitted to the FLAME template. The resulting mesh is sampled to create a 3D pointcloud.

B. Data Refinement Pipeline

The raw generated pointclouds include the entire head (hair, neck), which introduces noise for FER tasks. The authors designed a specific refinement pipeline to isolate the facial region:

1. 2D Processing: Projections of the pointcloud are analyzed using mathematical morphology, Canny edge detection, and the Hough Circle Transform to detect eye orbits and calculate their centers and radii.
2. 3D Processing: Based on the eye centers, a cropping plane is constructed. Points are filtered using a geometric inequality ($Ax + By + Cz + D > 0$) to retain only the facial region, discarding the neck and hair.
3. Standardization: The refined pointclouds undergo Farthest Point Sampling (FPS) to standardize the number of points (downsampled to 512 centroids) for consistent model input.

C. Model Architecture and Training Strategy

• Base Model: PointNet++ is selected as the backbone for processing 3D pointclouds due to its efficiency and ability to learn hierarchical local/global features without requiring pre-trained 2D weights.
• Training Strategy:
  1. Pre-training: The model is trained on the generated AffectNet3D dataset (both raw and refined versions).
  2. Fine-tuning: The pre-trained model is fine-tuned on a small subset (10% or 25%) of the unseen, high-quality BU-3DFE dataset.
  3. Simulation of Wearables: To test HFWS viability, the authors apply region-based masking to the pointclouds, simulating the limited field of view of smart glasses or Head-Mounted Devices (HMDs).

3. Key Contributions

1. AffectNet3D Dataset: The creation of a large-scale 3D FER dataset derived from the public AffectNet 2D database using FLAME-based reconstruction, bridging the gap between "in-the-wild" 2D data and 3D model training.
2. Privacy-Aware Pipeline: A novel end-to-end pipeline that generates texture-free 3D pointclouds, reducing identity disclosure risks while preserving emotional geometric cues.
3. Refinement Methodology: A robust geometric filtering technique that isolates facial regions from full-head reconstructions, significantly improving model learning efficiency.
4. Feasibility of HFWS: Demonstration that models trained on synthetic 3D data and fine-tuned on small real-world 3D datasets can achieve high accuracy, even under simulated wearable constraints (partial occlusion).

4. Experimental Results

The study evaluated the approach using the BU-3DFE dataset as a benchmark.

• Baseline Performance: Training PointNet++ solely on the full BU-3DFE dataset (Oracle) achieved ~78.9% accuracy (with 512 points).
• Small Data Baseline: Training solely on 25% of BU-3DFE resulted in poor accuracy (~49.1%), highlighting the difficulty of learning from small 3D datasets.
• Proposed Approach (Fine-tuning):
  • Pre-training on Refined AffectNet3D (512 points) and fine-tuning on just 25% of BU-3DFE achieved 70.64% accuracy.
  • This result is comparable to the Oracle solution (only ~8% lower) and significantly outperforms the baseline trained only on the small BU-3DFE subset.
• Wearable Simulation: When simulating limited coverage (masking parts of the face), the proposed method maintained a high ratio of classification accuracy compared to the oracle, suggesting robustness for smart glasses applications.
• Efficiency: While pre-training on AffectNet3D is computationally expensive (approx. 1764 minutes), the fine-tuning step is rapid (8 minutes), making the system viable for deployment where user-specific data collection is minimized.

5. Significance and Future Work

• Regulatory Compliance: The work provides a pathway for compliant FER systems under strict privacy laws (GDPR/EU AI Act) by replacing identifiable 2D imagery with non-identifiable 3D geometry.
• Continuous Monitoring: It validates the potential of HFWS-enabled wearables (e.g., smart glasses using THz sensing) for continuous, non-intrusive emotion monitoring in healthcare and fatigue detection.
• Hardware Implications: The paper discusses the engineering challenges of THz transceivers (power, miniaturization) and suggests that current PointNet++ models can run in real-time on embedded NPUs (approx. 10ms latency).
• Future Directions: The authors suggest exploring advanced architectures (Graph CNNs, Transformers), integrating thermal data for better robustness, and optimizing THz hardware for better coverage and lower power consumption.

In conclusion, this paper successfully demonstrates that synthetic 3D data generation from 2D images is a viable strategy to overcome data scarcity in 3D FER, enabling the development of privacy-preserving, continuous emotion recognition systems for next-generation wearables.