Subjective and Objective Quality Assessment of Rendered Human Avatar Videos in Virtual Reality

Technical Summary: Subjective and Objective Quality Assessment of Rendered Human Avatar Videos in Virtual Reality

Problem Statement
As immersive Virtual Reality (VR) and Augmented Reality (AR) applications increasingly rely on high-fidelity human avatars (often termed "holograms" or digital twins) for social interaction, telepresence, and the "metaverse," the transmission of these data streams faces significant bandwidth constraints. Real-time communication requires volumetric compression to minimize latency, yet compression and rendering artifacts can severely degrade the perceptual quality of 3D human avatars. While Video Quality Assessment (VQA) models are essential for optimizing the trade-off between perceptual quality and data volume, existing algorithms are largely insufficient for this domain. Traditional 2D VQA methods fail to account for the complexities of depth perception, spatial resolution, and the dynamic nature of 3D meshes. Furthermore, the development of specialized VQA models for human avatars is hindered by a scarcity of comprehensive, large-scale datasets containing subjective quality judgments collected within immersive 6-DoF (six degrees of freedom) VR environments. Most existing 3D graphics databases rely on 2D displays, static objects, or limited distortion types, failing to capture the nuances of streamed, rendered human avatars.

Methodology
To address these gaps, the authors constructed the LIVE-Meta Rendered Human Avatar VQA Database and conducted a rigorous evaluation of both Full-Reference (FR) and No-Reference (NR) VQA models.

1. Database Construction:

   • Source Content: The study utilized 36 pristine human avatar videos sourced from Metastage, featuring diverse demographics, attire, and poses (standing and sitting). These were reconstructed as textured 3D meshes (approx. 20,000 triangles) with 2048×2048 texture resolution.
   • Distortion Simulation: A custom volumetric simulation-rendering pipeline was developed within a Unity environment. This pipeline simulated 20 distinct distortion settings to mimic real-world streaming artifacts, including:
     • Temporal Artifacts: Delays (100–400 ms) causing temporal self-occlusions.
     • Spatial Artifacts: Reduced color resolution (480p to 1600p) and frame rates (10–30 fps).
     • Combined Conditions: Three categories of bandwidth limitations (Moderate, Heavy, and Severely Limited) combining delays, depth errors, and resolution reductions.
   • Stimuli Generation: This process yielded 720 video sequences (36 sources × 20 distortion settings).

2. Subjective Data Acquisition:

   • Environment: The study was conducted in an immersive 6-DoF VR environment using Oculus Quest Pro headsets (1800×1920 per eye, 72 Hz).
   • Participants: 78 human subjects (aged 18–33) rated the videos.
   • Protocol: Using a single-stimulus Absolute Category Rating with Hidden Reference (ACR-HR) methodology, subjects viewed 120 randomized videos per session. Ratings were collected via a continuous 5-point Likert scale (Bad to Excellent) using VR controllers.
   • Data Processing: Subjective scores were processed using the SUREAL MLE-MOS (Maximum Likelihood Estimate - Mean Opinion Score) method to account for subject bias and inconsistency, generating reliable Mean Opinion Scores (MOS) and Differential MOS (DMOS).

3. Objective Model Evaluation:

   • The authors evaluated a wide range of state-of-the-art (SOTA) FR and NR models on the new database.
   • Preprocessing: To focus on the avatar, background pixels were excluded, and bounding boxes for the body and face were extracted using YOLO models.
   • Metrics: Performance was measured using Spearman Rank Correlation Coefficient (SRCC), Kendall Rank Correlation Coefficient (KRCC), Pearson Linear Correlation Coefficient (PLCC), and Root Mean Square Error (RMSE).
   • Models Tested: The study included traditional metrics (PSNR, SSIM), deep learning-based models (LPIPS, CONTRIQUE, CONVIQT, Re-IQA), and a new model proposed by the authors, HoloQA. HoloQA utilizes a Mixture-of-Experts approach, leveraging visual neuroscience, information theory, and self-supervised deep learning to capture both low-level pixel quality and high-level body-specific features.

Key Contributions

1. Largest Comprehensive Human Avatar VQA Database: The paper introduces the LIVE-Meta Rendered Human Avatar VQA Database, containing 720 distorted stimuli from 36 source videos, rated by 78 subjects in an immersive 6-DoF VR setting. This is currently the largest 3D graphics VQA database of its kind.
2. New Quality Predictor (HoloQA): The authors present HoloQA, a novel model designed specifically for rendered human holograms. It achieves state-of-the-art performance on the new database by combining traditional and deep learning features tailored to human body content.
3. Benchmarking Framework: The paper provides a comprehensive benchmark of existing SOTA FR and NR VQA models against human judgments in VR, demonstrating the limitations of general-purpose models and the necessity of domain-specific approaches.
4. Public Resource: The metadata for the database is publicly released to facilitate further research in VR/AR quality prediction and compression optimization.

Results

• Subjective Analysis:
  • Distortion Impact: The study found that color resolution and frame rate had a significantly greater impact on perceived quality than delay artifacts. Specifically, reducing color resolution below 1600p and frame rates below 20 fps caused noticeable quality degradation.
  • Delay Tolerance: Delays up to 400 ms had minimal influence on perceived quality, suggesting that bandwidth could be traded for latency reduction without severe perceptual loss.
  • Consistency: High inter-subject (SRCC ~0.95) and intra-subject consistency was observed, validating the reliability of the collected data.
• Objective Model Performance:
  • FR Models: Among Full-Reference models, CONTRIQUE, Re-IQA, and CONVIQT (unsupervised deep learning models) demonstrated strong generalization. The new HoloQA model achieved the best performance (SRCC: 0.9144, PLCC: 0.9489 for body-only; SRCC: 0.9201, PLCC: 0.9523 for face-only), outperforming traditional metrics and other deep learning approaches.
  • NR Models: In the No-Reference category, deep learning-based models (CONTRIQUE, CONVIQT, RAPIQUE, GAMIVAL) significantly outperformed handcrafted neurostatistic models (e.g., BRISQUE, TLVQM). Notably, TLVQM failed to generalize to human avatar videos, while HoloQA again led the performance metrics.
  • Face vs. Body: Models generally performed better when evaluated on face-only crops compared to body-only crops, highlighting the critical importance of facial fidelity in avatar perception.

Significance and Claims
The paper claims that the LIVE-Meta Rendered Human Avatar VQA Database is a critical resource for advancing the field of immersive media. By providing a dataset that captures the specific perceptual nuances of 6-DoF VR human avatars, it enables the development of more accurate VQA algorithms. These algorithms are essential for optimizing compression protocols and streaming workflows in the metaverse, ensuring high-quality user experiences under variable bandwidth conditions. The authors modestly note that while their findings are robust within the tested environment (Oculus Quest Pro), future work should explore diverse VR headsets and additional distortion types (e.g., color errors, surface noise) to further refine these models. The release of the database and the HoloQA model (post-peer review) serves as a foundational step toward perceptually optimized human avatar transmission.