Towards Standardized Light Field Quality Assessment: Hybrid Subjective Benchmarking and Objective Metric Evaluation

Technical Summary: Towards Standardized Light Field Quality Assessment

Problem Statement
The standardization of immersive media, particularly Light Field (LF) imaging under the JPEG Pleno framework, requires reliable, reproducible subjective quality assessment (QA) and objective metrics that remain accurate across diverse distortion types. Existing challenges include:

1. Limitations of Conventional Methods: Traditional rating-based subjective methods (e.g., Absolute Category Rating, Double Stimulus Impairment Scale) often lack sensitivity when evaluating high-quality stimuli or stimuli with small perceptual differences. Conversely, pairwise comparison (PC) offers higher discriminability but is prohibitively time-consuming for dynamic, multi-view LF content.
2. Inadequacy of Current Metrics: Conventional 2D image quality metrics (e.g., PSNR, SSIM) fail to capture the unique multidimensional characteristics of LFs, such as angular consistency, parallax stability, and view-dependent appearance. Furthermore, existing objective metrics often struggle when LF processing pipelines involve not just compression artifacts, but also distortions arising from view synthesis, interpolation, and neural reconstruction (e.g., 3D Gaussian Splatting).
3. Lack of Standardized Benchmarks: There is a need for a unified workflow that integrates benchmark generation, hybrid subjective evaluation, and objective metric analysis to serve as a reproducible baseline for the JPEG Pleno Part 7 standardization activities.

Methodology
The paper proposes a standardized workflow comprising three main components:

1. Stimulus Generation and Dataset Construction:

   • Source Material: Eight diverse source scenes (natural indoor/outdoor and synthetic) were selected to cover a wide range of spatial resolutions, angular sampling densities, and scene geometries.
   • Distortion Families: The benchmark includes 144 stimuli divided into three categories:
     • Coding-only: Stimuli compressed using JPEG Pleno Model (JPLM) with 4D Transform Mode and x265/HEVC (pseudo-video coding).
     • Interpolation-based: Sparse views encoded and decoded, with missing views synthesized using RIFE or SepConv++.
     • 3DGS Reconstruction: Sparse views encoded/decoded, with dense LF grids reconstructed using 3D Gaussian Splatting (3DGS).
   • This design allows for the evaluation of metrics against both traditional coding artifacts and modern reconstruction-induced distortions.

2. Hybrid Subjective Evaluation (DSCS+PC):

   • Stage I (DSCS): Observers rate distorted stimuli relative to a reference using a four-level Double Stimulus Comparison Scale (Same, Slightly worse, Worse, Much worse). This establishes a global, reference-anchored quality structure.
   • Stage II (Selective PC): Pairwise comparisons are conducted only for stimulus pairs that received identical ratings in Stage I (specifically within the "Slightly worse" and "Worse" categories). This targets perceptually ambiguous regions where rating scales lack resolution.
   • Stage III (Score Reconstruction): DSCS ratings and PC preferences are fused using a Thurstone Case V model to generate a continuous, fine-grained perceptual quality scale. This approach preserves the semantic clarity of ratings while recovering local ordering in ambiguous regions.
   • Validation: Reliability was verified through cross-cohort consistency (two labs) and leave-one-observer-out analyses, confirming the hybrid scores' reproducibility.

3. Objective Metric Evaluation:

   • A comprehensive set of Full-Reference (FR) metrics was evaluated, including classical signal-fidelity metrics (PSNR, SSIM, VIF), deep feature-based metrics (LPIPS, DISTS, ST-LPIPS), and immersive video extensions (IV-PSNR, IV-SSIM).
   • Aggregation Strategies: The study analyzed the impact of view-pooling strategies (arithmetic mean, Minkowski pooling, and Worst-X% pooling) on metric performance.
   • Statistical Analysis: Metrics were assessed using Spearman Rank-Order Correlation (SRCC), Pearson Linear Correlation Coefficient (PLCC), and Root Mean Squared Error (RMSE) against the hybrid subjective ground truth.

Key Results

• Subjective Scores: The hybrid DSCS+PC method successfully generated a continuous quality scale with high cross-cohort consistency (SRCC = 0.943 between labs). The selective PC stage significantly improved the resolution of ambiguous regions, yielding a log-likelihood gain of +572.92 over DSCS-only scores in leave-one-observer-out tests.
• Metric Performance on Coding-Only vs. Reconstruction:
  • Metrics performed substantially better on the coding-only subset (e.g., ST-LPIPS SRCC = 0.889) compared to the full benchmark including reconstruction artifacts (ST-LPIPS SRCC = 0.776).
  • Deep learning-based metrics (ST-LPIPS, LPIPS, DISTS) and feature-based metrics (CVVDP, IW-SSIM) generally outperformed classical pixel-based metrics (PSNR, SSIM, FSIM).
  • Classical metrics showed significant performance drops when view synthesis distortions (interpolation, 3DGS) were introduced, failing to capture geometric shifts and view-dependent inconsistencies.
• Impact of Pooling: View-pooling strategies significantly affected metric performance. Learning-based perceptual metrics showed improved correlation with subjective scores when using Worst-X% or Minkowski pooling, suggesting that localized poor views heavily influence perceived quality. In contrast, traditional fidelity metrics often performed best with simple mean pooling.
• Cross-Distortion Robustness: Metrics calibrated solely on coding-only distortions exhibited systematic bias (under-prediction of degradation) when applied to reconstruction-based stimuli, indicating a lack of generalizability across distortion families.

Significance and Claims
The paper claims to provide a reproducible and standardization-ready framework for fine-grained Light Field QA. Its primary contributions are:

1. Methodological Foundation: It establishes a hybrid subjective protocol (DSCS+PC) that balances efficiency with high discriminative power, suitable for the complex nature of LF content.
2. Benchmarking Baseline: It introduces a diverse dataset (144 stimuli) that stresses objective metrics with both coding and reconstruction artifacts, moving beyond narrow benchmarks focused solely on compression.
3. Diagnostic Insights: The analysis highlights critical limitations of current objective metrics, specifically their inability to generalize from coding-only to reconstruction-based pipelines and their sensitivity to view-pooling strategies.
4. Standardization Support: The work serves as a methodological basis for the JPEG Pleno Part 7 standardization activities, providing the necessary subjective ground truth and evaluation protocols to guide the development of future, more robust LF quality metrics.

The authors explicitly state that the paper does not propose a new objective metric but rather presents the standardized test procedure itself as the necessary infrastructure for evaluating and developing such metrics. The subjectively annotated dataset is made publicly available to support this goal.