External Validation and Interpretability Analysis of a Deep Learning Model for Forensic Age Estimation Across Brazilian and Romanian Populations

Technical Summary: External Validation and Interpretability Analysis of a Deep Learning Model for Forensic Age Estimation

Problem Statement
Forensic age estimation is a critical component of human identification in legal and humanitarian contexts. While dental radiography, specifically orthopanoramic radiographs (OPG), offers predictable age-related changes, traditional morphological and morphometric methods are time-consuming, operator-dependent, and prone to variability. Although recent Convolutional Neural Network (CNN) models have demonstrated high accuracy in internal testing, a significant reliability gap exists. Most existing models are developed and validated on population-specific datasets without rigorous external validation on independent cohorts acquired via different clinical protocols and hardware systems. Furthermore, the literature often overlooks "shortcut learning," where models exploit non-anatomical, equipment-specific features (e.g., image borders, orientation markers) to achieve high internal performance, leading to poor generalizability when applied to diverse populations. This study addresses the need to assess the generalizability and biological consistency of forensic AI models across distinct demographics and imaging environments.

Methodology
The study evaluated the generalizability of three CNN architectures—VGG-16, InceptionV4, and ResNet-18—trained on a large-scale Brazilian dataset and tested on an independent Romanian cohort.

• Datasets:
  • Training/Internal: 10,036 OPGs from a Brazilian population (Federal University of Pernambuco), aged 2.3 to 96.5 years. The dataset was split into training (n=8,028), validation (n=1,004), and internal test (n=1,004) sets.
  • External Validation: 150 independent OPGs collected from three clinical departments in Cluj-Napoca, Romania, using Vatech PCH-2500 and Planmeca Promax machines.
• Preprocessing and Training: Images were resized to 224 × 224 pixels and normalized using ImageNet statistics. Data augmentation (horizontal flipping) was applied only to the training set. Models were initialized with ImageNet pre-trained weights and trained for 20 epochs using the Adam optimizer (learning rate 0.001) and Mean Squared Error (MSE) loss. The primary evaluation metric was Mean Absolute Error (MAE).
• Interpretability and Robustness Analysis:
  • Grad-CAM: Applied to the best-performing model on 20 external validation radiographs to visualize feature activation across convolutional layers, specifically examining attention to anatomical structures versus non-anatomical artifacts.
  • Latent Space Analysis: 512-dimensional feature vectors were extracted and projected using Principal Component Analysis (PCA), Diffusion Maps, and PHATE to analyze the model's internal representation of age progression.
  • Dataset Shift Analysis: Normalized per-channel mean pixel intensities and standard deviations were compared between the Brazilian and Romanian datasets to quantify domain shifts.

Key Results

• Model Performance: ResNet-18 emerged as the superior architecture. It achieved an internal test MAE of 3.61 years. However, during external validation on the Romanian dataset, the MAE increased to 5.80 years. InceptionV4 and VGG-16 performed less effectively, with external validation MAEs of 6.42 and 6.36 years, respectively. All models exhibited a significant performance decline when applied to the external cohort.
• Grad-CAM Findings: Early convolutional layers correctly prioritized relevant anatomical landmarks (dental follicles, mandibular thickness, tooth roots). However, deeper decision layers frequently activated non-anatomical regions, including radiographic orientation markers ("R") and image borders. This pattern was consistent across adolescent, adult, and elderly patients, indicating a susceptibility to shortcut learning.
• Latent Space Analysis: Dimensionality reduction techniques (PCA, PHATE, Diffusion Maps) revealed that the learned feature representations followed a clear, continuous chronological trajectory. PCA showed the first principal component explained 95.17% of the variance, aligning with age progression. This suggests the model successfully captured biological aging patterns despite the performance drop in external validation.
• Domain Shift: Analysis confirmed a significant shift in image intensity distributions between the training and external datasets (e.g., mean channel values shifting from [-0.264, -0.141, 0.082] to [-0.037, 0.092, 0.314]), attributed to differences in scanner hardware, exposure parameters, and image processing algorithms.

Key Contributions

1. Rigorous External Validation: The study provides a critical assessment of forensic AI generalizability by testing a model trained on a Brazilian population against an independent Romanian cohort with different imaging hardware. It demonstrates that high internal performance does not guarantee external reliability.
2. Interpretability via Dual Analysis: By combining Grad-CAM with latent feature space analysis, the study disentangles the model's behavior. It confirms that while the model learns biologically relevant age trajectories (latent space), it simultaneously relies on non-anatomical shortcuts (Grad-CAM) that compromise robustness across different imaging environments.
3. Benchmarking Architectures: The comparison of VGG-16, InceptionV4, and ResNet-18 identifies ResNet-18 as the most stable and accurate architecture for this specific task, offering a baseline for future forensic AI development.

Significance and Claims
The paper posits that external validation is a foundational quality control metric for forensic AI, essential for assessing reliability, transparency, and judicial admissibility. The study highlights that AI models for forensic age estimation are highly vulnerable to population shifts and hardware-driven distribution changes. While the models can successfully map biological aging patterns, the presence of shortcut learning (reliance on non-anatomical features) limits their forensic applicability in heterogeneous settings. The authors conclude that without independent external validation and interpretability analysis, the medical reliability and generalizability of forensic AI predictions remain questionable. The findings underscore the necessity of rigorous validation protocols before deploying such systems in real-world forensic or clinical practice.