Posterior simulation-based calibration tests of phylogenetic dating methods

1. Problem Statement

Bayesian phylogenetic inference relies on complex computational machinery (e.g., MCMC algorithms) to estimate parameters such as node ages, substitution rates, and tree topologies. A critical challenge in this field is ensuring that these inference algorithms are correctly implemented and unbiased.

• Limitations of Prior SBC: Standard Simulation-Based Calibration (SBC) validates algorithms by sampling parameters from the model priors and checking if the posterior recovers them. However, this approach can fail to detect biases that only manifest in specific, rarely sampled regions of the parameter space.
• Phylogenetic Specifics: Phylogenetic analyses involve vast tree spaces and often suffer from model misspecification (e.g., imperfect tree priors or substitution models). Consequently, the true posterior distribution often occupies regions of parameter space that are never visited during prior sampling.
• The Gap: There is a need for a validation method that checks inference algorithms specifically within the region of parameter space occupied by the empirical posterior, particularly in the presence of model misspecification.

2. Methodology

The author employs Posterior Simulation-Based Calibration (Posterior SBC), a method that treats the empirical posterior as a new prior to test algorithmic consistency.

Core Workflow:

1. Posterior Sampling: Run MCMC on empirical data to obtain $n$ parameter draws from the posterior distribution $p(q|y)$.
2. Predictive Simulation: Simulate $n$ new datasets ($y''$) using the sampled parameters from step 1 (Posterior Predictive Distribution).
3. Augmented Inference: Run the inference algorithm again on a combined dataset consisting of the original empirical data ($y$) and the simulated data ($y''$). This generates an "augmented posterior" $p(q|y, y'')$.
4. Calibration Check: Compare the original posterior draws ($q'$) against the augmented posterior draws ($q''$). If the algorithm is unbiased, the rank of $q'$ within the set of $q''$ should be uniformly distributed (checked via Probability Integral Transform (PIT) scores and ECDF plots).

Experimental Design:
The study tested two distinct phylogenetic dating approaches in BEAST 2:

• Case A: Tip-Dated Analysis (Indo-European Languages)
  • Data: 1,336 cognate sets across 46 meanings.
  • Model: Covarion substitution, Optimized Relaxed Clock, Birth-Death Skyline tree prior with sampled ancestors.
  • Constraint: Topology was fixed to a summary tree to isolate dating performance.
  • Validation: Included posterior predictive simulations for tree metrics (length, height, sampled ancestors) to confirm model misspecification.
• Case B: Node-Dated Analysis (Tabanidae Horseflies)
  • Data: 1,174 non-stem 28s rRNA sites.
  • Model: Yule tree prior, HKY substitution, Uncorrelated Relaxed Clock.
  • Calibration: Three nodes calibrated with lognormal priors (offsets at 44, 112, and 130 Ma).

Control: The author also performed Posterior SBC using a prior predictive dataset (simulated from the prior) to ensure results were not artifacts of the specific empirical data.

3. Key Contributions

• First Application of Posterior SBC in Phylogenetics: This is the first study to apply Posterior SBC to validate phylogenetic dating methods, moving beyond the limitations of Prior SBC.
• Validation Under Misspecification: The study demonstrates that inference machinery remains well-calibrated even when the tree prior is demonstrably misspecified (evidenced by discrepancies between posterior and posterior predictive tree statistics).
• Theoretical Insight on Identifiability: The study provides empirical evidence supporting theoretical limits on the identifiability of node ages, showing that adding posterior predictive data does not necessarily increase precision.
• Software Validation: It confirms that the implementation of dating methods in BEAST 2 is free from significant inference machinery biases for both tip-dating and node-dating approaches.

4. Results

• Calibration Status: Both the Indo-European (tip-dated) and Tabanidae (node-dated) datasets showed good calibration across all parameters. The PIT scores followed a uniform distribution, and ECDF difference plots showed minimal deviation from the expected uniform line.
• Model Misspecification: Posterior predictive simulations confirmed that the tree models were misspecified (e.g., separation in tree length and sampled ancestor counts between posterior and predictive trees). Despite this, the inference remained unbiased.
• Precision and Identifiability (Crucial Finding):
  • While parameters like substitution frequencies and clock rates showed increased precision in the augmented posterior, node age estimates did not.
  • The augmented posterior distributions for node ages were virtually indistinguishable from the original posterior distributions.
  • Even when simulating data on extremely young (6ka) or old (13ka) trees from the posterior tails, the resulting augmented posterior for node ages did not shift; it simply recapitulated the original posterior.
• Robustness: These results held true even when the empirical data was replaced with a prior predictive dataset, suggesting the lack of precision gain is a fundamental property of the problem, not a data artifact.

5. Significance and Discussion

• Validation of Controversial Results: Given historical debates regarding the dating of major language families (e.g., Indo-European origins), this study provides strong evidence that previous results were not artifacts of software bugs or inference bias.
• Theoretical Limits of Precision: The lack of precision gain in node ages aligns with theoretical work (Yang & Rannala, 2006; Rannala & Yang, 2007) suggesting that node ages have a theoretical limit of identifiability. This is because data informs branch lengths in terms of substitutions, not time. Uncertainty in node ages persists due to uncertainty in calibrations and clock rates, regardless of the amount of sequence data or the age of the simulated tree.
• Methodological Caveats: The author acknowledges a degree of circularity in using MCMC to generate prior samples for the test (since the same algorithm is used for sampling and inference). Ideally, direct simulation (independent of MCMC) would be preferred, but this is currently infeasible for complex models like the Birth-Death Skyline with sampled ancestors.
• Future Directions: The paper calls for the development of direct simulators for complex phylogenetic models and the use of cross-software validation (e.g., generating priors in RevBayes and inferring in BEAST 2) to further eliminate circularity.

Conclusion: The study confirms that BEAST 2's phylogenetic dating methods are robust and unbiased, even under model misspecification. However, it reinforces the theoretical understanding that node age precision is fundamentally limited by the nature of the data and calibration priors, not by the inference algorithm itself.