Different, same, or arrhythmic: an empirically informed benchmark of differential rhythmicity methods

Technical Summary: Different, same, or arrhythmic: an empirically informed benchmark of differential rhythmicity methods

Problem Statement
Circadian rhythms are fundamental biological dynamics regulated by gene-regulatory networks, often studied via time-series RNA-seq data to understand how gene expression changes across conditions. While tools for differential expression (DE) analysis are well-established, methods for differential rhythmicity (DR) analysis—identifying genes that change their rhythmic properties (amplitude, phase, or presence) between conditions—lack standardized guidance. Previous studies have highlighted inconsistencies in tool outputs and a lack of overlap in rhythmic gene identification across labs. Furthermore, existing benchmarks often rely on biological datasets where the "ground truth" is unknown, or they fail to account for specific complexities of RNA-seq data, such as the mean-dispersion relationship and the confounding effects of concurrent differential expression. Consequently, researchers face uncertainty in selecting appropriate methods for their experimental designs.

Methodology
The author conducted a systematic benchmarking study of 12 DR methods derived from 5 software packages (including compareRhythms, diffCircaPipeline, diffCircadian, dryR, and limorhyde).

• Data Generation: The study utilized a synthetic model of RNA-seq count data generated using a negative binomial (NB) distribution. Crucially, the model incorporated empirically estimated mean-dispersion trends derived from multi-tissue circadian time-series data. The simulation included two conditions (control and intervention) with tunable parameters for:
  • Sampling resolution (2h, 4h, 6h).
  • Replicates per time point (2, 4).
  • Sequencing depth (5M, 20M, 50M).
  • Fraction of rhythmic genes and fraction of differentially expressed (DE) genes.
  • Amplitude and phase differences between conditions.
  • Ground truth was established for four rhythmic categories: "same" (conserved), "gain" (new rhythm), "loss" (lost rhythm), and "change" (altered amplitude/phase), alongside arrhythmic genes.
• Evaluation Framework: The 12 methods were categorized into two frameworks: Hypothesis Testing (HT), which typically involves sequential testing for rhythmicity followed by differential rhythmicity, and Model Selection (MS), which uses criteria like BIC to select the best-fitting model.
• Performance Metrics: Methods were evaluated using precision, recall, and the F1 score for three aggregated classes: Arrhythmic (AR), Same (R), and Differentially Rhythmic (DR). Global performance was assessed using the multiclass Matthews Correlation Coefficient (MCC) and macro F1 scores. The study also evaluated the control of false discovery rates (FDR) and the accuracy of gene set size estimation.
• Statistical Analysis: Performance was analyzed using factorial mixed-effects regression models to determine the impact of experimental parameters (e.g., replicates, dispersion) on method efficacy.

Key Results

• Overall Performance: A subset of methods consistently outperformed others. The top performers included four count-based HT methods (CR-edger, CR-deseq2, CR-voom, CR-cosinor), one normalized-data HT method (limorhyde), and one model selection method (dryR). Notably, a simple baseline method (cosinor regression on normalized data) performed comparably to more complex bespoke methods.
• Input Data: Count-based methods generally achieved better performance across the benchmark datasets compared to those requiring normalized, transformed input, although the top-performing HT methods were robust to input type.
• Class-Specific Performance:
  • Arrhythmic (AR): All methods performed exceptionally well in identifying arrhythmic genes.
  • Differentially Rhythmic (DR): HT methods generally achieved high precision but lower recall, while MS methods (like dryR) showed higher sensitivity (recall) but lower precision.
  • Same (R): Paradoxically, many methods performed worst in identifying genes with conserved rhythms, often misclassifying them as DR or AR.
• Impact of Experimental Design: Performance improved with increased replicates, higher sampling resolution, and stronger rhythm amplitudes. The fraction of rhythmic genes influenced model selection methods more than HT methods. Concurrent differential expression did not significantly alter the relative ranking of methods.
• Gene Set Estimation: Most methods systematically underestimated the size of DR gene sets (HT methods) or overestimated them (MS methods). The author found that the number of detected genes should not be used as an unbiased estimate of the true proportion of rhythmic classes.
• False Discovery Control: HT methods generally adhered to the nominal FDR threshold (0.05), whereas MS methods exhibited higher false discovery proportions across all classes.

Key Contributions and "Rules of Thumb"
The paper distills its findings into six practical guidelines for researchers:

1. Prefer Count-Based Methods: When raw RNA-seq counts are available, use methods that accept them directly.
2. Select Based on Rhythmic Fraction: compareRhythms (edger, deseq2, voom, cosinor) and limorhyde perform well generally, especially with low fractions of rhythmic genes. dryR performs better when a large fraction of genes is rhythmic.
3. Prioritize Precision for Biomarkers: For high-confidence target discovery, use HT methods (compareRhythms variants, limorhyde) to minimize false positives.
4. Prioritize Sensitivity for Enrichment: For functional enrichment analyses where recovering a larger fraction of true positives is critical, use high-sensitivity methods like dryR or compareRhythms (mod sel).
5. Consider Computational Speed: If computationally constrained, limorhyde and compareRhythms (voom) are among the fastest top-performing methods.
6. Avoid Quantitative Inference from Counts: Do not infer the prevalence of rhythm classes (same vs. altered) solely from the number of detected genes, as methods have varying sensitivities and biases.

Significance and Claims
The author claims that this study provides the first systematic, empirically informed benchmark linking DR method choice to specific experimental conditions and analytical goals. By moving away from unknown-ground-truth biological datasets to a controlled synthetic environment with known mean-dispersion trends, the study offers evidence-based rules of thumb to improve the reliability of translating time-series RNA-seq data into biological insight. The author explicitly states that their goal is to provide objective directions for method selection rather than to analyze the underlying algorithms in depth. They also note that the benchmarking framework (using the DSC package) and the synthetic data generation tools (fauxseq and pyfauxseq) are made available to facilitate future method development and community benchmarking. The study modestly acknowledges limitations, such as the assumption of sinusoidal rhythms and the focus on two-condition comparisons, but asserts that the resulting guidelines will make DR analysis more transparent and reproducible.