Disentangling the Impacts of Incomplete Lineage Sorting and Gene Tree Estimation Error on Species Tree Inference

This study demonstrates through systematic simulations that gene tree estimation error (GTEE) typically exerts a stronger detrimental impact on species tree accuracy than incomplete lineage sorting (ILS), despite comparable overall discordance levels, by revealing that GTEE generates uniform, high-entropy noise while ILS produces structured, constrained skew in quartet distributions.

The Gist

Imagine you are trying to reconstruct the family history of a group of animals (like birds) by looking at their DNA. You have thousands of different "chapters" from their genetic books (genes), and you want to piece them together to draw the correct family tree.

However, there's a problem: these chapters often tell different stories. Sometimes they disagree with each other, and sometimes they even disagree with the "true" family tree you are trying to find. This disagreement is called Gene Tree Discordance.

This paper investigates why these chapters disagree and which reason is more dangerous for getting the right answer. The authors found two main culprits:

1. The "Real" Confusion (Incomplete Lineage Sorting - ILS): This is a biological reality. Imagine a family where three cousins are born in quick succession. If you ask them, "Who is your closest relative?" they might genuinely be unsure because their ancestors hadn't fully separated yet. The DNA is actually mixed up due to how evolution works. This is ILS.
2. The "Bad Translation" (Gene Tree Estimation Error - GTEE): This is a technical mistake. Imagine trying to read a very blurry, smudged, or tiny piece of paper. You might misread a word or guess the wrong sentence structure. The DNA isn't actually mixed up; you just didn't have enough information to read it correctly. This is GTEE.

The Big Experiment: The "Twin" Test

The researchers wanted to know: Which is worse for building the family tree? Is it the real biological confusion (ILS) or the mistake caused by bad data (GTEE)?

To find out, they created a controlled experiment using computer simulations. They built two sets of "fake" family trees:

• Set A: Had the same amount of disagreement, but it was caused only by the real biological confusion (ILS).
• Set B: Had the exact same amount of disagreement, but it was caused only by bad data/blurry reading (GTEE).

They then asked their best computer programs (the "detectives") to solve the mystery using both sets.

The Shocking Discovery

The "Bad Translation" (GTEE) is much more dangerous than the "Real Confusion" (ILS).

Here is the analogy:

• ILS (Real Confusion): Imagine a group of people trying to solve a puzzle, but they are genuinely confused about a few pieces. If you give them more people (more genes) to help, they eventually figure it out. The more data you add, the clearer the picture becomes.
• GTEE (Bad Translation): Imagine the same group, but now everyone is wearing foggy glasses. They are all making the same kind of mistake because the data is too short or blurry. If you give them more people (more genes), they just get more people making the same wrong guess. Adding more data doesn't help; it just adds more noise.

The study showed that even when the "noise level" was identical, the family trees built from the "bad translation" data were much more wrong than those built from the "real confusion" data.

The "Quartet" Clue

How did they tell the difference? They looked at the "voting patterns" of the DNA.

• Under ILS: The votes are messy, but they are structured. The "correct" answer is still the most popular vote, just slightly less popular than it should be. It's like a noisy room where the right answer is still being shouted the loudest.
• Under GTEE: The votes are flat and random. The "correct" answer loses its voice. The votes are spread out evenly among wrong answers. It's like a room where everyone is whispering different wrong things, and the right answer is drowned out.

The Bird Case Study

To prove this wasn't just a computer game, they looked at a real dataset of 48 bird species (which are famous for having rapid, confusing evolutionary histories).

They found that:

• Short DNA sequences (Exons): These were like the "blurry glasses." They caused a lot of "bad translation" errors. When they used only these short sequences, the family tree was messy and wrong.
• Long DNA sequences (Introns): These were like "clear glasses." They had less error. When they used these, the family tree became much more accurate.

The Lesson: If you have a dataset with thousands of genes, but they are all short and "blurry," adding more of them won't help. You need to filter out the bad data and focus on the long, clear sequences.

Summary in One Sentence

While nature sometimes genuinely mixes up family histories (which we can fix with more data), bad data quality creates a type of confusion that tricks our computers, and adding more bad data only makes the mistake worse. To get the right family tree, we must distinguish between "real biological noise" and "technical errors."

Technical Summary

1. Problem Statement

Accurate species tree inference from genome-scale data is complicated by gene tree discordance, where individual gene trees disagree with each other and the underlying species tree. This discordance arises from two primary sources:

• Biological Process: Incomplete Lineage Sorting (ILS), where lineages fail to coalesce in ancestral populations under the Multi-Species Coalescent (MSC) model.
• Technical Factor: Gene Tree Estimation Error (GTEE), caused by finite sequence lengths, short internal branches, alignment errors, and model limitations.

While both factors degrade the accuracy of summary methods (which infer species trees from pre-estimated gene trees), their relative impacts and the structural patterns they impose on gene tree distributions remain poorly understood. Most pipelines treat discordance as a single signal, making it difficult to distinguish whether poor inference is due to biological complexity or estimation noise. This study addresses the gap by systematically isolating and comparing ILS and GTEE under matched conditions.

2. Methodology

The authors employed a controlled simulation framework to create datasets where discordance arose exclusively from either ILS or GTEE, ensuring "equivalent" levels of overall discordance for fair comparison.

• Simulation Design:
  • Datasets: Two datasets were generated: a 15-taxon (1,000 loci) and a 21-taxon (2,000 loci) dataset, each with 10 independent replicates.
  • GTEE-only Condition: True species trees were used as true gene trees (no ILS). Sequence alignments were simulated using AliSim at varying lengths (250, 500, and 1,000 bp) to induce different levels of estimation error. Gene trees were inferred using IQ-TREE3.
  • ILS-only Condition: Gene trees were simulated using SimPhy under the MSC model. Branch lengths were systematically scaled to match the average normalized Robinson-Foulds (RF) distance observed in the corresponding GTEE-only datasets.
• Species Tree Inference Methods:
  • ASTRAL and wQFM: Quartet-based summary methods (Maximizing Quartet Consistency).
  • PhyloNet: Minimize Deep Coalescence (MDC) criterion.
  • Greedy Consensus: A topology-only approach agnostic to discordance sources.
• Metrics & Analysis:
  • Accuracy: Measured via normalized RF distance between estimated and true species trees.
  • Distributional Analysis: Examined quartet frequencies, Empirical Cumulative Distribution Functions (ECDFs) of dominant quartet probabilities, skewness, kurtosis, entropy, and dispersion.
  • Tree Space Exploration: Sampled 20,000 neighboring species trees via Subtree-Prune-and-Regraft (SPR) to analyze quartet score landscapes.
  • Biological Validation: Applied findings to the avian phylogenomics dataset (48 taxa, ~144k loci), analyzing Exons, Introns, and Ultra-Conserved Elements (UCEs) based on sequence length and quartet support.

3. Key Contributions

• Systematic Disentanglement: The study is the first to rigorously isolate ILS and GTEE effects at matched discordance levels, revealing that they are not interchangeable sources of noise.
• Identification of Structural Signatures: The authors characterize the distinct statistical "fingerprints" of ILS versus GTEE on gene tree distributions.
• Empirical Framework: Provides a framework for distinguishing biological discordance from estimation error in real-world datasets, particularly for filtering noisy loci.

4. Key Results

A. Impact on Species Tree Accuracy (RQ1)

• GTEE is More Detrimental: Even at matched levels of gene tree discordance, GTEE causes significantly lower species tree accuracy than ILS.
  • Example: At high discordance with 200 genes, RF error rates were ~3.3–6.7% for ILS-only but jumped to ~20.8–28.3% for GTEE-only.
• Effect of Gene Number:
  • ILS: Accuracy improves as the number of genes increases (consistent with statistical consistency of methods like ASTRAL).
  • GTEE: Accuracy does not improve (and can worsen) with more genes if the error rate per gene remains high. GTEE introduces noise that cannot be averaged out, violating the assumptions of summary methods.

B. Impact on Gene Tree Distributions (RQ2)

• Quartet Structure:
  • ILS: Produces a structured, biased distribution. The true quartet topology remains the dominant (most frequent) topology in >99% of cases, even at high discordance. The distribution is highly skewed with low entropy.
  • GTEE: Produces a uniform, high-entropy noise. The true quartet often loses dominance (dropping to ~76% dominance at high discordance). The distribution is flatter, with reduced probability gaps between the true and alternative topologies.
• Tree Space Landscapes:
  • Under ILS, the true species tree consistently sits at the peak of the quartet score landscape.
  • Under GTEE, the quartet score landscape is distorted. Neighboring trees often achieve higher quartet scores than the true species tree, causing quartet-based methods to "overshoot" and infer incorrect topologies with high support.

C. Biological Case Study (Avian Dataset)

• Locus Length Matters: Shorter loci (Exons) exhibited flatter ECDFs and higher entropy, indicative of high GTEE. Longer loci (Introns) showed steeper ECDFs and lower entropy, indicating stronger phylogenetic signal.
• Filtering Improves Inference: When filtering the avian dataset to retain only the top 60% of genes based on quartet scores:
  • Exons were heavily filtered out (low reliability).
  • Introns and UCEs were retained.
  • Result: Species trees inferred from the filtered "high-score" set recovered more established clades (e.g., Otidimorphae, Cursorimorphae) than trees inferred from the full dataset or exon-only subsets.

5. Significance and Implications

• Re-evaluating Summary Methods: The study confirms that summary methods are highly sensitive to GTEE. Simply increasing the number of loci (gene sampling) is insufficient if the loci are short and prone to estimation error; increasing sequence length (reducing GTEE) is critical.
• Diagnostic Tools: The distinct statistical properties (entropy, skewness, ECDF shape) of gene tree distributions can serve as diagnostics to identify whether discordance in a dataset is driven by biological processes (ILS) or technical artifacts (GTEE).
• Data Curation Strategy: For phylogenomic studies, especially in rapid radiations, researchers should prioritize loci with strong, concentrated quartet support (e.g., longer sequences) and filter out loci with diffuse support to mitigate the disproportionate impact of GTEE.
• Method Development: Future species tree methods must be designed to be robust against the specific "flat" noise patterns introduced by GTEE, rather than just the structured noise of ILS.