📄 evolutionary biology

Distribution of Gene Tree Topologies with Duplication, Loss, and Coalescence

This paper presents theoretical results and probability calculations for the multispecies coalescent with duplication and loss (MSC-DL) model, demonstrating that the unrestricted interaction of these evolutionary processes generates a vast number of gene tree topologies and highlighting the significant implications for the accuracy of existing reconciliation and ortholog identification methods.

Original authors: Mishra, S., Hahn, M. W.

Published 2026-10-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mishra, S., Hahn, M. W.

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

To understand how life evolves, biologists often look at two different kinds of family trees. One is the species tree, which maps the history of entire groups of animals or plants, showing when one species split into two. The other is the gene tree, which traces the history of a single piece of DNA within those organisms. Usually, these two trees tell the same story, but sometimes they disagree. This disagreement, known as discordance, happens because genes have their own complex histories. Sometimes, a gene copy is duplicated, creating an extra version that evolves separately. Other times, a gene copy is lost entirely. Occasionally, gene lineages fail to merge at the expected time, a process called incomplete lineage sorting, which scrambles the order in which they appear on the tree. For decades, scientists have used computer programs to reconcile these differences, trying to figure out exactly when duplications and losses happened by comparing the gene tree to the species tree. However, most of these programs have relied on a simplified view of how genes behave, ignoring the messy reality of how gene copies merge and sort themselves out over time.

A new study by researchers at Indiana University challenges the way we count and interpret these evolutionary events. The team investigated a more complete model that allows gene duplication, gene loss, and the merging of gene lineages to happen all at once, without artificial restrictions. They found that when you let these processes interact freely, the number of possible gene tree shapes explodes in a way that standard methods never anticipated. In a simple scenario involving just two species and a single duplication event, traditional models predicted only three possible outcomes for the gene tree. The new model showed that nine distinct tree shapes are actually possible. When the researchers expanded this to six species, the number of possible tree shapes jumped to more than 19 million. Adding even a single gene loss event to the mix nearly doubled that number again. This means that the space of possible evolutionary histories is vastly larger than previously thought, and many of the trees that standard software dismisses as impossible are actually quite probable under the new model.

The researchers did not just count the trees; they calculated how likely each one is to appear. They discovered that the timing of a duplication event relative to when species split apart dramatically changes the shape of the resulting gene tree. If a duplication happens long before or long after a species splits, the gene tree tends to look simple and familiar. But if the duplication occurs close to the moment of speciation, the merging of gene lineages creates a wide variety of complex shapes. This finding has a direct impact on how scientists interpret evolutionary history. Because standard methods do not account for this complexity, they often misinterpret a single duplication and a bit of natural sorting as multiple, separate duplication and loss events. For example, a gene tree that the new model explains with one duplication and no losses might be incorrectly reconstructed by older methods as having two duplications and two losses. As the number of species in a study increases, these errors become more extreme, potentially leading researchers to believe that a genome has undergone far more chaotic change than it actually has.

The study also looked at the physical length of the branches on these gene trees, which represents the amount of time or genetic change that occurred. They found that the expected length of these branches changes depending on how often duplications happen. If duplications are frequent, the gene trees that survive without any duplications tend to be slightly shorter than we would expect, because longer trees have more time to accumulate extra copies and get filtered out. The researchers confirmed these theoretical predictions by running massive computer simulations, showing that their mathematical formulas matched the simulated data almost perfectly. This work suggests that the tools currently used to identify orthologs—genes in different species that descended from a common ancestor without duplication—may be mislabeling parts of the tree. If a program cannot distinguish between a node caused by a species splitting and a node caused by a gene merging, it might incorrectly assume a gene is a direct ortholog when it is not.

Ultimately, this research highlights a gap between the biological reality of gene evolution and the simplified tools used to study it. The authors argue that ignoring the interaction between duplication, loss, and the merging of gene lineages leads to a distorted view of evolutionary history. While the new model reveals a much more complex landscape of possibilities, it also suggests that simply looking at the shape of a tree is often not enough to determine exactly what happened. Questions like whether a duplication occurred before or after a species split, or whether a gene was lost, may be impossible to answer with certainty using only the tree's shape. The authors suggest that future methods will need to incorporate the length of the branches and use more sophisticated statistical approaches to navigate this vast space of possibilities. By acknowledging the full complexity of how genes duplicate, die, and merge, scientists can begin to build more accurate maps of the evolutionary past, avoiding the trap of seeing more events than actually occurred.

Technical Summary: Distribution of Gene Tree Topologies with Duplication, Loss, and Coalescence

Problem Statement
Evolutionary biology relies on reconciling gene trees with species trees to infer evolutionary histories. However, gene tree discordance arises from multiple biological factors, including incomplete lineage sorting (ILS), gene duplication, and gene loss. While many reconciliation methods exist to infer duplication and loss (DL) events, most rely on models that either exclude the coalescent process or restrict its interaction with duplication and loss. Consequently, these methods often misinterpret coalescent events (ILS) as spurious duplication and loss events. Recent models, such as the Multispecies Coalescent with Duplication and Loss (MSC-DL), allow for unrestricted interaction between these processes, but the resulting space of possible gene tree topologies and their probabilities remain poorly characterized. This paper addresses the gap in understanding the frequency, topology, and probability distribution of gene trees generated under the MSC-DL framework.

Methodology
The authors utilize the MSC-DL model, which treats gene family evolution as a unified process involving three coalescent processes: coalescence at the parent locus, coalescence at the daughter locus, and coalescence between the parent and daughter loci following a duplication event.

Model Mechanics: A duplication event inserts a sequence from a parent locus into a pre-existing daughter locus. The duplicated sequence follows the daughter locus's gene tree from the point of insertion. Crucially, the duplicated lineage must subsequently coalesce with the parent locus's gene tree at a branch existing after the time of duplication (the "attachment branch").
Analytical Approach: The authors formalize the space of gene tree topologies by enumerating all possible "duplication branches" (where a duplicate can insert into the daughter tree) and "attachment branches" (where the duplicate can coalesce back into the parent tree).
Calculations:
- They derive closed-form formulas for the number of distinct gene tree topologies generated by one duplication and one loss, considering all possible combinations of lineages and tree structures.
- They compute exact probability distributions for gene tree topologies and branch lengths in a two-species system with one duplication, conditioning on the time of duplication relative to speciation ( $t_d$ vs. $t_s$ ).
- They derive expected branch lengths for gene trees conditioned on a specific number of duplications, accounting for the Poisson process of duplication rates ( $\lambda$ ).

Key Contributions and Results

Expansion of Gene Tree Space: The study demonstrates that the space of possible gene trees grows factorially with the number of species when duplication and coalescence interact.
- For a species tree with $N=2$ tips, a single duplication can generate 9 distinct gene tree topologies, compared to only 3 in standard DL models.
- For $N=3$ , the number rises to 135.
- For $N=6$ , the number exceeds 19 million.
- Adding a single loss event nearly doubles the number of possible topologies (e.g., for $N=2$ , the total rises to 16; for $N=6$ , it exceeds 33 million).
Probability Distributions: The authors provide exact probability formulas for all 9 possible gene tree topologies in the two-species, one-duplication case.
- They show that probabilities depend heavily on the timing of duplication relative to speciation.
- When duplication occurs far before or far after speciation, the probabilities converge to standard DL results.
- However, when duplication occurs near the speciation time, coalescence generates a wide variety of topologies that are equiprobable or follow complex exponential distributions, many of which are impossible under standard DL assumptions.
Expected Branch Lengths: The paper derives the expected internal branch lengths for these gene trees as a function of the duplication rate ( $\lambda$ ).
- Results indicate that conditioning on a specific number of duplications (e.g., zero duplications) alters the expected height of the gene tree compared to standard coalescent expectations. Trees with zero duplications are expected to be slightly shorter because longer trees accumulate more duplications.
- These theoretical expectations were validated against simulations using the software dupcoal.

Significance and Implications
The paper argues that ignoring the interaction between coalescence, duplication, and loss leads to significant errors in evolutionary inference:

Reconciliation Errors: Standard DL reconciliation methods, which assume a single duplication in a two-species tree yields only three topologies, will incorrectly infer extra duplication and loss events to explain the additional topologies generated by coalescence. For example, a single duplication event in a three-species tree might be misinterpreted as four duplications and eight losses.
Ortholog Identification: Methods that rely on identifying speciation nodes to find orthologs may fail, as coalescent events (which form gene tree nodes) can be confounded with duplication events. The authors note that identifying speciation nodes may not be necessary for accurate species tree inference, but the misidentification of nodes remains a critical issue for orthology detection.
Methodological Recommendations: The authors strongly recommend the use of reconciliation methods that explicitly model the interaction of duplication, loss, and coalescence (e.g., MSC-DL frameworks) to avoid these systematic biases. They suggest that while simple topological inference may be insufficient to distinguish all histories, future approaches utilizing expected branch lengths or machine learning on the expanded tree space may offer solutions.

The work establishes a theoretical foundation for understanding the complexity of gene tree evolution under realistic models, highlighting that even simple scenarios (two species, one duplication) produce a vast and non-intuitive distribution of gene trees.

More like this