A classification of structured coalescent processes with migration, conditional on the population pedigree

This paper extends coalescent theory to account for population subdivision and migration by conditioning on specific population pedigrees, revealing that traditional models fail to capture significant pedigree effects in most scenarios except for the well-known structured-coalescent model, with a notable exception involving intermittent gene flow.

The Gist

Imagine you are trying to figure out how a group of people are related to each other by looking at their DNA. Usually, scientists use a standard "family tree" model (called the Coalescent Model) that assumes everyone mixes freely, like a giant, well-stirred pot of soup. In this soup, it doesn't matter exactly who married whom; you just look at the average chances of two people sharing an ancestor.

However, in the real world, populations aren't always well-stirred soup. They are often divided into small villages (demes) where people mostly marry within their own village, with only occasional visitors from other villages.

This paper asks a crucial question: Does the specific, messy history of who married whom (the "pedigree") actually change the DNA patterns we see, or can we safely ignore it and just use the average "soup" model?

The authors found that the answer depends entirely on how the migration (travel between villages) happens. They tested four different scenarios, which we can explain using a "Traveling Party" analogy.

The Four Scenarios

1. The "Steady Stream" (Structured Coalescent)

• The Scenario: Imagine a steady, gentle stream of tourists moving between two large villages every day. The villages are huge.
• The Result: No Pedigree Effect.
• The Analogy: If you have a massive crowd and people are constantly trickling in and out, the specific path any single person took doesn't matter. The "average" model works perfectly. The specific family tree gets smoothed out by the sheer size of the population and the constant flow. You can safely use the standard scientific models here.

2. The "Infinite Villages" (Many-Demes Limit)

• The Scenario: Imagine a world with thousands of tiny villages, but the villages themselves stay small. People move between them, but the villages never grow big.
• The Result: Pedigree Effect Exists (but fades with size).
• The Analogy: If you pick two people from the same tiny village, their shared history matters a lot. Did their great-grandparents happen to be the same person? In a tiny village, that's a big deal. However, if the villages were to grow infinitely large, this specific history would stop mattering again. It's only a problem when the local groups are small.

3. The "Occasional Visitor" (Low-Migration Limit)

• The Scenario: Two villages, but people almost never travel between them. When they do, it's a rare, lonely traveler.
• The Result: Pedigree Effect Exists (but fades with size).
• The Analogy: If migration is super rare, the two villages are basically isolated islands. If you find a shared ancestor, it's likely because of a very specific, lucky (or unlucky) event in the past. Again, if the villages were huge, this wouldn't matter as much. But in small, isolated groups, the specific family tree changes the DNA story.

4. The "Sudden Pulse" (Rare-Migration Limit) — The Big Surprise

• The Scenario: Two villages that are completely isolated for decades, then suddenly, a massive event happens: a whole shipload of people from Village A arrives in Village B all at once, mixing their genes instantly. Then, silence for another 50 years.
• The Result: Strong, Persistent Pedigree Effect.
• The Analogy: This is the most interesting case. Imagine a party where everyone is isolated in separate rooms for years. Suddenly, a door bursts open, and 30% of the people from Room A rush into Room B, shaking hands with everyone.
  • Because this "pulse" happens so rarely but affects so many people at once, the specific timing and size of that event matter immensely.
  • Even if the villages are huge, this "pulse" leaves a permanent mark on the DNA. The standard "average" model fails here because it assumes migration is a steady drip, not a sudden flood. The specific history of when the flood happened changes the genetic outcome.

Why Does This Matter?

For decades, scientists have used the "Steady Stream" model (Scenario 1) for almost everything because it's mathematically easy. This paper tells us:

1. You're probably safe using the old model if you are studying large populations with steady migration (Scenario 1) or if the local groups are large enough (Scenarios 2 & 3).
2. You need a new model if you are studying:
   • Metapopulations: Systems of many small, isolated groups (like islands or fragmented forests).
   • Pulse Admixture: Events like sudden invasions, colonization events, or "bottlenecks" where a huge chunk of a population moves at once.

The Takeaway

Think of the "Pedigree" as the specific route a traveler took, and the "Coalescent Model" as the average weather of the journey.

• If you are walking through a busy city with thousands of people (Large Deme, Steady Migration), the specific route you took doesn't matter; the average weather is a good prediction.
• But if you are walking through a tiny, isolated cave (Small Deme) or if a sudden flash flood hits you (Pulse Migration), the specific route you took becomes the most important thing in the world. The "average weather" model will give you the wrong answer.

This paper provides a map for scientists to know when they can use the simple map and when they need to draw the detailed, specific route.

Technical Summary

1. Problem Statement

Traditional coalescent models are widely used to infer demographic history and genetic variation. However, these models typically marginalize over the population pedigree, meaning they average over all possible genealogical histories of individuals. They assume that as population size ($N$) tends to infinity, the distribution of gene genealogies conditional on a specific realized pedigree converges to the same distribution as the marginal (averaged) model.

The authors investigate whether this assumption holds for structured populations (subdivided into demes with migration). Specifically, they ask: Does the limiting coalescent process, conditional on a specific population pedigree, differ from the traditional pedigree-averaged coalescent model? If they differ, standard inference methods based on traditional models may be biased or invalid for certain population structures.

2. Methodology

The authors employ a rigorous mathematical framework based on Markov chain convergence and the separation of time scales (Möhle, 1998).

• Model Setup: They utilize a symmetric island model with $D$ demes, each of size $N$ (diploid, monoecious initially). Migration is governed by two parameters:
  • $m$: The fraction of gametes/offspring that are migrants.
  • $\alpha$: The probability that migration occurs in a given generation.
• The "Pedigree Effect" Test: To detect a pedigree effect, they analyze the conditional survival function $F_N(t)$ of pairwise coalescence times.
  • They calculate the mean $E[F_N(t)]$ (the marginal expectation) and the second moment $E[F_N(t)^2]$.
  • The variance is defined as $\text{Var}(F_N(t)) = E[F_N(t)^2] - (E[F_N(t)])^2$.
  • Criterion: If $\lim \text{Var}(F_N(t)) = 0$ as the relevant parameter tends to its limit, the conditional process converges to the marginal process (no pedigree effect). If the variance remains non-zero, the conditional process depends on the specific pedigree (pedigree effect exists).
• Two-Locus Analysis: To compute the second moment, they track two pairs of ancestral lines at two unlinked loci. This allows them to determine if lineages remain independent conditional on the pedigree.
• Four Limits: The study examines four distinct asymptotic limits of the model:
  1. Structured-Coalescent Limit: $N \to \infty$ with $m \propto 1/N$ (fixed $M=4Nm$).
  2. Many-Demes Limit: $D \to \infty$ with fixed $N$ and $m$.
  3. Low-Migration Limit: $m \to 0$ with fixed $N$.
  4. Rare-Migration Limit: $\alpha \to 0$ with fixed $N$ and $m$ (migration events are large but infrequent).
• Extensions: The analysis is extended to diploid migration (migration of offspring) and dioecy (separate sexes) to test robustness.

3. Key Contributions

• Classification of Pedigree Effects: The paper provides a definitive classification of when structured coalescent models are robust to the specific realization of the pedigree and when they are not.
• Identification of "Pulse" Effects: It identifies that "large" migration events (pulse admixture) occurring at random times create persistent dependencies on the pedigree, even in infinite population limits.
• Mathematical Rigor: It extends the "quenched" (conditional) coalescent theory to subdivided populations, providing explicit rate matrices and variance calculations for complex state spaces involving multiple loci and demes.

4. Results

The authors found that the presence of a pedigree effect depends critically on the limit taken and the initial state of the sample (whether genes are in the same deme or different demes).

Limit Scenario	Pedigree Effect?	Conditions & Details
1. Structured-Coalescent ($N \to \infty, m \to 0$)	No	The conditional and marginal processes are identical. This validates the standard structured coalescent model used in most software (e.g., MIGRATE, IMa2) for large demes.
2. Many-Demes ($D \to \infty, N, m$ fixed)	Yes (Weak)	A pedigree effect exists for samples starting in the same deme ($[••]$). However, this effect vanishes if $N \to \infty$ is also applied (returning to the structured-coalescent limit). For samples in different demes, no effect is found.
3. Low-Migration ($m \to 0, N$ fixed)	Yes (Weak)	A pedigree effect exists for samples starting in different demes ($[•][•]$). Like the many-demes case, this effect disappears if $N \to \infty$.
4. Rare-Migration ($\alpha \to 0, N, m$ fixed)	Yes (Persistent)	Strong Pedigree Effect. A non-zero variance persists even as $N \to \infty$. This occurs because migration happens in "pulses" (large fractions of the deme move at once) separated by long periods of isolation. The specific timing and size of these pulses affect the coalescence times of unlinked loci, breaking the independence assumed by marginal models.

• Robustness: These findings hold true for both gametic migration (gametes move) and diploid migration (offspring move), as well as for dioecious populations.
• Simulation Validation: Simulations confirm that for the Rare-Migration limit, the cumulative distribution functions (CDFs) of coalescence times vary significantly across different realized pedigrees, whereas in the Structured-Coalescent limit, they converge to a single deterministic curve.

5. Significance and Implications

• Validation of Standard Models: The study confirms that the widely used structured coalescent model is mathematically justified for populations with large deme sizes and continuous gene flow. Researchers can continue to use standard inference tools for these scenarios without worrying about pedigree-specific biases.
• Need for New Models: For metapopulations (many small demes) or populations experiencing pulse admixture/introgression (rare, large migration events), traditional models are insufficient. The "pedigree effect" implies that unlinked loci are not independent conditional on the pedigree, violating a core assumption of many likelihood-based inference methods (e.g., $\partial a \partial i$, fastsimcoal2).
• Interpretation of Genetic Data: In scenarios with rare, large migration events, the genetic variation observed is heavily dependent on the specific history of those events. Inference methods must account for the stochasticity of the pedigree (the "quenched" limit) rather than just the average expectation.
• Theoretical Insight: The results highlight that finite deme size is a primary driver of pedigree effects in subdivided populations, but large, rare events can sustain these effects even in infinite populations. This parallels findings in single populations with large offspring numbers (sweepstakes reproduction).

In summary, the paper delineates the boundary between regimes where traditional coalescent theory is robust and regimes where the specific history of the population (the pedigree) fundamentally alters the distribution of genetic ancestry, necessitating new conditional coalescent models for accurate inference.