Tracing Neanderthal ancestry patterns through successive population expansions in Europe

This study utilizes a three-population simulation framework to demonstrate that the Southeast-to-Northwest gradient of Neanderthal ancestry in Europe was established by early hunter-gatherer expansion and the Neanderthal range limit, then preserved through subsequent Neolithic migrations despite significantly higher admixture rates between human populations compared to Neanderthal-human interbreeding.

The Gist

Imagine the history of Europe not as a straight line, but as a giant, multi-layered painting being created over thousands of years. This paper is like a detective story where scientists try to figure out how a specific "stain" of color—Neanderthal DNA—ended up spread across the canvas in a specific pattern, and why that pattern survived even when new artists (farmers) came along to repaint parts of the picture.

Here is the story of how modern humans, Neanderthals, and farmers interacted, explained through simple analogies.

1. The Setup: The "Ink Stain" on the Map

Long ago, modern humans (us) walked out of Africa and into Europe. On their way, they met Neanderthals, a different but related group of humans living there. They mixed a little bit, like two different colors of paint blending together.

Because this mixing happened while humans were walking north and west, the "Neanderthal paint" didn't spread evenly. Instead, it created a gradient:

• Southeast Europe (near the entry point): Less Neanderthal DNA.
• Northwest Europe (farther away): More Neanderthal DNA.

Think of it like a drop of ink falling into a river. The ink is thickest where it falls and gets slightly more diluted as it flows downstream, but in this case, the "river" of migration actually concentrated the Neanderthal DNA as humans moved further away from the mixing zone.

2. The Mystery: Why Didn't the Pattern Wash Away?

Here is the puzzle: About 8,000 years ago, a massive wave of farmers arrived from the Near East. They didn't just walk; they swept across Europe, mixing with the local hunter-gatherers. Usually, when a new group moves in, they scramble the genetic map, like a child mixing up a deck of cards.

You would expect this "Farmer Wave" to erase the neat Southeast-to-Northwest gradient of Neanderthal DNA. But it didn't! The gradient survived, even though the total amount of Neanderthal DNA went down slightly.

The Question: How did the farmers manage to keep the old pattern while changing the overall color?

3. The Investigation: The "Three-Layer Cake" Simulation

The authors built a super-computer simulation (a digital time machine) to test this. They created a "three-layer cake" model:

1. Layer 1 (The Bottom): Neanderthals (who eventually disappeared).
2. Layer 2 (The Middle): Hunter-Gatherers (who mixed with Neanderthals).
3. Layer 3 (The Top): Farmers (who arrived later and mixed with Hunter-Gatherers).

They ran thousands of scenarios, changing the rules: What if the farmers came from the North? What if they came from the West? What if the Neanderthals lived further north?

4. The Big Discoveries

A. The "Direction Matters" Rule

The simulation showed that the gradient survived because both groups moved in the same direction.

• The Hunter-Gatherers moved from the Southeast to the Northwest. This created the gradient.
• The Farmers also moved from the Southeast (Anatolia) to the Northwest.

The Analogy: Imagine a line of people walking across a field, carrying buckets of water.

• The first group (Hunter-Gatherers) walked from the start, picking up "Neanderthal water" along the way. By the time they reached the end, the buckets were full.
• The second group (Farmers) started at the same beginning point and walked the same path. They picked up the "Neanderthal water" from the first group as they passed them. Because they followed the same route, they inherited the same "fullness" gradient.

If the farmers had started from the West (Iberia) and walked East, the pattern would have been completely different (and would have looked like a mess). The fact that the pattern stayed the same proves the farmers followed the same "highway" as the hunter-gatherers.

B. The "Dilution" Effect

While the pattern stayed the same, the amount of Neanderthal DNA dropped.

• Why? The farmers started with a "cleaner" bucket (less Neanderthal DNA) because they came from a region where mixing with Neanderthals hadn't happened as much.
• As they mixed with the local hunter-gatherers (who had high Neanderthal DNA), they diluted the mixture.
• The Result: The whole continent ended up with slightly less Neanderthal DNA, but the "more here, less there" gradient remained intact.

C. The "Reproductive Wall"

The study also calculated how often these groups actually mixed.

• Neanderthals vs. Humans: They built a "high wall." Only about 0.6% of their interactions resulted in babies. They were very different species, so mixing was rare and difficult.
• Hunter-Gatherers vs. Farmers: They built a "low fence." About 5% of their interactions resulted in mixing. They were both modern humans, so they mixed much more easily.

5. The Takeaway

This paper tells us that demography (population movement) is a powerful artist.

You don't need complex natural selection or special mutations to explain why Neanderthal DNA is distributed the way it is today. You just need to understand the traffic patterns of ancient humans.

• The "traffic" of the first humans created the pattern.
• The "traffic" of the farmers followed the same road, preserving the pattern while slightly diluting the paint.

In a nutshell: The map of our ancient DNA is a fossilized record of where our ancestors walked. By studying the "stains" of Neanderthal DNA, we can reconstruct the exact routes humans took thousands of years ago, proving that the farmers didn't erase history; they just walked the same path and left their own mark on top of it.

Technical Summary

1. Problem Statement

Modern humans (MH) outside Africa carry approximately 2% Neanderthal DNA, a legacy of interbreeding during the initial expansion out of Africa. This introgression created a distinct Southeast-to-Northwest gradient of Neanderthal ancestry across Europe, increasing with distance from the Near East. While this gradient was established during the Paleolithic, it is surprising that it persisted through the Neolithic transition (~8,500 years ago), a period characterized by massive demographic shifts, population turnover, and the replacement of hunter-gatherers (HG) by farming populations (FA) from Anatolia.

The central scientific questions addressed are:

• What demographic mechanisms created the initial Neanderthal ancestry gradient?
• How was this gradient maintained despite the subsequent Neolithic expansion and admixture between HG and FA?
• Why did the overall level of Neanderthal ancestry decline during the Neolithic while the spatial gradient remained?
• What were the specific rates of hybridization (Neanderthal-HG) versus assimilation (HG-FA) and relative population sizes?

2. Methodology

The authors developed a novel three-population, spatially explicit simulation framework using a modified version of the program SPLATCHE3.

• Simulation Model:

  • Populations: The model simulates interactions between three layers: Neanderthals (NE), Paleolithic Hunter-Gatherers (HG), and Neolithic Farmers (FA).
  • Dynamics: It models two successive expansions. First, HG expand into Europe and interact with resident Neanderthals (hybridization). Second, after Neanderthal extinction, FA expand from Anatolia and interact with HG (assimilation).
  • Mechanisms: The model incorporates density-dependent competition, variable migration (stepping-stone and long-distance dispersal), and distinct interaction parameters:
    • $\gamma_H$: Hybridization rate (NE $\leftrightarrow$ HG).
    • $\gamma_A$: Assimilation rate (HG $\to$ FA).
    • $K_{ratio}$: Ratios of effective population sizes ($N_e$) between groups.
  • Genetic Tracking: The model tracks the proportion of Neanderthal-derived loci (ancestry) in HG and FA samples across a grid of Europe (75 $\times$ 53 cells, 100 km each) over 2,800 generations (70,000 to 0 YBP).

• Data Integration:

  • Empirical Data: The simulations were compared against a large paleogenomic dataset of 715 European individuals (102 HG, 613 FA) from the AADR database.
  • Ancestry Estimation: Neanderthal ancestry was quantified using F4-ratios ($\alpha_{f4}$), comparing test genomes to Altai Neanderthals, Vindija Neanderthals, Dinka (unadmixed sister), and Chimpanzee (outgroup).

• Statistical Inference:

  • Approximate Bayesian Computation (ABC): The authors used ABC to perform model selection and parameter estimation.
  • Model Choice: They tested scenarios varying the northern limit of Neanderthal range (55°N to 70°N) and the expansion origins of HG and FA (Southeast vs. Southwest).
  • Parameter Estimation: Using the best-fitting scenario, they estimated posterior distributions for hybridization rates, assimilation rates, and population size ratios.

3. Key Results

A. Origins of the Ancestry Gradient

• Northern Limit of Neanderthals: Model selection indicated that the observed gradient is best explained if the northern limit of the Neanderthal range was at 55°N (Posterior probability 56%). This aligns with archaeological evidence and habitat suitability models.
• Expansion Direction: The Southeast-to-Northwest gradient in HG is robustly reproduced only when HG expanded from the Southeast (Near East/Africa). Scenarios where HG expanded from the Southwest (Iberia) failed to reproduce the observed cline.

B. Persistence During the Neolithic

• Maintenance of the Gradient: The study demonstrates that the Neolithic Farmers (FA) "inherited" the pre-existing Neanderthal ancestry gradient established by HG.
  • When FA expanded along the same axis as HG (Southeast to Northwest), the gradient was maintained.
  • If FA expanded orthogonally, the gradient would have reversed or disappeared.
• Dilution Effect: The overall level of Neanderthal ancestry in FA (mean 2.08%) was significantly lower than in HG (mean 2.42%). This decline is attributed to dilution: FA originated from Anatolia, where pre-Neolithic populations carried lower levels of Neanderthal ancestry than Central European HG. As FA expanded and admixed with HG, the global average dropped, but the spatial pattern (higher in the NW) persisted.

C. Parameter Estimation (ABC)

• Hybridization Rate ($\gamma_H$): The rate of Neanderthal-HG hybridization was estimated at ~0.6% (mode), with a 95% HDI of 0.38–0.8%. This confirms strong reproductive isolation between the species.
• Assimilation Rate ($\gamma_A$): The rate of HG assimilation into FA was estimated at ~4.9% (mode), with a broad HDI (0.2–18.1%).
  • Key Finding: The HG-FA assimilation rate is approximately 15 times higher than the Neanderthal-HG hybridization rate, reflecting the lack of interspecific barriers between human sub-populations.
• Population Sizes:
  • $N_e$(HG) / $N_e$(Neanderthal) $\approx$ 7.7 (HG were significantly larger).
  • $N_e$(FA) / $N_e$(HG) $\approx$ 4.25 (Farmers were larger than Hunter-Gatherers).
• Long-Distance Dispersal (LDD): The proportion of LDD events in the Neolithic expansion was estimated to be very low (<0.8%) and had minimal impact on the ancestry gradient.

4. Key Contributions

1. Methodological Innovation: Developed a three-layered spatially explicit simulation framework (SPLATCHE3 extension) capable of modeling sequential population replacements and archaic introgression simultaneously.
2. Demographic Explanation of Gradients: Provided a purely demographic explanation for the persistence of the Neanderthal ancestry cline, showing it is a result of expansion direction and "inheritance" of the gradient by subsequent populations, rather than selection or new hybridization events.
3. Quantification of Interaction Rates: Directly compared and quantified the magnitude of interspecific hybridization (Neanderthal-Human) versus intraspecific assimilation (HG-FA), demonstrating that the latter was an order of magnitude higher.
4. Reconstruction of Neanderthal Range: Used genetic data to independently corroborate the archaeological estimate of the northern Neanderthal limit at 55°N.

5. Significance

• Demography vs. Selection: The study reinforces that complex spatiotemporal patterns of archaic ancestry can arise from neutral demographic processes (migration, expansion, and drift) without invoking natural selection as the primary driver.
• Insight into Human History: It highlights how ancient DNA patterns serve as a record of past migration routes and population interactions, allowing researchers to reconstruct expansion axes and estimate relative population sizes.
• Broader Applicability: The framework is adaptable for studying other periods of human history (e.g., Bronze Age steppe migrations) and interactions with other archaic groups (e.g., Denisovans), offering a powerful tool for reconstructing the genomic history of species interactions.

In conclusion, the paper demonstrates that the current distribution of Neanderthal DNA in Europeans is a "fossilized" demographic signature of the initial Out-of-Africa expansion, preserved and slightly diluted by the Neolithic transition, rather than a result of continuous selection or recent mixing.