Unveiling hidden features of social evolution by inferring Langevin dynamics from data

This paper proposes a stochastic differential equation framework to model social evolution as continuous-time dynamics, enabling the quantification of irreversibility, detection of exogenous perturbations, and imputation of missing data to overcome the limitations of deterministic approaches in analyzing historical trajectories.

The Gist

Imagine trying to understand the history of human civilization by looking at a series of disconnected, blurry snapshots. Traditional history often tries to connect these dots with straight lines, assuming that if we know the past, we can predict the future with a simple formula: "If X happens, then Y will follow."

This paper argues that history isn't a straight line; it's more like a drunk person walking through a foggy forest. They have a general direction they want to go (the "drift"), but they are constantly stumbling, swaying, and getting pushed by the wind (the "random noise").

The authors propose a new way to study history using a mathematical tool called Langevin Dynamics. Think of this as a "weather forecast for history." Instead of just saying "it will rain tomorrow," it tells you the probability of rain, how hard the wind might blow, and how likely it is that a sudden storm will change the entire path.

Here is a breakdown of their ideas using simple analogies:

1. The Core Idea: History as a "Wobbly Walk"

Most history books treat societies like billiard balls: if you hit them one way, they go that way. The authors say this is wrong. Societies are more like a leaf floating down a river.

• The Drift (The River's Current): This represents the big, structural forces. For example, as a society gets richer, it tends to become more democratic. This is the "current" pushing the leaf in a specific direction.
• The Diffusion (The Wobble): This represents the random chaos. A sudden war, a brilliant leader, a plague, or a bad harvest can push the leaf sideways, even against the current.
• The Innovation: Instead of ignoring the wobble as "noise," this paper treats it as a crucial part of the story. They use math to separate the "river current" from the "random wobble."

2. Three Superpowers of This New Method

By treating history as this "wobbly walk," the authors unlock three special abilities that old methods couldn't do:

A. Measuring the "Arrow of Time" (Irreversibility)

Imagine playing a movie of history backward. Would it look normal?

• The Analogy: If you drop an egg and it shatters, you know time is moving forward. You can't un-shatter the egg.
• The Paper's Finding: The authors created a score to measure how "one-way" history is. They found that history generally has a strong forward direction (like a river flowing downhill). However, they can pinpoint exactly when a country went "upstream" against the natural flow.
• Real-world Example: They found that during major crises (like the 2008 financial crash or the 2020 pandemic), countries temporarily stopped following their usual path and moved in chaotic, unpredictable ways.

B. Spotting the "Plot Twists" (Exogenous Perturbations)

Sometimes, a leaf gets hit by a falling branch. That's an external shock.

• The Analogy: If you are walking down a street and suddenly get pushed by a stranger, that's a "perturbation."
• The Paper's Finding: Their math can tell the difference between a "normal wobble" (like a slight change in economy due to weather) and a "massive shock" (like a revolution or a war).
• Real-world Example: They identified that the 1968 protests in France were a "statistical anomaly"—a huge, unexpected jump that the normal rules of history couldn't explain. It was a moment where the "wind" blew the leaf off the river entirely.

C. Filling in the Blanks (Probabilistic Imputation)

Historical records are full of holes. We might know what a country was like in 1900 and 1920, but nothing in between.

• The Analogy: If you see a car's tire tracks in the mud from point A to point B, you can guess the path it took in the middle, even if you didn't see it.
• The Paper's Finding: Instead of just drawing a straight line between the dots (which is wrong), they use their "wobbly walk" model to simulate thousands of possible paths the country could have taken. This gives a much more realistic picture of the missing years, accounting for the fact that history is messy and uncertain.

3. What They Actually Tested

The authors didn't just talk about theory; they tested this on two real datasets:

• Modern Politics (The "Rich World" Test): They looked at 149 countries from 1995 to 2022, tracking Democracy, Inequality, and Wealth.
  • Discovery: They found that in wealthy countries, high inequality acts like a "trap." It weakens the "current" that usually pushes societies toward democracy, making them much more likely to randomly slide backward into authoritarianism.
• Ancient Civilizations (The "Seshat" Test): They looked at ancient societies (like Rome, China, and Egypt) using a database of historical facts.
  • Discovery: They found that civilizations generally grow in a funnel shape (getting bigger and more complex). However, they could spot specific moments where a civilization suddenly "jumped" in complexity (like Rome building its bureaucracy) or collapsed (like the fall of the Western Roman Empire), distinguishing between a slow decline and a sudden crash.

The Big Takeaway

This paper doesn't claim to predict the future or say that history is random. Instead, it offers a better map.

Old maps said: "If you go here, you will end up there."
This new map says: "If you go here, you are likely to end up there, but there is a 20% chance a storm will blow you off course, and here is exactly where those storms happen."

It helps historians stop asking "Why did this happen?" and start asking "How likely was this to happen, and what kind of shock was needed to make it happen?" It turns history from a story of fixed destiny into a story of probabilities, chances, and structural currents.

Technical Summary

Technical Summary: Unveiling Hidden Features of Social Evolution by Inferring Langevin Dynamics from Data

Problem Statement
The paper addresses a fundamental limitation in quantitative history and political economy: the reliance on deterministic snapshots or average effects (e.g., regression-based methods) that fail to capture the continuous, inherently uncertain nature of historical trajectories. Existing "credibility revolution" tools (e.g., difference-in-differences, instrumental variables) are effective for estimating average causal effects but offer limited insight into the variability of historical paths, the accumulation of uncertainty over time, or the distinction between structural trends and contingent events. Furthermore, historical data is characterized by sparsity, irregular sampling, missing entries, and heterogeneous measurement quality. The authors argue that treating historical change as a stochastic process, rather than a deterministic one, is essential for distinguishing between aleatoric uncertainty (intrinsic randomness from hidden degrees of freedom) and epistemic uncertainty (incomplete information and measurement error).

Methodology
The authors propose a framework modeling the state of a society as a vector evolving according to a Stochastic Differential Equation (SDE) in continuous time. The state $x_t$ represents macroscopic variables (e.g., democracy, inequality, complexity), and its evolution is governed by:
$$dx_t = F(x_t)dt + \sqrt{2D(x_t)}dW_t$$
where $F(x_t)$ is the drift field (deterministic structural trends) and $D(x_t)$ is the diffusion matrix (magnitude and covariance of intrinsic stochastic fluctuations).

The framework involves three main technical components:

1. State Space Construction: Raw historical indicators (often categorical or ordinal) are mapped to a continuous, low-dimensional latent state space using dimensionality reduction techniques (e.g., PCA) and log-transformations to handle heteroskedasticity. This creates a coherent vector $x_t \in \mathbb{R}^d$.
2. SDE Inference: To handle sparse and irregular data, the authors employ two complementary inference methods:
   • Langevin Bayesian Networks (LBN): Used for the modern political economy dataset (Application 1). This method uses neural networks to approximate drift and diffusion, trained to match Kramers-Moyal coefficients (conditional moments). It utilizes Stochastic Weight Averaging-Gaussian (SWAG) to quantify epistemic uncertainty via an ensemble of models.
   • Nonparametric Gaussian Process SDE (npSDE): Used for the historical civilizations dataset (Application 2), which is extremely sparse. This method models drift and diffusion as Gaussian Processes and employs simulation-based likelihood optimization, making it robust to irregular sampling intervals without requiring dense data.
3. Analytical Capabilities: The framework unlocks three specific analytical tools:
   • Irreversibility: Quantifies the "arrow of history" by calculating the log-likelihood ratio between a forward trajectory and its time-reversed counterpart. This measures the degree of time-asymmetry and structural directionality.
   • Exogenous Perturbation Detection: Identifies transitions that are statistically improbable under the learned endogenous dynamics. This is done via a "normalized surprisal" metric, distinguishing true external shocks (e.g., wars, plagues) from rare but possible internal fluctuations.
   • Probabilistic Imputation: Uses the learned SDE to sample from the conditional distribution $P(x_t | x_0, x_T)$ to fill missing historical data, respecting the nonlinear dynamics and intrinsic volatility rather than using simple linear interpolation.

Key Results
The framework was applied to two distinct domains:

• Application 1: Evolution of Political Economy (Modern Era):

  • Using data from 149 countries (V-Dem, WID, Maddison), the authors inferred a drift field showing that high economic inequality creates a "destabilizing pressure" on democratic institutions, particularly in high-GDP contexts where structural resilience vanishes.
  • The diffusion field revealed that democracy scores exhibit high volatility, especially in high-inequality, high-GDP regimes, whereas inequality is relatively robust.
  • Irreversibility analysis confirmed a persistent global trend away from equilibrium but identified specific dips corresponding to major crises (Great Recession, COVID-19) where trajectories deviated from the structural drift.
  • Perturbation detection successfully flagged historical anomalies like the 9/11 attacks and the Mensalão Scandal, distinguishing between events that were structurally impactful and those that were statistically extreme.

• Application 2: Historical Civilizations (Seshat/Polaris Dataset):

  • Analyzing 27 Natural Geographic Areas (NGAs) over millennia using Scale and Computation as state variables.
  • The global drift field resembles a "funnel," pulling low-complexity polities toward higher complexity, with a plateau in gains at intermediate levels.
  • Case Studies:
    • Paris Basin: Identified a rapid, statistically surprising acceleration in "Computation" (institutional capacity) that was directionally structured (irreversible), suggesting a discrete institutional transition.
    • Latium: Distinguished between an "institutional takeoff" (directionally structured) and a later imperial contraction (statistically surprising but not strongly irreversible), suggesting the latter was an event-like disruption rather than a deterministic drift toward collapse.
    • Middle Yellow River Valley: Showed sustained drift with episodic contractions, highlighting intervals where periodization assumptions (e.g., dynastic transitions) merit closer historical scrutiny due to low transition probabilities.

Significance and Claims
The paper claims that adopting a stochastic perspective unlocks analytical capabilities unavailable to static or purely deterministic models. Specifically:

1. Unified Methodology: It provides a single framework to dissect stability, contingency, and dynamics, moving beyond the dichotomy of "structural determinism vs. pure chance."
2. Quantification of Uncertainty: It explicitly separates and quantifies aleatoric and epistemic uncertainty, allowing researchers to distinguish between genuine surprises and artifacts of poor model constraints.
3. Diagnostic Power: The metrics of irreversibility and perturbation serve as "triage" tools, guiding historians to specific intervals where data implies rapid reconfiguration, external shocks, or coding ambiguities, thereby prioritizing targets for qualitative investigation.
4. Philosophical Shift: The authors argue that treating social dynamics as stochastic reframes the concept of agency and responsibility, viewing history not as a single realizable path but as a distribution over possible trajectories shaped by structural constraints and irreducible contingency.

The paper maintains a modest stance, acknowledging that SDEs are reduced-form representations and that estimated drift/diffusion terms should not be interpreted as direct causal forces. Instead, they summarize conditional statistical tendencies that can generate testable hypotheses about underlying mechanisms and institutional constraints. The framework is presented as a complementary layer to existing social science methods, particularly valuable for handling the unique challenges of fragmentary and irregular historical data.