Econometric vs. Causal Structure-Learning for Time-Series Policy Decisions: Evidence from the UK COVID-19 Policies

This study compares four econometric methods against eleven causal machine learning algorithms using UK COVID-19 data to demonstrate that while econometric approaches offer clear temporal rules, causal ML algorithms provide broader discovery capabilities by exploring a larger space of graph structures that often results in denser, more identifiable causal networks for policy decision-making.

The Gist

The Big Picture: Who is the Detective?

Imagine you are a detective trying to solve a mystery: Why did the number of COVID-19 cases in the UK go up and down?

You have a massive pile of clues (data) collected every day for two years: how many people were in parks, how many took the bus, when schools closed, and how many people got sick.

Your goal isn't just to see what happened together (e.g., "When schools closed, cases went down"). You want to know what caused what. Did closing schools cause the drop in cases? Or did both happen because of a third thing, like the weather?

To solve this, the researchers pitted two different types of "detectives" against each other:

1. The Econometric Detectives: These are the old-school experts. They are like accountants with a strict rulebook. They believe that in time-series data (data that happens over time), the past must cause the future. You can't have a future event causing a past event. They use mathematical formulas to find connections, but they are very strict about the timeline.
2. The Causal Machine Learning (ML) Detectives: These are the modern AI explorers. They are like a swarm of bees looking for patterns in a giant flower field. They don't care much about the timeline; they just look for the strongest statistical links between variables. They are very good at finding many connections, but sometimes they get confused about which way the arrow points.

The researchers asked: Which detective team is better at helping the government make policy decisions?

---

The Tools They Used

1. The Econometric Team (The Rule-Bound Accountants)

These methods (LASSO, LAR, JS, SIMONE) look at the data and ask: "If I know what happened yesterday, can I predict what happens today?"

• The Analogy: Imagine a train track. The train (the virus) can only move forward. These detectives only look for tracks that go from yesterday to today. They are great at ensuring the timeline makes sense, but they might miss some hidden shortcuts or complex loops because they are so focused on the rules.

2. The Causal ML Team (The Pattern-Seeking Bees)

These methods (Hill-Climbing, Tabu Search, etc.) look at the whole field of data at once.

• The Analogy: Imagine a spider web. These detectives look at every strand and try to figure out which one is pulling on which. They are very good at finding lots of connections. However, because they are so eager to find links, they sometimes draw a web that is too messy (too many lines) or draw a line that goes backward in time (which is impossible in reality).

3. The "Knowledge Graph" (The Map)

The researchers also had a "Gold Standard" map created by other experts. This map shows what we think we know about how the virus spreads (e.g., "Lockdowns reduce movement, which reduces infection"). They used this map to grade how well the detectives did.

---

The Investigation: What Happened?

The researchers ran both teams of detectives on the UK COVID-19 data. Here is what they found:

1. The "Messy Web" Problem

The Causal ML detectives found way more connections than the Econometric team.

• The Result: They drew a graph with hundreds of lines connecting variables.
• The Catch: While they found many "identifiable" effects (things they could calculate), the graph was so dense and messy that it was hard to trust. It was like a spider web so thick you couldn't see the fly in the middle. Also, because they didn't strictly follow the "past causes future" rule, some of their connections were physically impossible (like saying tomorrow's weather caused yesterday's rain).

2. The "Strict Accountant" Problem

The Econometric detectives drew much cleaner, simpler graphs.

• The Result: They respected the timeline perfectly. If they said "Schools closed," they knew it happened before the drop in cases.
• The Catch: They missed a lot of connections. Their graphs were so sparse (few lines) that they often couldn't find any causal effects to calculate. They were so careful not to make mistakes that they ended up saying "I don't know" too often.

3. The Winner? A Tie with a Twist

Neither team won outright.

• The ML team was better at finding potential answers, but the answers were often buried in a messy, confusing graph.
• The Econometric team was better at keeping the timeline logical, but they missed too many details.

The Best Insight:
When they looked at the specific policies that did work, both teams (eventually) agreed on one thing: Reducing travel and social gatherings works.

• Specifically, the data showed that reducing "Citymapper journeys" (people moving around) and "OpenTable restaurant bookings" (people eating out) lowered the risk of reinfection.
• This makes sense intuitively: If people stop mixing, the virus stops spreading.

---

The Takeaway for Policymakers

The paper concludes that we need a hybrid approach.

• Don't just use the AI: If you let the AI run wild, you get a messy map that might suggest impossible things (like future causing past).
• Don't just use the Accountants: If you are too strict, you might miss the subtle ways the virus spreads.

The Lesson:
To make good policy decisions during a pandemic (or any crisis), you need a system that combines the logic of the timeline (from Econometrics) with the pattern-finding power of AI (from Causal ML).

Think of it like building a house:

• The Econometric methods are the architect who ensures the foundation is solid and the walls are straight (respecting time and rules).
• The Causal ML methods are the interior designer who finds all the creative ways to connect rooms and maximize space (finding complex patterns).

You need both to build a house that is safe and functional. The paper suggests that future tools should try to combine these strengths so we can better predict which policies will save lives in the next crisis.

Technical Summary

1. Problem Statement

The paper addresses a critical gap in Causal Machine Learning (Causal ML): the difficulty of recovering causal structures from time-series data to inform policy decisions.

• The Challenge: Traditional ML models focus on associations, which are insufficient for policy design (e.g., determining if a lockdown causes a reduction in infections). While Causal ML algorithms (score-based, constraint-based, hybrid) exist, they are primarily designed for cross-sectional data and often struggle with temporal dynamics.
• The Alternative: Econometrics has a long history of analyzing time-series causality (e.g., VAR models, shrinkage methods). However, there is limited comparative research on whether these econometric methods outperform modern Causal ML algorithms in recovering causal graphs and identifying valid intervention targets.
• The Objective: To evaluate and compare the performance of four econometric methods against eleven traditional Causal ML algorithms using real-world UK COVID-19 time-series data. The goal is to determine which approach better recovers causal structures and supports policy decision-making (specifically regarding infection control).

2. Methodology

A. Dataset

• Source: Publicly available UK COVID-19 data (Jan 30, 2020 – June 13, 2022).
• Dimensions: 46 variables (including mobility indices, policy severity, testing capacity, hospitalizations, and vaccination rates) across 866 daily observations.
• Preprocessing:
  • Imputation: Missing data (up to 74% in some vaccination variables) was handled using a Kalman Filter within a state-space model to preserve temporal dynamics. Categorical variables were converted to dummy variables for imputation and reconverted.
  • Discretization: Continuous variables were discretized into three categories (Low, Medium, High) using k-means clustering. This was necessary for the bnlearn library's parameterization constraints (continuous nodes as children of discrete variables).
  • Lag Selection: A Vector Auto-Regressive (VAR) process was assumed. Despite information criteria (AIC, HQ, FPE) suggesting high lags (up to 17), the authors selected VAR(1) (one lag) to avoid parameter explosion and because the Partial Autocorrelation Function (PACF) showed the strongest signal at lag 1.

B. Algorithms Evaluated

The study compared two families of methods:

1. Econometric Methods (4):

   • LASSO (Least Absolute Shrinkage and Selection Operator): Uses L1 regularization to shrink coefficients to zero, identifying sparse dependencies.
   • LAR (Least Angle Regression): Similar to LASSO but uses a different selection path; also produces sparse graphs.
   • JS (James-Stein Shrinkage): Transforms correlation networks into DAGs using partial correlations and time-ordering to direct edges.
   • SIMONE (Statistical Inference for Modular Networks): Infers network modules (clusters) and uses a VAR process to learn structure, optimizing for modularity.
   • Model Averaging: A meta-approach combining the graphs from the four methods above, retaining edges that appear in at least two models (optimized via BIC).

2. Causal ML Algorithms (11):

   • Constraint-Based: PC-Stable, GS, IAMB, Fast-IAMB, Inter-IAMB, IAMB-FDR.
   • Score-Based: Hill-Climbing (HC), Tabu Search.
   • Hybrid: H2PC, RSMAX2, MMHC.
   • Note: These algorithms were implemented using the bnlearn R package with standard hyperparameters (e.g., mi-g-sh test, ebic-g score).

C. Evaluation Metrics

The performance was assessed using five distinct criteria:

1. Structural Hamming Distance (SHD): Measures the structural difference between the learned graph and a "Knowledge Graph" (a benchmark based on domain expert consensus).
2. Model Complexity: Number of free parameters and number of edges (density).
3. Goodness of Fit: Bayesian Information Criterion (BIC) and Log-Likelihood (LL).
4. Identifiability: The number of causal effects (specifically from 12 mobility variables to 3 infection variables) that can be identified by the graph structure.
5. Policy Direction Accuracy: The frequency with which the identified causal effects align with common knowledge (e.g., reduced mobility $\rightarrow$ reduced infections).

3. Key Results

Structural Learning Performance

• Econometric vs. ML: Econometric methods generally produced graphs with higher SHD (worse structural match to the knowledge graph) and lower BIC/LL scores compared to Causal ML algorithms.
• Density Trade-off:
  • Score-based ML (HC, Tabu): Produced the densest graphs (697–699 edges) and achieved the best BIC/LL scores, but at the cost of high complexity and potential overfitting. They identified the most causal effects (27 each).
  • Constraint-based ML: Produced sparse graphs with competitive BIC scores.
  • Econometric Methods: Generally produced sparser graphs than HC/Tabu but denser than constraint-based ML. SIMONE performed best among econometric methods regarding fit (BIC/LL) and sparsity.
• Temporal Constraints: Econometric methods inherently enforced a "past-to-present" temporal structure (DAGs derived from VAR). Causal ML algorithms did not enforce this, leading to graphs that might violate temporal logic despite having better statistical fit.

Policy Evaluation (Intervention Analysis)

• Identifiability: Only HC and Tabu (score-based) identified a substantial number of causal effects (27 each), though only ~50% aligned with common knowledge.
• Econometric Findings:
  • LASSO, LAR, SIMONE, and the Average model identified zero causal effects that matched the direction of common knowledge.
  • JS (James-Stein) identified 2 relevant causal effects that matched common knowledge:
    1. Reducing Citymapper journeys $\rightarrow$ Reduces reinfections.
    2. Reducing OpenTable restaurant bookings $\rightarrow$ Reduces reinfections.
• Consistency: The effects identified by JS were also found by HC and Tabu, suggesting these specific interventions are robust signals. The magnitude of the effects was modest but directionally consistent with the mechanism that close-contact interactions drive transmission.

4. Key Contributions

1. Comparative Framework: Provides one of the first rigorous comparisons between econometric time-series methods and modern Causal ML algorithms for causal structure discovery.
2. Code & Reproducibility: Offers code to translate econometric results into the bnlearn Bayesian Network library, facilitating the integration of econometric rules into Causal ML pipelines.
3. Policy Insight: Demonstrates that while score-based ML finds more "identifiable" effects, they often come with dense, complex graphs. Econometric methods, particularly those using shrinkage (JS), offer clearer, more interpretable rules for temporal structures, even if they recover fewer total edges.
4. Temporal vs. Statistical Fit: Highlights a trade-off: Causal ML algorithms often optimize for statistical fit (BIC/LL) by ignoring temporal constraints, whereas econometric methods enforce temporal logic at the cost of statistical fit metrics.

5. Significance and Conclusion

The study concludes that neither family of methods is universally superior; they possess complementary strengths:

• Causal ML (Score-based): Excels at exploring a vast space of graph structures to find many potential causal links, but risks producing overly dense, uninterpretable graphs that may violate temporal logic.
• Econometric Methods: Provide clear, rule-based temporal structures and are better at penalizing complexity, making them potentially more reliable for "stress-testing" policy interventions, even if they recover fewer edges.

Implication for Policymakers:
For time-series policy decisions (like pandemic management), models that explicitly encode temporal precedence and penalize complexity (like the econometric approaches) may be more reliable for decision-making than purely data-driven, dense graphs. The study suggests that future Causal ML research should incorporate econometric ideas (shrinkage, modularity, and explicit temporal constraints) to improve the reliability of causal discovery in dynamic environments.

Limitations:
The study notes limitations including non-normal residuals, residual autocorrelation, potential omitted variables, and the necessity of discretization, which may attenuate causal signals. The "Knowledge Graph" used for benchmarking was static and not designed for dynamic settings, potentially skewing SHD interpretations.