📊 statistics

Stop Suppressing the Tail: Causal Inference for Extreme Events

This paper introduces a novel causal inference estimator that simultaneously recovers the Average Dose-Response Function and provides robust, method-invariant diagnostics for extreme tail events, effectively overcoming the limitations of standard double machine learning in suppressing heavy-tailed outcomes and outperforming existing methods in accuracy and reliability for high-stakes, data-scarce scenarios.

Original authors: Eichi Uehara

Published 2026-05-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Eichi Uehara

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict the weather for a specific town. Usually, meteorologists care about the "average" temperature. If you want to know the average, you can ignore the occasional freak storm or heatwave because they are rare and don't change the daily average much.

But what if you are an insurance company or a disaster planner? In those cases, the "average" doesn't matter at all. You care about the 1-in-1000-year storm. You need to know exactly how bad the worst-case scenario is.

This paper is about a new way to do "weather forecasting" for data, specifically when the data has these rare, extreme "storms" (called heavy tails).

The Problem: The "Noise-Canceling" Headphones

Standard statistical tools (called Double Machine Learning) are like high-end noise-canceling headphones. They are great at blocking out the loud, crazy outliers (the extreme storms) so they can give you a very clear, stable picture of the average weather.

However, the authors point out a fatal flaw: If you are trying to predict the storm, you can't just cancel the noise.

The Flaw: When these tools block out the extreme data to get a clean average, they throw away the exact information you need.
The Circular Trap: Some people try to fix this by looking at the "leftover" data (the noise the headphones blocked) to guess how bad the storm might be. But the paper shows this is a trap. The shape of the "storm" you see depends entirely on which headphones you used to block the noise in the first place. It's like trying to guess the size of a fish by looking at the water ripples left behind by a net that changed its mesh size every time you used it. You get a different answer every time, and none of them are reliable.

The Solution: A Two-Part Detective Team

The authors propose a new method that acts like a two-part detective team. They don't just look at the leftovers; they look at the crime scene directly.

1. The "Steady Hand" (The Core Estimator)
First, they use a special tool (called TailWelsch-DML) to get the average. It's designed to be very careful with the extreme data, not by deleting it, but by gently holding it back so it doesn't ruin the average calculation. Think of it as a steady hand that keeps the average stable without throwing away the extreme values.

2. The "Storm Spotter" (The Tail Diagnostic)
This is the big innovation. Instead of looking at the messy leftovers from the first step, this part of the team looks at the raw data directly, but they first remove the "average" using a very simple, neutral method (a median).

Why this matters: Because they look at the raw data directly, their answer about the "storm" doesn't change based on which tool they used for the average. It's invariant.
The "Refusal" Button: This is a crucial feature. If the data is too messy or doesn't actually have a predictable "storm pattern," this tool will refuse to guess. It will say, "I cannot predict this extreme event because the data doesn't support it." Most other tools would just guess anyway and give you a wrong number. This tool prefers to say "I don't know" rather than lie.

What Does This Team Produce?

Instead of just giving you one number (the average), this new method gives you a full "Extreme Weather Report":

The Shape of the Storm: How heavy is the tail? Is it a bounded storm (it has a limit) or an unbounded one (it could get infinitely worse)?
The "Return Level": If you wait 1,000 years, how bad will the storm be?
The "Shortfall": If the storm hits that 1,000-year mark, how much worse than that mark will it actually be? (This is crucial for insurance reserves).
The "Refusal" Flag: A clear warning if the data is too chaotic to make these predictions.

The Results: Better at the Extremes

The authors tested this new team against the old methods (like standard "Quantile Regression") using simulated data and real-world car insurance claims (which are famous for having huge, rare payouts).

Accuracy: When predicting the deepest, darkest parts of the tail (the 1-in-1000 events), their method was 11% more accurate than the standard tools.
Money Saved: When calculating the "conditional shortfall" (how much extra money you need to prepare for a disaster), their method was 25.5% more accurate.
Small Data: When there isn't much data (fewer than 2,000 samples), their method was 20–29% better.
Real World Test: On a real dataset of French car insurance claims, the old tools tried to give a prediction. The new tool looked at the data, realized the pattern wasn't there, and refused to guess. The authors argue this "refusal" is actually the most valuable output, because it prevents people from making dangerous decisions based on false confidence.

In Summary

This paper says: "Stop trying to hide the extreme events to get a clean average. If you need to know about the extremes, you need a tool that looks at them directly, doesn't get confused by the tools it uses for the average, and has the honesty to say 'I can't predict this' when the data is too messy."

It's a shift from "ignoring the storm to get a calm average" to "building a better storm detector that knows when to stop guessing."

Technical Summary: Stop Suppressing the Tail: Causal Inference for Extreme Events

1. Problem Statement

The estimation of the Average Dose-Response Function (ADRF), $\theta(t) = E[Y(t)]$ , under continuous treatment is a fundamental task in causal inference, typically addressed via Double Machine Learning (DML). However, when outcomes $Y$ exhibit heavy tails (e.g., financial returns, insurance claims, climate losses), standard robust DML approaches face a critical dilemma:

Tail Suppression: To stabilize the bulk average, robust DML methods (using Huber, Quantile, or Winsorized losses) deliberately suppress extreme values. While this stabilizes the point estimate, it discards the tail information that is often the primary quantity of interest in high-stakes domains (e.g., the 1-in-1000 loss).
Circular Dependence in Tail Estimation: A naive alternative is to estimate tail characteristics (e.g., Generalized Pareto Distribution shape $\xi$ ) from the residuals of the core DML estimator. This introduces a structural circularity: the inferred tail shape $\hat{\xi}$ becomes a function of the core estimator's choice. For instance, on the same dataset, switching the core loss from Huber to Welsch can shift the inferred tail shape from bounded ( $\hat{\xi} \approx -0.42$ ) to heavy-tailed ( $\hat{\xi} \approx +0.28$ ), rendering the diagnostic unreliable.
Lack of Tail Output: Standard robust DML returns only a bulk-stabilized point estimate, omitting tail-shape estimates, extreme quantiles, or conditional shortfalls.

2. Methodology

The authors propose a two-component estimator that decouples the core ADRF estimation from the tail diagnostic, ensuring the tail output is invariant to the core method's choice.

A. Core Estimator: TailWelsch-DML

The first component is a robust core estimator designed to preserve the standard Neyman-orthogonal DML convergence rate while minimizing tail suppression.

Mechanism: It employs a kernel-weighted local-linear M-estimator using a redescending Welsch weight function: $w(r) = \exp(-\gamma (r/\sigma)^2)$ .
Fixed Clipping: Unlike adaptive approaches, the clipping parameter $\gamma$ is held constant at a mild value ( $\gamma = 0.10$ ). The authors argue that tying $\gamma$ to a data-driven tail estimate does not improve the core ADRF Mean Absolute Error (MAE) and that tail information is better utilized in the post-core output.
Benefit: This design ensures the second-stage loss is free of data-dependent tuning, allowing standard DML rates to apply directly.

B. Tail Diagnostic: PDHTE+JK

The second component estimates the per-treatment tail shape $\hat{\xi}(t)$ and scale $\hat{\sigma}(t)$ without relying on the residuals of the core estimator, thereby breaking the circular dependence.

Pilot Median Centering: Instead of using $Y - \hat{\theta}(T)$ , the method centers the outcome $Y$ using a pilot kernel-weighted median $\tilde{Y}(t)$ . This location estimate is independent of the downstream core method.
Plateau Detection: The method applies a Hill estimator across a range of order-statistic fractions ( $\kappa$ ) to detect a "plateau" (stability in $\hat{\xi}$ ). A coefficient of variation (CV) threshold ( $<0.25$ ) determines if a regular-varying tail exists.
Hybrid Estimation: If a plateau is detected, the final shape estimate uses a Dekkers-Einmahl-de Haan (DEdH) moment estimator (unbiased across Fréchet, Gumbel, and Weibull domains).
Jackknife Correction: A shrinkage-damped jackknife is applied to correct the finite-sample bias of the DEdH estimator.
Refusal Mechanism: If the plateau condition is not met (indicating the data does not follow a regular-varying tail regime), the estimator explicitly refuses to extrapolate, returning an undefined output rather than a spurious tail estimate.

C. Post-Core Causal Functionals

Using the outputs from the core and the diagnostic, the framework constructs five tail-conditional quantities:

Per-Tail Shape $\hat{\xi}(t)$ : The invariant tail shape estimate.
Mean Recovery $\hat{E}[Y(t)]$ : A formula reconstructing the full mean ADRF by adding sign-conditional GPD contributions (for positive and negative exceedances) to the robust core estimate $\hat{\theta}_W(t)$ . This is crucial for asymmetric contamination where the M-functional differs from the mean.
Hybrid Return Levels $\hat{Q}_\alpha(t)$ : Extreme quantiles estimated via a hybrid approach:
- Empirical Mode: Uses kernel-weighted empirical quantiles when effective sample size is sufficient ( $\alpha n_{eff} \ge 1$ ).
- GPD Mode: Uses GPD extrapolation only when data is scarce ( $\alpha n_{eff} < 1$ ).
Conditional Shortfall $\hat{S}_\alpha(t)$ : The expected loss given an exceedance (analogous to CVaR), derived in closed form from the GPD fit.
Causal-Tail Effect: The sensitivity of the return level to treatment, $\partial \hat{Q}_\alpha / \partial t$ .

Deconfounding: For confounded settings ( $T \not\perp X$ ), the method applies Generalized Propensity Score (GPS) stabilization weights to the kernel weights. While this does not achieve $\sqrt{n}$ -consistency, it provides a first-order partial deconfounding, reducing bias in return levels by approximately 63%.

3. Key Contributions

TailWelsch-DML: A redescending Welsch-DML core estimator with constant clipping ( $\gamma=0.10$ ) that improves core ADRF MAE by 11.6% over Standard-DML on verification panels.
PDHTE+JK: A plateau-detection hybrid tail estimator that consumes only pilot-median-centered outcomes. It establishes method-invariance, ensuring tail diagnostics are not confounded by the choice of the core robust loss. It includes an explicit refusal mechanism when EVT assumptions fail.
Tail-Conditional Causal Functionals: A suite of five outputs (shape, mean recovery, return levels, shortfall, tail effect) that provide actionable extreme-event metrics alongside the point estimate.
Empirical Superiority: Demonstrated reductions in deep-tail error compared to standard baselines.

4. Experimental Results

The paper validates the approach across synthetic Data Generating Processes (DGPs) and real-world data (freMTPL2 motor insurance claims).

Core Performance: TailWelsch-DML reduces core-ADRF MAE by 11.6% (90% CI: $[-17.3\%, -5.8\%]$ ) compared to Standard-DML and marginally outperforms Huber-DML.
Method Invariance: On asymmetric DGPs where residual-based estimators (Standard, Huber, Welsch) produced conflicting tail shapes (ranging from $-0.42$ to $+0.28$ ), PDHTE+JK correctly refused to estimate, demonstrating its independence from the core method.
Deep-Tail Accuracy ( $\alpha = 0.001$ ):
- The proposed estimator reduced Return-Level MAE by 11% compared to Kernel-Weighted Quantile Regression (QR).
- It reduced Conditional Shortfall MAE by 25.5% compared to QR.
- In small sample regimes ( $n \le 2000$ ), the MAE reduction ranged from 20% to 29%.
Regime Classification: The method correctly classified heavy-tail regimes (Fréchet) in 6/6 cases, whereas QR-derived proxies succeeded in only 1/6 cases.
Real-World Application (freMTPL2): On motor insurance claims with extreme kurtosis (1667), the proposed estimator explicitly refused to extrapolate on the log-claim scale (where kurtosis was low), a critical safety output that neither QR nor loss-only DML produced.

5. Significance and Claims

The paper argues that the standard practice of suppressing heavy tails to stabilize ADRF estimates discards the very quantities that drive high-stakes decisions (e.g., extreme losses). Furthermore, it highlights that inferring tail properties from core-estimator residuals creates a circular dependency that invalidates the diagnostic.

The proposed estimator addresses these issues by:

Preserving the Tail: It avoids suppressing extremes in the loss function, instead using a constant-clipping robust core.
Ensuring Validity: It provides a tail diagnostic (PDHTE+JK) that is invariant to the core method choice, breaking the circularity.
Providing Actionable Outputs: It delivers structured tail quantities (shape, return levels, shortfalls) and an explicit refusal mechanism when data does not support extreme-value modeling.

The authors modestly note limitations: the method targets univariate residual tails, the GPS deconfounding is not $\sqrt{n}$ -consistent, and the estimator assumes the tail index is causal via max-stability (Proposition 2), while return levels and shortfalls require partial deconfounding. The approach is presented as a practical framework for causal inference in heavy-tailed settings where extreme events are the primary concern.