The Honest Truth About Causal Trees: Accuracy Limits for Heterogeneous Treatment Effect Estimation

1. Problem Statement

The paper addresses the fundamental statistical limitations of recursive decision trees (specifically CART-type algorithms) when used to estimate Heterogeneous Treatment Effects (HTE) in both experimental and observational studies.

While causal trees (e.g., the framework by Athey and Imbens, 2016) are widely adopted for their interpretability and ability to discover subgroups with different treatment effects, their theoretical properties regarding uniform convergence rates remain poorly understood. The authors investigate whether standard greedy recursive partitioning can achieve polynomial convergence rates (i.e., $O(n^{-\gamma})$ for some $\gamma > 0$) in the uniform norm (supremum norm) over the covariate space, even under ideal conditions where the true treatment effect is constant.

2. Methodology and Setup

Data Generating Process (DGP)

The authors analyze a simplified but canonical setting (Assumption 1):

• Randomized Experiment: Treatment assignment $D_i$ is independent of potential outcomes and covariates.
• Constant Treatment Effect: The true Conditional Average Treatment Effect (CATE), $\tau(x) = E[Y_i(1) - Y_i(0) | X_i = x]$, is constant ($\tau$) across the entire covariate space $X$.
• Covariates: $X_i$ are independent and continuously distributed (e.g., Uniform on $[0,1]^p$).
• Errors: Potential outcome errors are sub-exponential.

This setup is chosen because if a method fails to converge uniformly in the simplest case (constant effect), it cannot be guaranteed to work for more complex, heterogeneous functions.

Estimators Studied

The paper evaluates three main classes of causal tree estimators based on different splitting criteria and data usage schemes:

1. Splitting Criteria:
   • Variance Maximization (DIM/IPW): Splits that maximize the squared difference in estimated treatment effects between child nodes.
   • Sum of Squared Errors (SSE): Minimizes the residual sum of squares for a linear model within nodes.
2. Data Usage Schemes:
   • No Sample Splitting (NSS): The same data is used for tree construction and effect estimation.
   • Honesty (HON): Sample splitting is used; one subset builds the tree, and an independent subset estimates effects.
   • X-Adaptivity: A theoretical intermediate where covariates are shared, but outcomes are refreshed at each split depth.

3. Key Contributions

The paper makes three primary theoretical contributions:

1. Lower Bounds on Uniform Convergence: The authors establish that causal tree estimators constructed via standard greedy splitting cannot achieve polynomial-in-$n$ convergence rates in the uniform norm. The estimators converge slower than any $n^{-b}$ rate for $b \in (0,1)$.
2. Ineffectiveness of "Honesty": While sample splitting (honesty) is widely believed to fix overfitting and improve inference validity, the authors prove it yields at best negligible logarithmic improvements in convergence rates. It does not resolve the fundamental inconsistency caused by the tree construction mechanism.
3. Clarification of $\alpha$-Regularity: The paper demonstrates that existing theoretical guarantees for Causal Forests (which rely on $\alpha$-regularity, i.e., balanced splits) are incompatible with standard CART-type greedy splitting. Under basic conditions, greedy splitting selects highly imbalanced splits with non-vanishing probability, violating the assumptions required for existing forest consistency proofs.

4. Key Results and Mechanisms

The Mechanism: "End-Cut" Preference and Small Cells

The core mechanism driving the failure is the greedy nature of recursive partitioning:

• Stochastic Fluctuations: When the true signal is flat (constant treatment effect), the splitting criterion is driven entirely by noise (stochastic fluctuations in the data).
• Imbalanced Splits: With non-vanishing probability, the "optimal" split (maximizing the criterion due to noise) occurs near the boundary of the parent node.
• Small Terminal Nodes: These boundary splits create child nodes with very few observations (e.g., $O(n^b)$ for small $b$).
• Variance Explosion: Treatment effects in these small nodes are estimated by local averaging. With very few samples, the variance of the estimator becomes arbitrarily large.
• Recursive Propagation: As the tree deepens, these small, high-variance regions propagate, creating a "fractal" of small cells scattered throughout the covariate space.

Formal Theorems

• Theorem 1 (Uniform Accuracy): For any $b \in (0,1)$, the probability that the supremum error exceeds $C n^{-b/2} \sqrt{\log \log n}$ (for NSS) or $C n^{-b/2}$ (for HON) is bounded away from zero. This implies the estimator is inconsistent in the uniform norm.
• Theorem 2 (Mean Square Accuracy): In contrast to uniform convergence, the Integrated Mean Squared Error (IMSE) can still achieve favorable rates (near $O(n^{-1})$). This highlights a tension: trees may perform well on average (prediction) but fail catastrophically at specific points (estimation/inference).
• Theorem 3 (X-Adaptivity & Inconsistency): Even under X-adaptive designs (which decouple outcomes at each split), if the tree depth $K_n$ grows faster than $\log \log n$, the estimator remains uniformly inconsistent.

Correction of Eicker [1979]

As a technical by-product, the authors identify and correct an error in Eicker (1979) regarding the limiting distribution of the supremum of an Ornstein-Uhlenbeck process, which is crucial for their proof of the lower bounds.

5. Significance and Implications

• Theoretical Limits of CART: The paper provides a rigorous "no-go" theorem for the uniform estimation of heterogeneous effects using standard greedy trees. It suggests that the adaptivity of CART, which makes it attractive for prediction, is its Achilles' heel for uniform causal inference.
• Inference Validity: Standard confidence intervals based on asymptotic normality (e.g., $\hat{\tau} \pm z_{\alpha} \cdot SE$) are likely invalid in regions where terminal nodes are small. Since small nodes occur with non-vanishing probability, the global validity of inference is compromised.
• Role of Regularization: The paper discusses that while regularization (e.g., minimum node size) can prevent small cells, it introduces approximation bias if the true effect is not locally constant. Furthermore, aggressive regularization destroys the adaptivity that trees are designed to exploit.
• Causal Forests: The results imply that theoretical guarantees for Causal Forests relying on $\alpha$-regularity (balanced splits) do not automatically extend to implementations using standard CART splitting rules. To achieve consistency, forests may require explicit constraints that force balanced splits, potentially sacrificing the ability to detect small, meaningful subgroups.
• Prediction vs. Estimation: The contrast between Theorem 1 (uniform failure) and Theorem 2 (average success) underscores a fundamental distinction in machine learning: a model can be an excellent predictor (low average error) while being a terrible estimator for specific subgroups (high worst-case error).

Conclusion

The paper delivers a "honest truth": Standard causal trees are fundamentally flawed for uniform estimation of heterogeneous treatment effects. They suffer from slow convergence rates and potential inconsistency due to the generation of highly imbalanced partitions driven by noise. While they may perform well on average, they cannot provide reliable, uniform guarantees for subgroup analysis without significant algorithmic modifications that may compromise their adaptivity.