🤖 machine learning

Counterfactual Residual Data Augmentation for Regression

This paper proposes Counterfactual Residual Data Augmentation (CRDA), a model-agnostic technique for tabular regression that generates realistic training samples by exploiting the invariance of residuals under small feature perturbations, thereby significantly reducing mean squared error in data-scarce and noisy settings.

Original authors: Hossein Mohebbi, Oliver Schulte, Ke Li, Pascal Poupart

Published 2026-06-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hossein Mohebbi, Oliver Schulte, Ke Li, Pascal Poupart

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to predict the price of a house. You only have a small pile of photos and price tags (data) to show it. The robot looks at the photos, learns that big houses in good neighborhoods cost more, and starts making guesses. But sometimes, it gets it wrong. Maybe it didn't know that a specific buyer was in a huge rush to buy, or that the seller was desperate. These "mistakes" or "surprises" are called residuals.

Usually, when data is scarce, the robot just memorizes the few examples it has, which is bad. Traditional ways to fix this (like in image editing) involve flipping pictures or changing colors, but that doesn't work well for numbers like house prices or medical stats.

This paper introduces a new trick called CRDA (Counterfactual Residual Data Augmentation). Here is how it works, using simple analogies:

1. The "Systematic" vs. The "Noise"

Think of the robot's prediction as a two-part recipe:

The Systematic Part: The predictable stuff. "If the house is 2,000 sq ft, the price is $X." The robot learns this well.
The Residual (Noise): The unpredictable stuff. "But wait, this specific house sold for $X +$ 5,000 because the buyer was emotional."

The paper's big idea is: The "emotional buyer" part (the noise) stays the same even if we change some other details.

2. The "What-If" Game (Counterfactuals)

Imagine you have a house with a garage and a garden.

The robot knows that a bigger garage usually means a higher price (Systematic).
The robot also knows that this specific house had a weird price jump because of a "bidding war" (Residual).

CRDA asks: "What if this house had a different garage finish, but the same bidding war?"

The paper argues that changing the garage finish changes the base price, but it doesn't change the fact that a bidding war happened. The "bidding war" (the residual) is independent of the garage.

3. How CRDA Creates New Data

Instead of just guessing new numbers, CRDA does this:

Train a base robot on the original data.
Find the "safe" features: It uses a detective (an algorithm) to figure out which features (like garage finish) can be changed without messing up the "bidding war" (residual). It avoids changing features that are the bidding war (like the buyer's urgency).
Create a "What-If" scenario: It takes a real house, changes the "safe" feature (e.g., makes the garage finish different), calculates the new base price, and adds back the exact same "bidding war" noise from the original house.
Result: You now have a brand new, realistic-looking house example that the robot has never seen before, but it follows the same rules of reality.

4. Why This is Better Than Other Methods

Old ways (like mixing data): Imagine taking a house from New York and a house from Texas and blending them together to make a "New-Tex" house. That house doesn't exist and confuses the robot.
Generative AI (Deep Learning): Imagine a robot that tries to invent a house from scratch. It might make a house that looks real but has a price that makes no sense because it forgot the specific "noise" of the original data.
CRDA: It's like taking a real house, swapping out the paint color, and keeping the exact same story about why it sold for that price. It creates data that is faithful to the original patterns.

5. The Safety Net

The paper admits this trick only works if the robot is smart enough to understand the "systematic" part first. If the robot is too dumb, the "noise" it calculates is just garbage, and changing features won't help.

So, CRDA includes a safety check:

It tries the new data on a test.
If the new data makes the robot smarter, it keeps it.
If the new data confuses the robot, it throws it away and sticks to the original data.

The Results

The authors tested this on 9 different real-world datasets (like predicting house prices, wine quality, and energy efficiency).

For Neural Networks (MLP): It reduced errors by about 23% on average.
For Tree Models (XGBoost): It reduced errors by about 6% on average.
It consistently beat other fancy data-creation methods, which often made things worse.

In short: CRDA is a way to stretch a small dataset by asking "What if?" about specific details, while carefully keeping the "unpredictable" parts of the story exactly the same. It's a simple, safe way to give a robot more practice without lying to it.

Technical Summary: Counterfactual Residual Data Augmentation for Regression

Problem Statement

Real-world regression tasks in domains such as medicine, finance, and manufacturing frequently suffer from data scarcity, high data collection costs, and noisy observations. While data augmentation is a cornerstone of success in computer vision and natural language processing, existing techniques (e.g., geometric transformations, Mixup) often rely on label-preserving symmetries or domain-specific invariances that do not translate naturally to tabular regression. Consequently, supervised learning models often fail to capture underlying processes when training examples are limited, leading to overfitting and poor generalization.

Methodology: Counterfactual Residual Data Augmentation (CRDA)

The authors propose Counterfactual Residual Data Augmentation (CRDA), a model-agnostic framework designed to synthesize new, realistic training samples for tabular regression by exploiting the invariance of residual noise.

Core Insight

The method is grounded in the observation that once a regressor models the systematic component of the data, the remaining noise (residual) can be viewed as an invariant component under small perturbations of carefully selected features. By preserving this residual while modifying specific features, CRDA generates counterfactual samples that adhere to the underlying data distribution without requiring additional real-world data.

Theoretical Foundation

CRDA operates under the assumption of an Additive-Noise Structural Causal Model (SCM):
$Y = g(X_P, X_R) + Z$
Where:

$Y$ is the target variable.
$g(X_P, X_R)$ is the systematic component (conditional expectation).
$Z$ is the structural noise (residual).
$X_P$ represents perturbable features and $X_R$ represents fixed features.

The method relies on Assumption 3.1 (Residual Invariance): The noise term $Z$ is conditionally independent of the perturbable features $X_P$ given the fixed features $X_R$ ( $P(Z | X_P, X_R) = P(Z | X_R)$ ). This implies that perturbing $X_P$ changes the systematic prediction but does not alter the instance-specific noise.

Algorithmic Workflow

The CRDA pipeline (Algorithm 1) proceeds as follows:

Baseline Training: A base regressor $\hat{g}$ (e.g., MLP or XGBoost) is trained on the original dataset $D_{train}$ .
Residual Computation: Empirical residuals are calculated: $\hat{z}_i = y_i - \hat{g}(x_i)$ .
Feature Selection (Independence Filtering): To satisfy Assumption 3.1, the algorithm identifies a subset of features $X_P$ $X_{P}$ that are likely independent of the residuals. This is achieved via:
- PC Algorithm: A causal graph check to remove features with direct edges to the residual $\hat{Z}$ .
- Correlation Check: Removing features strongly correlated with $\hat{Z}$ .
- The remaining features form $X_P$ ; the rest form $X_R$ .
Counterfactual Generation: For each training sample, $M$ $M$ synthetic samples are generated:
- Perturb $X_P$ using a uniform scaling factor: $x'_{i,P} = x_{i,P} \cdot (1 + \delta)$ , where $\delta \sim \text{Unif}[-p, p]$ .
- Keep $X_R$ fixed.
- Compute the counterfactual label by preserving the original residual: $y'_i = \hat{g}(x'_i) + \hat{z}_i$ .
Safety Validation: Before final retraining, a Wilcoxon signed-rank test compares the performance of the baseline model against a model trained on the augmented data via K-fold cross-validation. If the p-value indicates no statistically significant improvement, the augmentation is rejected, and the baseline is retained.
Final Retraining: If validation passes, a final model is trained on the union of original and augmented data.

Key Contributions

New Data Augmentation Framework: A model-agnostic strategy for tabular regression centered on counterfactual reasoning and residual invariance, avoiding the need for domain-specific heuristics.
Residual Invariance Principle: A formalization of how specific features can be perturbed without corrupting the noise structure, providing a theoretical basis for feature selection in augmentation.
Empirical Validation: Comprehensive evaluation across synthetic benchmarks and real-world datasets (UCI, PMLB, Kaggle), demonstrating consistent improvements across neural (MLP) and ensemble (XGBoost) models.

Experimental Results

The authors evaluated CRDA on nine benchmark datasets with varying sample sizes, comparing it against:

Baselines: No augmentation.
Competitors: State-of-the-art regression augmentations (C-Mixup, ADA) and deep generative models (CTGAN, TVAE, TabDDPM).

Key Findings:

Performance Gains: On average, CRDA reduced the Mean Squared Error (MSE) by 22.9% for MLP regressors and 6.4% for XGBoost regressors across all datasets.
Robustness: Unlike geometric methods (e.g., ADA, C-Mixup) which showed catastrophic failure modes (increasing MSE by >100% in some cases) and deep generative models which often degraded performance, CRDA consistently improved or maintained performance without significant degradation.
Sample Size Dependency: The method is most effective in a "sweet spot" of sample sizes (e.g., 2.5k–20k for MLPs), where the base model has learned the main signal but still benefits from localized feature space exploration. For very small datasets, the base model is too inaccurate to produce meaningful residuals; for very large datasets, the baseline is already optimal.
Safety Mechanism: The Wilcoxon safety gate successfully prevented augmentation in cases where it would not help (e.g., when using a weak linear regression baseline), ensuring the method does not degrade performance.

Significance and Claims

The paper positions CRDA as a simple and efficient remedy for noise-prone, small-sample regression settings. Its significance lies in:

Practicality: It requires no domain-specific knowledge or complex generative modeling, making it readily applicable to various regressors.
Theoretical Grounding: By framing augmentation through the lens of causal counterfactuals and residual invariance, it offers a principled alternative to heuristic interpolation methods.
Reliability: The built-in safety validation ensures that the method acts as a "guardrail," preventing performance degradation in scenarios where augmentation assumptions (residual independence) are violated or the base model is insufficient.

The authors conclude that CRDA offers a robust approach to expanding datasets without requiring additional real data, effectively reducing variance and improving model fit in data-scarce regimes. Future work is suggested to extend these principles to classification tasks and to explore more robust causal discovery algorithms for feature partitioning.