💻 computer science

Hybrid Imbalanced Regression Through Unified Data-Level and Algorithm-Level Balancing

This paper proposes a unified hybrid framework for imbalanced regression that combines adaptive data-level balancing (via target-conditioned representation learning and feature-space clustering) with a novel algorithm-level Latent-Density Weighted Loss to effectively address the limitations of existing standalone methods and improve predictive performance on rare target values.

Original authors: Shermin Shahbazi, Hossein Mohammadi, Mohsen Afsharchi

Published 2026-06-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shermin Shahbazi, Hossein Mohammadi, Mohsen Afsharchi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to predict the price of houses. In a perfect world, you'd show it 1,000 examples of $100,000 homes, 1,000 examples of $200,000 homes, and 1,000 examples of $300,000 homes. The robot would learn the pattern perfectly.

But in the real world, data is messy. Maybe you have 1,000 examples of $100,000 homes, but only one example of a $10 million mansion. This is the problem of Imbalanced Regression. The robot gets so good at predicting the common $100k homes that it completely ignores the rare $10M ones. When it finally sees a mansion, it guesses "$100k" because that's what it knows best.

This paper proposes a "Hybrid Framework" to fix this. Think of it as a five-step coaching program designed to help the robot pay attention to the rare, expensive houses without losing its mind on the common ones.

Here is how the five steps work, using simple analogies:

Step 0: The "Smart Map" (Adaptive Bin Partitioning)

The Problem: You can't just say "rare houses" because the prices are a smooth line, not separate boxes like "Red House" vs. "Blue House."
The Solution: The team creates a dynamic map. Instead of cutting the price range into equal slices (like a ruler), they look at the data to see where the "clumps" are. If there's a huge gap between $100k and $1M, they draw a line there. If the data is smooth, they don't cut it.
The Analogy: Imagine you are organizing a library. Instead of putting books on shelves by exact page count (which is messy), you look at the stories. You group all the "short stories" together and all the "novels" together based on how the stories actually flow. This helps the robot see the "rare" sections clearly.

Step 1: The "Translator" (Representation Learning)

The Problem: The raw data (square footage, number of rooms) is too noisy and complex for the robot to find the rare patterns.
The Solution: They use a special tool called a CVAE (Conditional Variational Autoencoder). Think of this as a translator that converts the messy house data into a "secret language" (a latent space) where the rare houses look very distinct from the common ones.
The Analogy: Imagine the robot is trying to understand a foreign language. This step translates the data into a language the robot speaks fluently, making the "rare" words stand out clearly against the "common" words.

Step 2: The "Copy-Paste & Polish" (Data-Level Balancing)

The Problem: Even with the secret language, there are still too few examples of the rare houses. The robot needs more practice.
The Solution: They don't just copy-paste the rare houses (which would be cheating and confusing). Instead, they find the "neighborhoods" of rare houses in the secret language and create new, synthetic examples that fit perfectly in that neighborhood.
The Analogy: Imagine you are a chef trying to learn a rare recipe, but you only have one ingredient list. You don't just photocopy the list; you use the list to understand the flavor profile and then create a few new, slightly different versions of the dish that taste exactly right. Now you have enough practice dishes to learn the recipe.

Step 3: The "Strict Coach" (Algorithm-Level Balancing)

The Problem: Even with more practice data, the robot might still ignore the rare examples because it's lazy and wants to minimize its overall mistakes.
The Solution: They change the scoring system (the loss function). If the robot makes a mistake on a common house, it gets a small penalty. If it makes a mistake on a rare house, it gets a massive penalty.
The Analogy: Imagine a video game. Usually, you get 10 points for killing a goblin. But if you kill a rare dragon, you get 1,000 points. The robot realizes, "Hey, I better pay attention to the dragons!" This forces the robot to care about the rare data points.

Step 4: The "Mixer" (Final Fusion)

The Problem: The robot now has two different ways of thinking: one based on the extra practice data (Step 2) and one based on the strict scoring system (Step 3). How do we combine them?
The Solution: They use a Gated Fusion mechanism. This is like a smart manager who looks at every specific house and decides, "For this house, I trust the practice data more," or "For that house, I trust the strict scoring more."
The Analogy: It's like a judge listening to two lawyers. For some cases, the judge listens to Lawyer A; for others, Lawyer B. The judge (the fusion) knows exactly when to listen to which expert to get the best verdict.

What Did They Find?

The authors tested this "coaching program" on 16 different datasets (like predicting house prices, wine quality, and machine torque).

The Result: The hybrid approach (using all 5 steps) was significantly better than using just the "Copy-Paste" method or just the "Strict Coach" method alone. It was also much better than standard robots that didn't get any special coaching.
The Catch: This program works best when you have a lot of data (thousands of examples). If you only have a tiny dataset (like 100 examples), the program can get confused and actually perform worse than a simple robot. It needs enough "students" to teach effectively.

Summary

This paper builds a universal training system for predicting continuous numbers (like prices or temperatures) when the data is skewed. It combines making more data (to fill in the gaps) and changing the rules (to force attention on the gaps) into one powerful pipeline. It's like giving a student both a better textbook and a stricter teacher to ensure they learn the difficult, rare topics just as well as the easy ones.

Technical Summary: Hybrid Imbalanced Regression Through Unified Data-Level and Algorithm-Level Balancing

Problem Statement

Imbalanced learning is a critical challenge in machine learning where underrepresented target values lead to biased models that perform poorly on rare but high-impact phenomena. While extensively studied in classification tasks with skewed class distributions, imbalanced regression—characterized by non-uniform densities in continuous target spaces—remains underexplored.

Existing approaches in imbalanced regression typically fall into two categories, each with distinct limitations:

Data-level methods: These synthesize samples (e.g., via oversampling) to enrich sparse regions. While intuitive and model-agnostic, they risk introducing synthetic noise or overfitting, particularly in complex distributions.
Algorithm-level methods: These adjust loss functions to prioritize rare target ranges without modifying data. While principled, they often struggle with severe imbalances and complex distributions, lacking the complementary benefits of data enrichment.

Current literature lacks a truly hybrid approach that tightly integrates data-level sample synthesis with explicit, sample-wise cost-sensitive or reweighting losses during the training of a single neural regressor. Most existing "hybrid" attempts rely on ensemble strategies or are limited to non-tabular domains, failing to unify these strategies in an end-to-end pipeline for standard regression tasks.

Methodology

The authors propose a five-phase, regressor-agnostic hybrid framework that synergistically integrates data-level and algorithm-level strategies into a unified pipeline. The framework is designed to be applied upstream of any standard regressor.

Phase 0: Adaptive Target-Space Bin Partitioning

To address the lack of discrete class labels in regression, the framework first segments the continuous target space into statistically coherent bins.

Mechanism: It employs adaptive bin partitioning driven by local $R^2$ coherence scores. A simple linear regressor is fitted within candidate intervals; if the $R^2$ score exceeds a threshold ( $\tau$ ) and the bin size meets a data-driven minimum ( $M$ ), the bin is accepted. Otherwise, the interval is recursively split.
Significance: This transforms the continuous problem into a class-like structure, enabling localized awareness of sparsity and facilitating subsequent balancing operations. The minimum bin size $M$ is dynamically calculated using a combination of a proportion of the dataset size and an inverse of the peak Kernel Density Estimation (KDE) density to ensure statistical reliability.

Phase I: Representation Learning

Raw features in sparse regions often fail to capture necessary latent structures.

Mechanism: A Conditional Variational Autoencoder (CVAE) is trained, conditioned on the continuous target variable $y$ . The encoder maps input features $x$ and target $y$ to a latent space $z$ , while the decoder reconstructs $x$ from $z$ and $y$ .
Significance: This produces target-aware latent embeddings that capture predictive patterns aligned with the output distribution, including sparsely populated regions. The probabilistic nature of the CVAE also facilitates uncertainty-aware representations.

Phase II: Data-Level Balancing

This phase addresses sample scarcity in both the target and feature spaces through a multistage strategy.

Step 1 (Clustering): Within each adaptive bin from Phase 0, DBSCAN is applied to the latent representations to identify sparse, minority clusters.
Step 2 (Oversampling): A localized SMOGN (Synthetic Minority Oversampling Technique for Regression with Gaussian Noise) technique is applied to these minority clusters. Unlike global oversampling, this is guided by a cluster-specific relevance function $\phi_C(y)$ to prioritize rare target values within the specific cluster context.
Output: A shallow MLP is trained on the augmented dataset (original + synthetic), and the hidden activations of the original samples ( $h_{data}$ ) are extracted. This ensures the features benefit from the synthetic neighbors' geometry while maintaining exact correspondence with original instances.

Phase III: Algorithm-Level Balancing

This phase adapts the learning objective to emphasize underrepresented regions without further data modification.

Mechanism: A novel Latent-Density Weighted Loss (LDWL) is introduced. The loss function weights samples based on their joint rarity in two spaces:
1. Latent Space ( $Z$ ): Density estimated via KDE over the CVAE latent representations.
2. Target Space ( $Y$ ): Density estimated via KDE within the adaptive bins defined in Phase 0.
Formula: The weight $w_i$ for a sample is inversely proportional to the product of its latent density and its bin-conditioned target density. A regularization term penalizes excessively large weights to prevent overfitting to outliers.
Output: An MLP is trained using LDWL, and the hidden activations of the original samples ( $h_{alg}$ ) are extracted.

Phase IV: Final Fusion

The final phase integrates the complementary strengths of the data-level and algorithm-level representations.

Mechanism: An attention-based gated fusion mechanism concatenates $h_{data}$ and $h_{alg}$ . A gating network dynamically learns instance-specific weights ( $\alpha_i$ ) to blend the two feature streams: $\tilde{x}_i = \alpha_i \odot h_{data} + (1-\alpha_i) \odot h_{alg}$ .
Significance: This allows the model to adaptively prioritize the most informative features for each sample, producing a robust, enriched representation ready for any downstream regressor.

Key Contributions

Unified Hybrid Framework: The first end-to-end pipeline to tightly integrate data-level sample synthesis (via localized SMOGN) with explicit, sample-wise cost-sensitive reweighting (via LDWL) for imbalanced regression.
Adaptive Bin Partitioning: A novel preprocessing step that uses local $R^2$ coherence to dynamically segment continuous target spaces, emulating class-like structures for effective imbalance handling.
Latent-Density Weighted Loss (LDWL): A new loss function that prioritizes rare samples by jointly considering their density in the latent feature space (learned via CVAE) and the target space (within adaptive bins).
Multistage Data Balancing: A strategy combining feature-space clustering (DBSCAN) with localized oversampling to enrich sparse regions while minimizing synthetic bias.
Regressor-Agnostic Design: The framework outputs a fused feature vector that can be seamlessly integrated with any standard regression algorithm (linear, tree-based, or neural).

Experimental Results

The framework was evaluated on 16 benchmark regression datasets (e.g., California Housing, Wine Quality, Abalone) across six different regressors (MLP, XGBoost, Linear Regression, KNN, SVR, Ridge).

Performance: The hybrid pipeline consistently outperformed both standalone data-level and algorithm-level methods, as well as standard baseline regressors without imbalance handling.
Metrics: Improvements were observed across Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), and the Coefficient of Determination ( $R^2$ ). For instance, in the airfoild dataset, the hybrid approach yielded a 568.71% improvement in $R^2$ compared to the standalone regressor.
Skewness Correlation: Analysis revealed a significant positive correlation between the absolute skewness of the target variable and the performance improvement of the hybrid pipeline. The framework is most effective in scenarios with high target skewness.
Limitations: The framework exhibits a dependency on data volume. In datasets with fewer than ~1,000 samples, the pipeline occasionally underperformed standalone regressors due to overfitting in the adaptive binning and CVAE phases, and instability in density estimation.

Significance and Claims

The paper claims that the proposed hybrid framework addresses a critical gap in imbalanced regression literature by moving beyond isolated data or algorithm-level interventions. By unifying these strategies, the framework provides a more comprehensive solution for predicting rare but critical target values.

The authors assert that their approach:

Synergistically combines the diversity benefits of data augmentation with the principled optimization of cost-sensitive learning.
Enhances generalization in sparse regions without the noise risks of naive oversampling or the distributional blindness of standard loss functions.
Scales effectively to large-scale imbalanced regression tasks, demonstrating that the degree of target skewness is a reliable proxy for the framework's effectiveness.

The study concludes that while the framework requires sufficient data volume to stabilize its five-phase architecture, it offers a robust, scalable, and regressor-agnostic enhancement for real-world applications involving extreme weather forecasting, anomaly detection, and medical or financial risk prediction where rare events are of paramount importance.