📊 statistics

Robust Joint Modeling for Data with Continuous and Binary Responses

This paper proposes a robust, sparse joint modeling framework for mixed continuous and binary responses in high-dimensional settings that utilizes density power divergence and $\ell_1$ regularization to effectively mitigate the impact of outliers and mislabeled data while outperforming existing methods in both simulations and a real-world semiconductor manufacturing application.

Original authors: Yu Wang, Ran Jin, Lulu Kang

Published 2026-03-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yu Wang, Ran Jin, Lulu Kang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a quality control manager at a semiconductor factory. Your job is to make sure computer chips are perfect. To do this, you run a "lapping" process (like sanding a surface) and you need to check two things simultaneously:

The Thickness: A continuous number (e.g., "The wafer is 0.5mm thick").
The Defect: A simple Yes/No (e.g., "Is this spot scratched? Yes or No").

In the past, statisticians would try to predict these two things separately. But the authors of this paper realized that these two outcomes are actually best friends—they influence each other. So, they built a joint model to predict both at once.

However, there's a big problem: Real-world data is messy.

Sometimes a sensor glitches, a machine jams, or a human labels a "good" chip as "bad" by mistake. In statistics, these are called outliers. If you use a standard math model (like a straight line drawn through a scatter plot), one single crazy data point can drag the whole line off course, ruining your predictions.

The Solution: A "Smart Filter" (DPD)

The authors propose a new method called Robust Joint Modeling. Here is how it works, using simple analogies:

1. The "Density Power Divergence" (DPD) – The Smart Filter

Imagine you are trying to guess the average height of people in a room.

Old Method (Likelihood): You ask everyone, write down their height, and take the average. If one person is a 7-foot-tall basketball player (an outlier), your average skyrockets and becomes useless for the rest of the group.
New Method (DPD): This is like having a smart filter. When the basketball player walks in, the filter says, "Okay, you're very tall, but you look like an anomaly. I'm going to listen to you, but I'm going to turn your volume down." It doesn't ignore the data completely, but it stops that one weird point from screaming over the rest of the crowd.

This "volume control" is based on a mathematical concept called Density Power Divergence. It allows the model to learn the true pattern of the data without getting confused by the noise.

2. The "Lasso" – The Picky Editor

Now, imagine you have 100 different sensors (predictors) telling you about the chip. Maybe 90 of them are just noise, and only 10 actually matter.

The authors added a feature called $\ell_1$ regularization (often called Lasso).
Think of this as a picky editor who reads your report and says, "This sentence is useless, delete it. This paragraph is fluff, cut it."
The result is a sparse model: a simple, clean model that only uses the important sensors. This makes the model easier to understand and less likely to get confused by random noise.

3. The "Proximal Gradient" – The Efficient Climber

Solving this math problem is like trying to find the lowest point in a foggy mountain valley.

Standard methods might get stuck in a small dip (a local minimum) and think they found the bottom.
The authors use a Proximal Gradient Algorithm with a special step size (Barzilai-Borwein).
Imagine a hiker who doesn't just take small, cautious steps. Instead, this hiker looks at the slope, takes a giant leap, checks if they are going down, and if not, adjusts their stride instantly. This allows the computer to find the best solution very quickly, even with thousands of variables.

Why Does This Matter? (The Results)

The authors tested their method in two ways:

Simulated Chaos: They created fake data and then deliberately "poisoned" it with errors (like flipping Yes/No answers or adding huge numbers to thickness measurements).
- Result: The old methods (like Lasso or Bayesian models) crashed or gave terrible predictions. The new DPD method stayed calm, ignored the poison, and kept predicting accurately.
Real Factory Data: They applied it to the actual wafer lapping data.
- Result: The new method predicted the chip thickness much better than the competition. For the "Yes/No" defect check, it was just as good as the best existing methods but with a better balance of errors (fewer false alarms).

The Takeaway

This paper is about building a super-reliable prediction machine for mixed data (numbers and Yes/No answers) that refuses to be fooled by bad data.

Old way: "If the data is weird, the whole model breaks."
New way: "If the data is weird, we turn down the volume on that specific point, ignore the noise, and focus on the real signal."

This is a huge win for industries like semiconductor manufacturing, where a single bad prediction can cost millions of dollars, and where data is often messy and full of glitches.

1. Problem Statement

In many supervised learning applications, particularly in manufacturing (e.g., semiconductor wafer lapping), the response variable consists of mixed types: a continuous outcome (e.g., Total Thickness Variation) and a binary outcome (e.g., pass/fail indicator).

The Challenge: Existing joint modeling approaches (e.g., Bayesian Hierarchical Quantitative and Qualitative models) often rely on likelihood-based estimation, making them highly sensitive to outliers, measurement errors, and mislabeled samples.
The Gap: While robust regression methods exist for single-response types, and joint modeling exists for clean data, there is a lack of a unified framework that simultaneously handles mixed-type responses (continuous + binary) while maintaining robustness against data contamination and sparsity in high-dimensional settings.

2. Methodology

The authors propose a new Robust Joint Modeling Framework based on the Density Power Divergence (DPD) criterion combined with $\ell_1$ regularization.

A. Statistical Model

The joint distribution of the continuous response $y$ and binary response $z$ given predictors $x$ is modeled as:

Binary Component ( $z$ ): Modeled via logistic regression with probability $p(x) = \text{logit}^{-1}(x^\top \eta)$ .
Continuous Component ( $y$ ): Modeled conditionally on $z$ and $x$ using a linear regression with different coefficients for $z=1$ and $z=0$ :
$y | z, x \sim N(zx^\top \beta + (1-z)x^\top \omega, \sigma^2)$
where $\beta, \omega, \eta$ are the regression coefficient vectors.

B. The DPD Loss Function

Instead of the traditional Maximum Likelihood Estimation (MLE), the authors minimize the Density Power Divergence (DPD) between the empirical data distribution $g$ and the model distribution $f$ .

The DPD loss function $Q_\alpha(\theta, \sigma^2)$ $Q_{α} (θ, σ^{2})$ includes a tuning parameter $\alpha > 0$ $α > 0$ .
- As $\alpha \to 0$ , DPD approaches the Kullback-Leibler (KL) divergence (standard MLE).
- As $\alpha$ increases, the method becomes more robust to outliers by down-weighting the influence of data points with low probability density under the model.
The loss function is derived explicitly for the joint normal-Bernoulli distribution, separating terms for the continuous and binary components.

C. Regularization and Optimization

Sparsity: An $\ell_1$ penalty (Lasso) is added to the loss function to handle high-dimensional inputs ( $p \gg n$ ) and perform variable selection:
$h(\theta) = Q_\alpha(\theta) + \lambda_1\|\beta\|_1 + \lambda_2\|\omega\|_1 + \lambda_3\|\eta\|_1$
Algorithm: A Proximal Gradient Algorithm is developed to solve the non-convex optimization problem.
- It utilizes Block Coordinate Descent to update $\beta, \omega, \eta$ iteratively.
- Barzilai-Borwein (BB) spectral step sizes are used to accelerate convergence.
- Soft-thresholding operators are applied to enforce sparsity.
Variance Estimation: A robust "plug-in" strategy is used for $\sigma^2$ (based on Pseudo Standard Error) to prevent instability during the initial optimization phase.

D. Model Selection

A Robust Information Criterion (RIC) is proposed for selecting the tuning parameters ( $\alpha$ and $\lambda$ ). Unlike AIC/BIC, RIC is derived from the DPD framework and is less sensitive to outliers, balancing model fit and complexity.

3. Key Contributions

Unified Framework: First method to jointly model continuous and binary responses using a robust divergence-based approach (DPD) rather than likelihood.
Robustness: The framework naturally down-weights influential outliers in predictors ( $X$ ) and responses ( $Y, Z$ ), as well as mislabeled samples.
Theoretical Guarantees: The authors prove that the proposed DPD estimator is consistent and asymptotically multivariate normal under mild regularity conditions, enabling valid statistical inference.
Computational Efficiency: Development of an efficient proximal gradient algorithm with BB step sizes and a grid-search strategy based on RIC for parameter tuning.

4. Experimental Results

The method was evaluated through extensive simulations and a real-world case study.

A. Simulation Studies

Settings: Tested in both low-dimensional ( $p=8$ ) and high-dimensional ( $p=50$ ) settings with varying sparsity levels.
Contamination: Data was contaminated with outliers in predictors, continuous responses, binary responses, and joint contamination (up to 20% contamination).
Comparisons: Outperformed Lasso, SparseLTS, Lasso-QR, Ada-LAD-Lasso, and the Bayesian Hierarchical Quantitative and Qualitative (BHQQ) model.
Findings:
- Parameter Estimation: The DPD method achieved the lowest $\ell_2$ -norm errors for coefficients ( $\beta, \omega, \eta$ ) across almost all contamination scenarios.
- Prediction: It yielded the lowest Root Mean Squared Prediction Error (RMSPE) for continuous outcomes and the lowest Misclassification Error (ME) for binary outcomes, maintaining stability even as contamination levels increased.

B. Case Study: Wafer Lapping

Data: 450 semiconductor wafer samples with 10 predictors, predicting TTV (continuous) and STIR (binary).
Results:
- Continuous Prediction: DPD achieved the lowest median RMSPE and highest stability compared to all competitors.
- Binary Classification: DPD performed competitively with BHQQ (which had the lowest error) but offered a better trade-off between False Positives (FP) and False Negatives (FN).
- Interpretability: The $\ell_1$ penalty provided sparse, interpretable models, unlike the non-sparse BHQQ.

5. Significance

Industrial Impact: The framework addresses a critical need in manufacturing and quality control where sensor data is often noisy or corrupted. It allows for simultaneous optimization of continuous quality metrics and binary pass/fail decisions without being derailed by outliers.
Statistical Advancement: It bridges the gap between robust statistics (DPD) and high-dimensional sparse modeling, providing a theoretically sound alternative to likelihood-based joint modeling.
Practical Utility: The proposed method offers a "best of both worlds" solution: the robustness of divergence-based estimation and the interpretability of sparse regression, making it highly suitable for real-world, high-stakes applications.