A Generative Approach to Joint Modeling of Quantitative and Qualitative Responses

Technical Summary: A Generative Approach to Joint Modeling of Quantitative and Qualitative Responses (GAQQ)

Problem Statement
In many scientific domains, such as material science and genetics, researchers encounter datasets containing both quantitative (continuous) and qualitative (categorical) responses, often accompanied by a large number of predictor variables (high-dimensional data). Existing literature on mixed outcomes typically employs conditional regression models, where one response type is treated as the outcome and the other as a predictor, or utilizes latent variable approaches. However, these methods often overlook the dependency among the predictor variables themselves, which can provide crucial information for modeling the joint behavior of the responses. Furthermore, handling high-dimensional inputs ($p \ge n$) in these conditional frameworks often leads to computational complexity and difficulties in establishing asymptotic properties.

Methodology: The GAQQ Framework
The authors propose a novel generative approach, termed GAQQ, which models the joint distribution of the predictor variables ($X$), the quantitative response ($y$), and the qualitative response ($Z$).

• Model Assumptions: The method assumes that the combined vector $W = (X', y)'$ follows a multivariate normal distribution conditional on the class label $Z$. Specifically, for $K$ classes, $W | Z=k \sim N(\mu_k, \Sigma)$. The model assumes a common covariance matrix $\Sigma$ across classes but allows for class-specific mean vectors $\mu_k$.
• Regularized Estimation: To address high-dimensional settings where $p \ge n$, the authors formulate a penalized log-likelihood optimization problem. This involves imposing $L_1$ regularization (Lasso) on the mean differences ($\delta_k = \mu_k - \mu_1$) and the inverse covariance matrix (precision matrix) $C = \Sigma^{-1}$. The objective function minimizes:
  $$-n \ln |C| + \sum_{k=1}^K \sum_{i \in G_k} (w_i - \mu_k)'C(w_i - \mu_k) + \lambda_1 \|C\|_1 + \lambda_2 \sum_{k=2}^K |\mu_k - \mu_1|_1$$
  where $\|C\|_1$ is the sum of absolute off-diagonal elements of $C$, and $\lambda_1, \lambda_2$ are tuning parameters.
• Algorithmic Solution: The optimization is solved via an iterative procedure that decomposes the problem into two sub-problems:
  1. Estimating $C$ using the Graphical Lasso (Glasso) technique, treating the mean differences as fixed.
  2. Estimating the mean differences $\delta_k$ using the Lasso technique, treating $C$ as fixed.
     This alternating minimization continues until convergence. Tuning parameters are selected using a BIC-type criterion.
• Prediction Strategy:
  • Quantitative Response ($y$): Predicted using the conditional expectation of a multivariate normal distribution, conditioned on the predicted class label.
  • Qualitative Response ($Z$): Classified using a Linear Discriminant Analysis (LDA) rule derived from the estimated joint distribution.
  • Joint Prediction: The authors demonstrate that the order of prediction (predicting $y$ then $Z$, or vice versa) does not affect the final outcome. The procedure calculates candidate values for $y$ under each class hypothesis, evaluates the joint density, and selects the class and corresponding $y$ value that maximize the posterior probability.
• Extension: The framework is naturally extended to multi-class qualitative responses ($Z \in \{1, \dots, K\}$) by regularizing the differences between each class mean and a baseline class mean.

Key Contributions

1. Generative Perspective: Unlike existing conditional models, GAQQ models the joint distribution of predictors and responses, leveraging the dependency structure among predictors to improve estimation.
2. Theoretical Guarantees: Under regularity conditions (including restricted eigenvalue and irrepresentable conditions), the authors establish the asymptotic optimality of the classification rule. Specifically, the misclassification rate converges to the Bayes risk. Additionally, the mean squared error (MSE) of the quantitative prediction converges to the MSE of the optimal Bayes predictor.
3. Computational Efficiency: By decomposing the complex joint optimization into iterative Glasso and Lasso steps, the method provides an efficient procedure for parameter estimation in high-dimensional settings.
4. Handling Mixed Outcomes: The method simultaneously handles multi-class qualitative responses and multivariate quantitative responses, a capability not easily achieved by latent variable or standard conditional regression approaches.

Results

• Simulations: Extensive simulations were conducted with varying covariance structures (sparse, dense, compound symmetry) and sparsity levels of mean differences.
  • Classification: GAQQ consistently outperformed benchmark methods (GLDA, CL, ENET, WT, CHWE) in terms of misclassification error (ME), particularly as the dimensionality increased and the underlying models became sparser.
  • Prediction: GAQQ demonstrated superior performance in predicting the quantitative response (measured by Root Mean Squared Prediction Error, RMSPE) compared to competitors. The authors attribute this to the accurate classification of $Z$ and the regularized estimation of the covariance matrix.
• Case Studies:
  • Material Science (Heusler Compounds): Applied to predict thermodynamic stability (binary qualitative) and mixing enthalpy (quantitative) from elemental features. GAQQ achieved the lowest ME (10.49%) and RMSPE (0.142) compared to GLDA, ENET, and CL.
  • Genetics (IBD): Applied to gene expression data for Ulcerative Colitis and Crohn's disease (multi-class qualitative) with a randomly selected gene as a quantitative response. GAQQ yielded the lowest ME (15.77%) and RMSPE (0.661) among GLDA, WT, and CHWE.

Significance and Claims
The paper claims that the GAQQ method offers a unique and advantageous perspective by treating the problem as a generative modeling task rather than a conditional regression task. The authors assert that this approach:

• Enables efficient parameter estimation and accurate prediction for both response types in high-dimensional settings.
• Provides a solid theoretical foundation, establishing asymptotic optimality for classification and prediction under standard regularity conditions.
• Allows parameters related to different response types to be "mutually learned," improving performance over methods that model one response type while only indirectly benefiting from the other.

The authors conclude that while the current framework assumes a common covariance matrix (LDA), future work could explore Quadratic Discriminant Analysis (QDA) for more flexible covariance structures or extend the approach to semi-continuous and ordinal responses. However, they note that such extensions would introduce significant technical and computational challenges.