🤖 machine learning

Improving Performance in Classification Tasks with LCEN and the Weighted Focal Differentiable MCC Loss

This paper introduces a modified LASSO-Clip-EN (LCEN) algorithm for interpretable, sparse feature selection in classification tasks and demonstrates that combining it with a novel weighted focal differentiable MCC (diffMCC) loss function significantly outperforms standard models and loss functions across multiple datasets.

Original authors: Pedro Seber, Richard D. Braatz

Published 2026-04-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Pedro Seber, Richard D. Braatz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a complex mystery, like figuring out why a patient is sick, predicting if a customer will buy a product, or identifying what type of glass you are holding. You have a massive pile of clues (data), but most of them are just noise—red herrings that distract you from the real answer.

This paper introduces two powerful new tools to help solve these mysteries more accurately and clearly: a smarter detective named LCEN and a new way of grading the detective's work called diffMCC.

1. The Smarter Detective: LCEN

Previously, the LCEN algorithm was like a brilliant detective who could only solve "how much" questions (regression), such as "How much rain will fall tomorrow?" But it couldn't handle "which category" questions (classification), like "Will it rain or shine?" or "Is this tumor benign or malignant?"

The authors upgraded LCEN to handle these classification mysteries. Here's what makes it special:

The "Clue Cleaner": Imagine you have 100 clues, but 56 of them are useless (like the color of the suspect's socks). LCEN is incredibly good at ignoring the noise. It acts like a strict editor, cutting out the average of 56% of all the clues it doesn't need. This leaves you with a short, sharp list of the most important facts.
The "Explainable" Detective: Many modern AI models are "black boxes." You give them data, they give an answer, but they can't explain why. LCEN is different. Because it strips away the useless clues, it tells you exactly which few factors mattered. It's like a detective saying, "I know he did it because he was seen near the scene at 8 PM and his shoes match the mud," rather than just saying, "The computer says he did it."
The "Team Booster": The researchers tested LCEN by letting it pick the clues, and then handing those specific clues to other detectives (like Random Forests or Neural Networks). Surprisingly, when these other detectives used only the clues LCEN picked, they got better at solving the case than when they had the whole messy pile of clues. It's like a coach telling a team, "Don't look at the whole field; just focus on these three players," and suddenly the team wins.

2. The Better Grading System: diffMCC

Now, imagine you are training a student (a machine learning model) to take a test. Usually, teachers use a standard grading rubric called Cross-Entropy. It's like a teacher who only cares about the average score across all questions.

The problem is, in real life, some mistakes are worse than others. If a model predicts a patient is healthy when they are actually sick, that's a disaster. But the standard grading system might not penalize that specific mistake enough because the student got the other 99 questions right.

The authors introduced a new grading system called diffMCC (Weighted Focal Differentiable MCC).

The "Fair Judge": Think of diffMCC as a judge who cares deeply about the quality of the predictions, not just the average. It specifically penalizes the model for missing the "hard" or "important" cases.
The Result: When they trained their AI models using this new grading system, the models became significantly better. On average, they scored 4.9% higher on accuracy and 8.5% higher on a metric called "Matthews Correlation Coefficient" (which is like a score that measures how well the model distinguishes between different groups) compared to models trained with the old, standard grading system.

The Big Picture: What Did They Find?

The authors tested these tools on four different real-world mysteries:

Heart Failure: Predicting if a patient will have heart failure.
Bank Marketing: Predicting if a customer will buy a term deposit.
Wine Quality: Rating wine based on chemical properties.
Glass Identification: Identifying the type of glass based on its chemistry.

The Results:

LCEN was a top performer. It often beat 10 other popular AI models. Even when it wasn't the absolute #1, it was usually in the top tier, but with the added superpower of being simple and explainable.
The "Clue Cleaning" worked: Using the clues LCEN picked out made almost every other model perform better.
The "New Grading System" (diffMCC) was a winner: Models trained with this new method were consistently the best performers across all four mysteries, beating the standard methods every time.

Why Should You Care?

In the past, you often had to choose between a model that was accurate but confusing (like a black-box AI) or a model that was simple but less accurate.

This paper shows you don't have to choose.

LCEN gives you a model that is both accurate and easy to understand (interpretable).
diffMCC gives you a way to train models that are incredibly precise, especially when the stakes are high (like in medicine or engineering).

It's like upgrading from a blurry, confusing map to a high-definition GPS that not only tells you the fastest route but also explains why it's the best route, all while making sure you don't miss any turns.

1. Problem Statement

The paper addresses two primary challenges in machine learning classification tasks:

Limitations of Existing Feature Selection: While the LASSO-Clip-EN (LCEN) algorithm was previously successful for regression tasks due to its ability to provide nonlinear, interpretable, and sparse models, it had not been adapted for classification tasks.
Loss Function Discrepancy: Deep learning models for classification are typically trained using Cross-Entropy (CE) loss, which minimizes prediction error probability. However, models are evaluated using metrics like the Macro F1 Score and Matthews Correlation Coefficient (MCC). Since CE and these metrics optimize different objectives, a model with the lowest CE loss is not guaranteed to have the best classification performance. Furthermore, standard classification metrics are often non-differentiable, preventing their direct use as loss functions.

2. Methodology

A. Modified LCEN Algorithm for Classification

The authors adapted the LCEN algorithm (originally for regression) to handle classification tasks while preserving its core properties of sparsity and interpretability. Key modifications include:

LASSO Step: Changed from linear regression with an $L_1$ penalty to logistic regression with an $L_1$ penalty.
Elastic Net (EN) Step: Changed from linear regression with an $L_1$ - $L_2$ penalty to logistic regression with an $L_1$ - $L_2$ penalty.
Clip Step: Remains unchanged for binary classification. For multiclass classification, the algorithm generates a feature set for each class. A new hyperparameter, LCEN_clip_min_classes_selected, was introduced to harmonize these sets. A feature is retained only if it appears in at least $k$ class-specific feature sets (where $k$ is the hyperparameter value).
Complexity: The modified algorithm maintains a computational complexity of $O(NP^2FAC + Np^2FALC)$ , where $N$ is samples, $P$ is features, and $p$ is the reduced feature set size. Theoretical analysis shows it remains faster than the Elastic Net-Clip (ENC) variant in most scenarios.

B. Weighted Focal Differentiable MCC (diffMCC) Loss

The paper evaluates the diffMCC loss function (previously introduced by Seber, 2024) as an alternative to Weighted Cross-Entropy.

Mechanism: It is a differentiable approximation of the Matthews Correlation Coefficient (MCC), a robust metric for imbalanced classification.
Goal: To align the optimization objective (loss function) directly with the evaluation metric (MCC), thereby reducing the discrepancy between training and testing performance.

C. Experimental Setup

Datasets: Four real-world datasets were used:
1. Heart Failure (Binary): 299 samples, 11 features.
2. Bank Marketing (Binary): 41,188 samples, 20 features (expanded to 54 via one-hot encoding).
3. Wine Quality (Multiclass): Red wine physicochemical properties (1,599 samples, 11 features).
4. Glass Identification (Multiclass): 8 chemical elements (214 samples, 6 classes grouped into 3).
Baselines: The study compared LCEN and MLP-diffMCC against 10 other models, including Logistic Regression (LR), LASSO, Ridge, Elastic Net, Random Forest (RF), Gradient Boosted Decision Trees (GBDT), AdaBoost, SVM, and Multi-Layer Perceptrons (MLP) trained with Cross-Entropy (MLP-CE).
Feature Selection Validation: To test LCEN's feature selection capability, all models were retrained using only the features selected by LCEN. These results were compared against using all features and features selected by other methods (e.g., domain expert selection, statistical tests).

3. Key Contributions

Extension of LCEN to Classification: Successfully modified the LCEN algorithm to perform nonlinear, interpretable feature selection and machine learning for both binary and multiclass classification.
Systematic Evaluation of diffMCC: Demonstrated that training MLPs with the diffMCC loss function consistently outperforms the standard Weighted Cross-Entropy loss across diverse datasets.
Superior Feature Selection: Proved that LCEN-selected features not only create sparse models (eliminating ~56% of features on average) but also improve the performance of other machine learning models when used as input, outperforming feature selection methods based on domain knowledge or statistical tests.
Hyperparameter Guidance: Investigated the LCEN_clip_min_classes_selected hyperparameter, recommending a value no higher than $2/3$ of the number of classes to avoid performance degradation in multiclass settings.

4. Key Results

A. Model Performance (Machine Learning)

diffMCC Superiority: MLPs trained with the diffMCC loss function achieved the highest test-set performance in all four datasets.
- Average Improvement: Compared to MLPs trained with Weighted CE, diffMCC models achieved 4.9% higher Macro F1 scores and 8.5% higher MCCs.
LCEN Performance: LCEN consistently ranked among the top models, often surpassing deep learning models trained with CE.
- LCEN's performance was only 7.3% lower in Macro F1 and 15.0% lower in MCC compared to the absolute best model (usually MLP-diffMCC) on each dataset.
- In the "Bank Marketing" dataset, LCEN outperformed MLP-CE.

B. Feature Selection Impact

Sparsity: LCEN eliminated an average of 56% of input features across experiments.
Performance Boost: Retraining other models using LCEN-selected features led to statistically significant performance improvements in 3 out of 4 datasets.
- Heart Failure: LCEN-selected features improved Macro F1 by 3.3% and MCC by 10.0% compared to using all features.
- Bank Marketing: LCEN-selected features improved Macro F1 by 1.2% and MCC by 7.5%. Notably, using features selected by domain experts (Moro et al., 2014) caused a massive performance drop (MCC decreased by 84.9%), highlighting the superiority of LCEN.
- Glass Identification: LCEN-selected features improved Macro F1 by 5.4% and MCC by 19.1%.
Model Size: Using LCEN-selected features allowed for the training of significantly smaller MLPs (e.g., 78.1% smaller in Heart Failure, 88.4% smaller in Bank Marketing) without sacrificing accuracy.

C. Statistical Significance

Paired t-tests and Tukey's HSD tests confirmed that the performance gains from using LCEN-selected features and the diffMCC loss function were statistically significant (p-values often $< 10^{-4}$ ), ruling out random artifacts.

5. Significance and Conclusion

This work bridges a critical gap in interpretable machine learning by extending LCEN to classification tasks. The results demonstrate that interpretability and high performance are not mutually exclusive.

Practical Impact: The ability of LCEN to select a sparse subset of features that improves the performance of other models suggests that LCEN effectively reduces data variance and removes noise, leading to more robust models.
Loss Function Innovation: The study validates that optimizing directly for classification metrics (via diffMCC) yields superior results compared to the standard Cross-Entropy approach, particularly in scenarios where metric alignment is crucial.
Future Directions: The authors suggest applying LCEN to generate physical equations for hybrid physics-informed machine learning models and expanding the evaluation of diffMCC to non-tabular data (e.g., computer vision and text).

In summary, the paper establishes LCEN as a state-of-the-art, interpretable feature selection and learning algorithm for classification, and diffMCC as a superior loss function for training accurate deep learning classifiers.