🤖 machine learning

A Leakage-Free Stacked Ensemble Method for Multiclass Classification

This paper introduces LFS-FRAME, a leakage-free stacked ensemble framework that combines Kolmogorov-Arnold Networks and XGBoost with a strict out-of-fold strategy to achieve robust and generalizable multiclass classification by effectively integrating functional patterns and rule-based decision boundaries.

Original authors: S. P. Sharmila, Aruna Tiwari

Published 2026-07-27

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: S. P. Sharmila, Aruna Tiwari

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a computer to recognize different types of animals just by looking at their photos. This is a classic puzzle in the world of machine learning called "multiclass classification." It's like asking a student to sort a mixed bag of marbles not just into "red" and "blue," but into "red," "blue," "green," "yellow," "purple," and so on, all at once. The trouble is, some marbles look suspiciously alike, and sometimes you have a huge pile of red ones but only a few green ones.

To solve this, scientists often use "ensemble learning," which is a fancy way of saying "let's ask a committee of experts instead of just one." Imagine a panel of judges: one is great at spotting patterns in smooth, flowing shapes (like a painter), and another is great at following strict, step-by-step rules (like a detective). If you ask them to vote, you usually get a better answer than if you asked just one. However, there's a sneaky trap in how these committees are often formed. If the judges are allowed to peek at the answer key while they are being trained, they might start "cheating" by memorizing the answers instead of actually learning the patterns. This is called "data leakage," and it makes the computer look super smart in the classroom but terrible in the real world.

This paper introduces a new way to build that committee, called LFS-FRAME. The authors, S. P. Sharmila and Aruna Tiwari, created a system that combines two very different types of computer brains: one based on Kolmogorov-Arnold Networks (KAN), which are like smooth, flowing artists good at understanding complex curves, and another based on XGBoost, which is a rule-following detective good at making sharp, clear decisions. The magic of their method isn't just in mixing these two; it's in how they train them. They use a strict "out-of-fold" strategy, which is like making sure the judges practice on a set of questions they have never seen before, so they can't cheat. This ensures that when the final "head judge" (the meta-classifier) combines their votes, it's based on honest, unbiased opinions.

The researchers tested this new system on a challenging dataset involving 16 different categories of malware (computer viruses) hidden in computer memory. They found that their "leakage-free" team was significantly better at sorting these tricky categories than previous methods. While older techniques struggled when the number of categories grew, dropping in accuracy, LFS-FRAME stayed strong. In a test with 16 different classes, their method achieved an accuracy of 81.74%, and when looking at broader groups of 4 classes, it hit 89.85%. This suggests that by keeping the training process honest and mixing smooth learning with rule-based learning, we can build much more reliable computers for sorting complex data.

The Problem: The Cheating Committee

To understand why this paper matters, let's look at the problem it fixes. In the world of machine learning, "stacking" is a popular technique where you take the predictions from several different models and feed them into a final model to make the ultimate decision. Think of it like a sports team where you have a striker, a defender, and a goalie. You ask them all, "Who should we pick for the team?" and then a coach (the meta-learner) makes the final call based on their answers.

The problem arises when the coach is trained using the answers the players gave during their own practice. If the players practiced on the exact same questions the coach is testing them on, they might just memorize the answers. This is "data leakage." The coach thinks the team is a genius because they got 100% on the practice test, but when they face a real game with new questions, they fail miserably.

The authors argue that many existing stacking methods suffer from this. They let the base models see the data they are supposed to predict on, which inflates the scores and gives a false sense of security. This is especially dangerous in "multiclass" problems, where there are many categories to choose from. If the system is cheating, it might look great at spotting 4 types of malware but fall apart when asked to distinguish between 16.

The Solution: The "No-Peeking" Rule

The paper proposes LFS-FRAME (Leakage-Free Stacked Framework). The core idea is simple but powerful: No model is allowed to see the data it is predicting on.

They achieve this using a technique called Out-of-Fold (OOF) training. Imagine you have a deck of cards (your data) and you split it into 5 piles (folds).

You take 4 piles to train your models.
You leave the 5th pile hidden in a box.
You ask the models to predict the cards in that hidden 5th pile. Since they haven't seen these cards before, their predictions are honest.
You repeat this process, rotating which pile is hidden, until every card has been predicted on by a model that didn't know it was coming.

These "honest" predictions are then used to train the final "head judge" (the meta-classifier). Because the head judge was trained on predictions made by models that had never seen the specific data points, there is no cheating. The system learns to combine the strengths of its members without relying on memorized answers.

The Team: Artists and Detectives

The authors didn't just fix the training method; they also picked a very specific team of models to work together. They realized that different problems need different kinds of thinking.

The Artist (KAN): They used Kolmogorov-Arnold Networks (KANs). Think of KANs as artists who are really good at understanding smooth, flowing relationships. They can see how variables change together in a continuous curve. They are great at capturing the "big picture" and complex, non-linear patterns. However, they sometimes struggle with sharp, sudden changes or very specific rules.
The Detective (XGBoost): They used XGBoost, a powerful tool based on decision trees. Think of XGBoost as a detective who follows a strict checklist of rules. "If the door is open, check the window. If the window is broken, call the police." It is excellent at making sharp, clear decisions and handling specific, rule-based patterns.

By combining the Artist and the Detective, the system gets the best of both worlds. The KAN handles the smooth, complex curves in the data, while XGBoost handles the sharp, distinct boundaries. The "head judge" learns how to weigh the Artist's intuition against the Detective's rules to make the best final call.

The Results: Proving It Works

The authors tested their new system on a dataset called CIC-MalMem-2022, which contains memory data from computer attacks. They created two versions of the test: one with 4 classes (broader categories) and one with 16 classes (very specific sub-categories).

They compared their LFS-FRAME against other methods that had been used before, such as:

HyStack Ensemble: A previous stacking method.
Hybrid CNN-BiLSTM: A deep learning approach.
SMOTE-DNN: A method that tries to balance the data.
Random Forest with Hyperparameter Tuning: A classic rule-based approach.

Here is what they found:

In the 4-Class Test: The new method achieved an accuracy of 89.85%. This was slightly better than the Random Forest method (which got 89.07%) and significantly better than the deep learning methods.
In the 16-Class Test (The Hard Part): This is where the other methods started to crumble. As the number of categories increased, the accuracy of the other methods dropped sharply.
- The HyStack method fell from 85.04% (in 4 classes) down to 70.29%.
- The Random Forest method dropped from 89.07% down to 68.2%.
- The deep learning methods also struggled, falling into the 60-70% range.
- LFS-FRAME, however, held its ground. It achieved 81.74% accuracy in the 16-class test.

The authors suggest that the reason their method succeeded is twofold. First, by preventing data leakage, they ensured the system was actually learning patterns and not memorizing answers. Second, by mixing the smooth learning of KANs with the rule-based strength of XGBoost, they created a system that could handle the complexity of having 16 different categories without getting confused.

Why It Matters

The paper concludes that this approach offers a more reliable way to handle complex classification problems. In the real world, especially in fields like cybersecurity where malware is constantly changing, you cannot afford a system that looks smart in the lab but fails in the field. By using a "leakage-free" strategy, the authors provide a framework that gives a more honest estimate of how well a model will perform.

While the method does require a bit more computing power because of the extra steps needed to ensure no data leakage (training models multiple times on different subsets), the authors argue this cost is worth it. It prevents the "over-optimistic" results that plague other methods and leads to a system that is robust, generalizable, and ready for the real world. The study suggests that for difficult multiclass problems, combining different types of learning styles in a strictly honest training environment is a winning strategy.

Technical Summary: A Leakage-Free Stacked Ensemble Method for Multiclass Classification

Problem Statement
Multiclass classification remains a challenging task in machine learning, particularly when datasets exhibit high inter-class similarity, class imbalance, and variability in data distributions. While rule-based classifiers like XGBoost excel on structured features, they often struggle to capture smooth functional relationships. Conversely, neural network models can represent complex nonlinear interactions but are prone to overfitting and generalization issues. Furthermore, standard stacking ensemble methods face critical limitations:

Data Leakage: Training meta-learners on in-sample predictions from base models inadvertently leaks label information, leading to unrealistically inflated performance metrics.
Meta-learner Overfitting: High-dimensional probability outputs from base learners can cause the meta-learner to overfit.
Model Diversity: Correlated base models provide redundant signals, diminishing the benefits of ensembling.
Generalization Failure: In multiclass settings, error propagation is amplified, and traditional stacking often fails to generalize to unseen data, particularly as the number of classes increases.

Methodology: LFS-FRAME
The authors propose LFS-FRAME (Leakage-Free Stacked Ensemble Framework), a structured integration of functional and rule-based learning designed specifically for robust, leakage-free multiclass classification.

Base Learners: The framework employs two complementary base models:
- Kolmogorov–Arnold Networks (KAN): A functional learning paradigm based on the Kolmogorov–Arnold representation theorem. KANs approximate multivariate functions through compositions of univariate nonlinear functions, effectively capturing smooth, global functional patterns and continuous feature interactions.
- XGBoost: A rule-based gradient-boosted decision tree model. XGBoost excels at learning sparse, non-linear decision rules and capturing sharp, localized decision boundaries.
Leakage-Free Stacking Strategy: The core innovation is a strict Out-of-Fold (OOF) training protocol to prevent information leakage:
1. The dataset is partitioned into $K$ folds using stratified cross-validation.
2. For each fold $k$ , base models are trained only on the training subset ( $D \setminus D_k$ ).
3. Predictions (class probability vectors) are generated only for the held-out validation subset ( $D_k$ ).
4. These OOF predictions are concatenated to form an unbiased meta-feature matrix ( $Z_{oof}$ ), ensuring that no base model has seen the specific sample it is predicting during its training phase.
5. A multinomial meta-classifier is trained on $Z_{oof}$ to learn optimal fusion weights.
6. For inference on unseen test data, base models are retrained on the full dataset, and their probability outputs are fused via the trained meta-classifier.

Key Contributions

Principled Leakage Prevention: The paper formalizes a leakage-free stacking definition where meta-features are constructed strictly from OOF predictions, eliminating the correlation between training labels and base model parameters that plagues traditional stacking.
Hybrid Functional-Rule Integration: The framework uniquely combines KANs (for smooth functional approximation) and XGBoost (for sharp rule-based boundaries), leveraging complementary inductive biases to handle both continuous interactions and discrete decision regions.
Robust Multiclass Scalability: The method addresses the degradation of performance often seen in high-cardinality classification tasks (e.g., moving from 4-class to 16-class problems) by ensuring unbiased meta-learning.

Experimental Results
The framework was evaluated on the CIC-MalMem-2022 dataset (volatile memory malware analysis) and an EnhancedCIC dataset (augmented with synthetic samples via GANs and interpolation). The experiments focused on 4-class (4CC) and 16-class (16CC) classification tasks.

Performance: LFS-FRAME achieved 89.85% accuracy in identifying major families (4CC) and 81.74% in identifying sub-families (16CC).
Comparison:
- Compared to HyStack (85.04% on 4CC, dropping to 70.29% on 16CC), LFS-FRAME showed significant gains in the high-class regime.
- Compared to Hybrid CNN-BiLSTM and SMOTE-DNN, which suffered from performance drops in 16CC scenarios (falling to ~68-72%), LFS-FRAME maintained superior stability.
- Compared to Random Forest with hyperparameter tuning (89.07% on 4CC, dropping to 68.2% on 16CC), the proposed method demonstrated better generalization as decision boundaries became more complex.
Baseline Comparison: The ensemble outperformed individual KAN-only (79.32% on 16CC) and XGBoost-only (79.32% on 16CC) baselines on the EnhancedCIC dataset, validating the benefit of the hybrid approach.

Significance and Claims
The paper claims that LFS-FRAME establishes a reliable and generalizable strategy for multiclass classification by resolving the tension between functional and rule-based learning while rigorously eliminating data leakage.

Reliability: By using OOF predictions, the framework ensures that performance estimates are unbiased and that the meta-learner does not overfit to training-specific patterns.
Scalability: Unlike existing methods that degrade significantly as the number of classes increases, LFS-FRAME maintains macro-level performance in high-class regimes, making it suitable for fine-grained classification tasks like malware family detection.
Theoretical Grounding: The approach is grounded in functional approximation theory (via KAN) and ensemble theory, offering a more principled alternative to empirical stacking methods.

The authors conclude that while leakage-free stacking introduces a linear computational overhead due to $K$ -fold training, this cost is necessary to ensure statistical validity and robust generalization, particularly in adversarial environments like malware detection where over-optimistic metrics from traditional methods can lead to failure in real-world deployment.