💻 computer science

Dynamic ensemble approach for multi-class classification based on neighborhood rough sets and sequential three-way decisions

This paper proposes EM-S3WD, a dynamic ensemble framework that integrates neighborhood rough sets with sequential three-way decisions and a conditional dynamic integration strategy to overcome the limitations of fixed reference tuples and binary constraints in Xu et al.'s original model, thereby achieving adaptive and competitive multi-class classification performance.

Original authors: Wenyan Xu, Qiang Chen, Yangyang Guo

Published 2026-09-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wenyan Xu, Qiang Chen, Yangyang Guo

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of computer science, there is a constant struggle to teach machines how to make decisions when the information they receive is incomplete or messy. Imagine trying to identify a fruit in a basket where some are bruised, some are partially hidden, and the lighting is poor. A simple "yes" or "no" answer often fails because the data is too uncertain. To handle this, researchers have developed a method called three-way decision. Instead of forcing a binary choice, this approach allows the system to say "yes," "no," or "wait and see." The "wait" option is crucial; it acknowledges that the current evidence is not strong enough to make a final call, preventing the machine from guessing wildly. This concept has been refined further into a sequential process, where the system looks at the data through a series of lenses, each one slightly different, to gradually narrow down the possibilities until a confident decision can be made.

Building on this foundation, a team of researchers at Xidian University in China has tackled a specific limitation in how these systems handle multiple categories at once. While the existing methods worked well for distinguishing between two options, they struggled when asked to choose among many, such as identifying different types of seeds or medical conditions. The old approach relied on rigid rules that treated data points as either identical or completely different, which often failed to capture the subtle variations found in real-world numbers. Furthermore, when the system tried to guess the correct category among many, it sometimes hit a deadlock where two or more options appeared equally likely, leaving the computer stuck. The researchers proposed a new framework, which they call EM-S3WD, designed to make these decisions more flexible and to resolve these deadlocks without losing the clarity of the original method.

The core of this new framework lies in how it builds its reference points. In the older systems, the computer created a fixed list of "ideal" examples from the training data. Once this list was made, it never changed, regardless of how the data was distributed or how noisy the environment became. The new approach replaces this rigidity with adaptability. Instead of using strict equality, the system uses a concept called neighborhood rough sets, which allows it to group data points based on how close they are to one another, rather than whether they are exactly the same. Think of it like defining a neighborhood not by a hard fence, but by how far you can walk from a central point before the character of the area changes. By adjusting the size of this neighborhood, the system can select reference examples that fit the specific data it is looking at, making the model much more robust against noise and variation.

Once the system has these flexible reference points, it faces the challenge of sorting data into many different categories. The researchers used a strategy known as "one-versus-all," where the computer builds a separate decision maker for each category, asking, "Is this item part of this group, or is it something else?" Each of these decision makers then outputs a confidence score. However, a problem arises when two or more categories give the exact same highest score. In the past, the computer might have just picked one at random, which is unreliable. The new framework introduces a smart, conditional rescue mechanism. It only activates extra helpers—simpler, secondary classifiers—when it detects this tie. If the scores are clear, the system sticks to its original, primary decision. But if there is a conflict, it brings in the helpers to provide a second opinion, weighing their input based on how much they agree with the primary system and how accurate they have been in the past. This ensures that the system only adds complexity when it is absolutely necessary.

The researchers tested this approach on nine different public datasets, ranging from identifying types of seeds and animals to diagnosing skin conditions and analyzing banknotes. They found that the new method successfully resolved the tie-breaking problems that plagued the older systems. In cases where the primary system was stuck between two equally likely answers, the conditional dynamic integration strategy was able to use the extra evidence to pick the correct category significantly more often than random guessing or fixed weighting methods. For example, on a dataset involving dry beans, the new method correctly identified the conflict and used the auxiliary information to improve the final accuracy. The study also showed that by making the reference points adaptive, the system became better at handling data that had been corrupted by noise, maintaining its performance even when the input data was imperfect.

Despite these successes, the authors are careful not to claim that their method is a universal solution that beats every other existing technique. In head-to-head comparisons with other well-known machine learning algorithms, the new framework performed competitively, often leading in specific metrics like accuracy and consistency, but it did not statistically dominate every other method across every single dataset. The researchers noted that the benefits of their approach are most visible when the data is complex or when the system frequently encounters those difficult tie-breaking situations. The computational cost is also a factor, as the system requires more processing power to calculate the neighborhood relationships and manage the conditional checks. Ultimately, the work demonstrates that by making the reference points flexible and by only using extra resources when a conflict arises, it is possible to build a multi-class classifier that is both more adaptable and more reliable in uncertain situations.

Technical Summary: Dynamic Ensemble Approach for Multi-class Classification Based on Neighborhood Rough Sets and Sequential Three-Way Decisions (EM-S3WD)

1. Problem Statement
This paper addresses two specific limitations in the binary Sequential Three-Way Decision (S3WD) classifier proposed by Xu et al. [19] when extending it to multi-class classification tasks:

Static Reference Representation: The original model relies on Classical Rough Sets (CRS) to generate reference tuples. Because CRS depends on strict equivalence relations, these reference tuples become fixed once the training set is defined. This rigidity limits the classifier's adaptability to continuous-valued data and varying data distributions, as the reference representation cannot adjust based on validation performance.
Confidence Conflicts in Multi-class Extension: A natural extension of a binary classifier to multi-class problems is the "One-Versus-All" (OVA) decomposition strategy. However, applying OVA to S3WD classifiers often results in "confidence conflicts," where multiple binary classifiers assign the same maximal confidence score to a test sample. Standard tie-breaking methods (e.g., random selection or fixed weighting) can reduce decision stability or ignore sample-specific evidence.

2. Methodology: The EM-S3WD Framework
The authors propose EM-S3WD, a dynamic ensemble framework that integrates Neighborhood Rough Sets (NRS), OVA decomposition, and a conditional dynamic integration strategy. The methodology proceeds in three main stages:

NRS-Based Adaptive Reference Tuple Construction:
Instead of using CRS, the framework employs Neighborhood Rough Sets (NRS) to construct reference tuples. In NRS, lower approximations depend on a neighborhood radius ( $\delta$ ), allowing the reference tuples to adapt to local data structures in continuous-valued data.
- Validation-Guided Selection: The neighborhood radius ( $\delta$ ) and the similarity threshold ( $\sigma$ ) are not manually fixed. Instead, a grid search is performed on a validation set to jointly select the optimal pair $(\delta^*, \sigma^*)$ that maximizes validation accuracy. The reference tuples are then generated based on this optimal $\delta^*$ .
OVA-Based Multi-class Extension:
The framework constructs $m$ binary NRS-based S3WD classifiers for an $m$ -class problem. Each classifier distinguishes one class from all others.
- Finite-Valued Confidence: Theoretical analysis (Proposition 2) reveals that the confidence scores generated by these classifiers belong to a finite set determined by the count of matching attributes and the decision level. This structural property explains why different classifiers frequently produce identical maximal scores, leading to confidence conflicts.
Conditional Dynamic Integration for Conflict Resolution:
The framework introduces a conditional strategy to handle confidence conflicts without replacing the primary decision mechanism:
- Non-Conflict Cases: If the OVA-based classifiers produce a unique maximum confidence score, the original decision is preserved.
- Conflict Cases: If two or more classes share the maximal confidence score, auxiliary classifiers (specifically SVM and Naive Bayes) are invoked.
- Weighting Mechanism: The final decision is made by dynamically weighting the outputs of the main S3WD classifiers and the auxiliary classifiers. The weights are determined by:
  1. The validation accuracy of the auxiliary classifiers.
  2. The Kullback–Leibler (KL) divergence between the main S3WD output and the auxiliary output on the conflicting classes.
- This ensures that auxiliary evidence is used only when necessary and weighted according to its consistency with the main model and its own reliability.

3. Key Contributions
The paper outlines four primary contributions:

Adaptive Reference Construction: Development of an NRS-based mechanism where reference tuples depend on neighborhood granularity. A validation-guided procedure is introduced to jointly select the neighborhood radius and similarity threshold, enhancing adaptability to data distributions.
Structural Analysis of Conflicts: Theoretical analysis demonstrating that confidence conflicts in OVA-based S3WD extensions arise naturally from the finite-valued nature of the confidence scores (determined by matched-attribute counts and decision levels).
Conditional Dynamic Integration: A novel strategy that resolves confidence conflicts by activating auxiliary classifiers only for conflict samples. The integration dynamically weights these auxiliary outputs based on KL divergence and validation performance, preserving the original S3WD decision for non-conflict samples.
Comprehensive Evaluation: Theoretical proofs regarding finite-step termination and decision preservation, alongside extensive experiments on nine benchmark datasets evaluating parameter sensitivity, conflict resolution, noise robustness, and overall performance.

4. Experimental Results
Experiments were conducted on nine UCI benchmark datasets (e.g., Iris, Seeds, DryBean, PageBlocks) using MATLAB.

Parameter Sensitivity: The study confirms that both the neighborhood radius ( $\delta$ ) and similarity threshold ( $\sigma$ ) are highly sensitive parameters. Optimal values vary significantly across datasets, validating the necessity of the validation-guided selection process.
Conflict Resolution: When evaluated specifically on samples with confidence conflicts, EM-S3WD achieved the highest mean accuracy on eight of the nine datasets compared to random tie-breaking (SM-S3WD) and fixed-weight integration (MM-S3WD).
Noise Robustness: The adaptive reference tuple construction demonstrated improved robustness against Gaussian noise on several datasets (particularly Iris, Seeds, Dermatology, and Abalone) compared to fixed reference tuple methods.
Overall Performance: In comparisons with six representative classifiers (including Random Forest, AdaBoost, KNN, and 3WD-NRS), EM-S3WD achieved competitive results. It outperformed all competitors on the DryBean dataset across all metrics and led in Accuracy and Kappa on Balance-scale and Abalone.
Statistical Significance: While EM-S3WD showed strong performance, Friedman and Holm-adjusted Wilcoxon tests indicated that it is statistically on par with the strongest baselines but does not demonstrate statistically significant superiority over every individual competitor across all datasets and metrics.

5. Significance and Claims
The paper positions EM-S3WD as a targeted model-level extension of Xu et al.'s binary S3WD classifier. Its primary significance lies in:

Adaptability: Moving from static CRS-based reference tuples to adaptive NRS-based tuples that respond to data distribution via validation-guided parameter selection.
Explicit Conflict Handling: Providing a structured, sample-dependent mechanism to resolve confidence conflicts in multi-class S3WD scenarios, rather than relying on arbitrary tie-breaking.
Interpretability: Maintaining the interpretability of the S3WD framework while extending it to multi-class problems.

The authors modestly claim that the framework's contribution is its adaptive reference representation and explicit conflict-aware decision mechanism. They acknowledge that while the method is competitive, it does not universally outperform all existing ensemble or rough-set-based classifiers across all scenarios. The computational cost is noted as a limitation due to neighborhood computations and parameter searches, suggesting future work in efficient search mechanisms and alternative auxiliary model selection.

More like this