💻 computer science

Mask-Aware MLP and Entropy-Weighted MCDM for Robust Multiclass Paper Ranking

This paper proposes a robust framework combining a mask-aware multilayer perceptron for imputing missing evaluation metrics and an entropy-weighted MCDM method for ranking scientific articles, demonstrating superior performance in accuracy, imputation error reduction, and ranking preservation under moderate-to-high missingness compared to existing techniques.

Original authors: Masoud Kargar, Shahin Sharbaf Movassaghpour, Habib Izadkhah

Published 2026-09-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Masoud Kargar, Shahin Sharbaf Movassaghpour, Habib Izadkhah

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast landscape of modern science, researchers constantly rely on machine learning to solve complex problems, from detecting early signs of disease to recognizing objects in images. To know if these computer programs are working well, scientists measure their success using specific numbers, such as how often they are correct or how well they find the right answer when it is rare. These measurements are usually derived from a table that tracks every success and failure of the model. However, a persistent problem plagues the scientific community: when researchers publish their findings, they often leave out parts of this data, report numbers that do not quite add up, or make simple calculation errors. When these pieces are missing or wrong, it becomes nearly impossible for other scientists to verify the results or fairly compare one study against another. Without a complete and accurate picture, the trustworthiness of the entire field can be compromised, making it difficult to know which methods are truly the best.

To address this challenge, a team of researchers has developed a new system designed to fix broken or incomplete reports and then use them to rank scientific studies fairly. They created a computer program that acts like a careful editor, capable of spotting when a set of reported numbers is inconsistent or missing a piece. The system works by first generating thousands of perfect, simulated examples of how these performance numbers should look when everything is correct. It then teaches a neural network, a type of computer brain, to recognize the patterns of these correct reports while also learning to identify when numbers have been corrupted by error or are simply missing. Crucially, the program does not just guess the missing numbers; it pays close attention to the fact that they are missing. It uses a special indicator, a digital flag that marks exactly which values are present and which are absent, to help the computer understand the context of the data. This allows the system to fill in the gaps more accurately than older methods, especially when a large portion of the data is gone.

Once the missing numbers are estimated and the validity of the report is checked, the system moves to the second stage: ranking. It takes the now-complete set of numbers and applies a decision-making method that weighs each metric based on how much it varies across the different studies. This ensures that the final ranking reflects the true performance of the models rather than being skewed by the way the data was reported. The researchers tested this approach using simulated data where they knew the correct answers in advance. They found that the system could successfully identify invalid reports and fill in missing values with high accuracy. When the amount of missing data was low, the new system performed just as well as existing methods. However, when the missing data increased to thirty or forty percent, the new system outperformed all other techniques, producing the most accurate estimates and preserving the correct order of the rankings better than any other method.

The team also applied their tool to real-world examples from published studies in medical imaging and heart disease detection. In these tests, the system flagged specific reports where the numbers did not align, such as a study where the reported score for success did not match the precision and recall numbers provided. In another case, it identified a set of metrics that could not be found in the original source material at all. These findings suggest that the tool can serve as a powerful screening mechanism, helping editors and researchers quickly spot reports that need further verification before they are accepted as fact. The researchers built a user-friendly application that allows anyone to upload their data, see which values are estimated, and receive a ranked list of the studies based on their corrected performance.

The study makes it clear that no single method works best in every situation. While older techniques are still very good when only a few numbers are missing, the new approach shines when the data is sparse or heavily incomplete. The key insight is that explicitly telling the computer which numbers are missing helps it make better guesses about what those numbers should be. This is particularly important because simply filling in a blank number is not enough; the goal is to ensure that the final comparison between different scientific models remains fair and accurate. By combining the ability to fix broken data with a robust method for ranking, the researchers have provided a way to restore confidence in scientific comparisons, ensuring that the best methods are recognized even when the original reports are imperfect. The work suggests that in a world where data is often incomplete, acknowledging what is missing is just as important as analyzing what is there.

Technical Summary: Mask-Aware MLP and Entropy-Weighted MCDM for Robust Multiclass Paper Ranking

Problem Statement
The scientific evaluation of classification models relies heavily on performance metrics such as accuracy, precision, recall, specificity, and F1-scores. However, a persistent challenge in the literature is the incomplete or erroneous reporting of these metrics. Studies often omit confusion matrices or report only a subset of essential metrics, hindering independent verification and fair cross-study comparisons. Furthermore, reported values may contain calculation errors or inconsistencies due to implementation flaws. Existing approaches to handle missing data (e.g., mean imputation, KNN, MICE, MissForest) generally treat numerical values as the primary source of information, failing to explicitly represent the pattern of missingness. Additionally, while Multi-Criteria Decision Making (MCDM) methods are used to rank models, they are rarely integrated with robust missing-value reconstruction that accounts for the specific structural relationships between classification metrics. Consequently, there is a lack of automated frameworks that can simultaneously screen for inconsistent metric profiles, impute missing values while respecting missingness patterns, and preserve the integrity of downstream model rankings.

Methodology
The authors propose an integrated framework comprising three main stages: simulated dataset generation, mask-aware metric reconstruction, and entropy-weighted ranking.

Simulated Dataset Generation: To train and evaluate the system, the authors generated a controlled dataset derived from confusion matrices for binary and multiclass scenarios (2 to 10 classes).
- Data Construction: Confusion matrices were generated using probabilistic distributions (Dirichlet for class proportions, Beta for prediction probabilities) to ensure internal consistency. Standard metrics were derived from these matrices.
- Noise and Missingness: To simulate real-world imperfections, Gaussian noise was added to create "noisy" (erroneous) profiles, and random feature masking was applied to create "incomplete" profiles at varying missingness levels (10% to 40%).
- Labels: Each record was labeled as "Valid" (internally consistent) or "Invalid" (perturbed/noisy).
Mask-Aware Multilayer Perceptron (MaskAwareMLP):
- Architecture: A deep neural network was developed with five dense layers, batch normalization, and LeakyReLU activations.
- Input Mechanism: Unlike standard imputation models, the MaskAwareMLP receives two inputs: the feature vector (where missing values are initially filled via KNN for training stability) and a binary observation mask. This mask explicitly indicates which values were originally observed (1) and which were missing (0).
- Dual Objective: The model is trained to perform two tasks: (a) classify records as valid or invalid to screen for inconsistent reporting, and (b) estimate missing metric values.
- Imputation Strategy: For missing values, a multi-stage search procedure (coarse, fine, and super-fine) is employed to generate candidate values, which are then evaluated by the model's confidence scores to select the optimal estimate.
Entropy-Weighted MCDM Ranking:
- Validated articles (those classified as "Valid" by the MLP) are ranked using an entropy-weighted TOPSIS approach.
- Entropy Weighting: Criterion weights are derived objectively based on the dispersion of metric values across the articles, reducing subjective bias.
- Ranking: The weighted metrics are aggregated to produce a final ranking of the scientific articles.

Key Contributions

Mask-Aware Architecture: The primary methodological novelty is the explicit incorporation of a binary missingness mask into the MLP learning process. This allows the model to distinguish between observed measurements and imputed values, leveraging the missingness pattern as an additional source of information.
Integrated Workflow: The study integrates missing-value imputation, validity screening, and MCDM-based ranking into a single automated workflow, addressing the gap between data reconstruction and decision-level objectives.
Dual Evaluation Metric: The framework evaluates performance not only at the cell level (using MAE, RMSE, $R^2$ ) but also at the decision level, measuring the preservation of model rankings (using Kendall's $\tau$ -b and Spearman's $\rho$ ) against a complete-data reference.
Controlled Benchmarking: The creation of a dedicated simulated dataset with known ground-truth consistency and controlled missingness patterns enables rigorous, reproducible evaluation of imputation and ranking robustness.
User-Friendly Tool: The methodology is implemented as a graphical user interface (GUI) application that supports data entry, automated imputation, reliability assessment, and visual ranking.

Experimental Results

Validity Classification: The MaskAwareMLP achieved classification accuracies between 0.8961 and 0.9274 and F1-scores between 0.90 and 0.93 across 17 different class configurations (2–10 classes), demonstrating robustness in distinguishing consistent from inconsistent metric profiles.
Imputation Performance:
- At low missingness levels (10–20%), conventional iterative methods like MissForest and MICE performed competitively, often achieving lower error rates than the proposed method.
- At moderate-to-high missingness levels (30–40%), MaskAwareMLP outperformed all competing methods (including KNN, MICE, MissForest, and standard MLP), achieving the lowest MAE and RMSE and the highest $R^2$ .
- Ablation Study: Comparing MaskAwareMLP against a standard MLP (without the mask) showed that the mask-aware variant consistently reduced reconstruction errors and improved $R^2$ , with the performance gap widening as missingness increased.
Ranking Preservation:
- In ranking-preservation experiments, MaskAwareMLP achieved the highest Kendall's $\tau$ -b (0.7879 at 30% missingness; 0.7165 at 40% missingness) and Spearman's $\rho$ among all methods at higher missingness levels.
- While iterative methods excelled at low missingness (10–20%), the mask-aware approach demonstrated superior ability to maintain the relative ordering of models when data was significantly incomplete.
Real-World Application: When applied to published studies in medical and image-processing domains, the framework successfully identified potential inconsistencies, such as F1-scores that did not align with reported precision and recall, and flagged articles where reported metrics could not be located in the source text.

Significance and Claims
The paper claims that its principal contribution lies not in introducing a fundamentally new imputation algorithm or MCDM method in isolation, but in the integration of mask-aware learning, metric-consistency assessment, and entropy-weighted ranking. The authors argue that explicitly representing the missingness pattern provides information unavailable to conventional models, which is particularly beneficial under moderate-to-high missingness conditions.

The study emphasizes that accurate cell-level reconstruction does not guarantee the preservation of downstream rankings. Therefore, the significance of the proposed framework is its ability to link data reconstruction directly to the decision-level objective of comparative model evaluation. The results suggest that while no single imputation method is universally optimal across all missingness levels, the MaskAwareMLP offers a robust solution for preserving ranking integrity when data is incomplete, thereby supporting more transparent and reliable scientific comparisons. The authors modestly note that the model's performance is based on simulated data generation and that future work should focus on validation with larger, independently verified real-world datasets and more diverse missingness mechanisms.

More like this