WISCA: A Consensus-Based Approach to Harmonizing Interpretability in Tabular Datasets

1. Problem Statement

While Machine Learning (ML) models achieve high predictive accuracy, their "black-box" nature poses significant challenges in high-risk domains such as healthcare and finance, where transparency and interpretability are legally and ethically mandated.

• The Core Conflict: Existing Explainable AI (XAI) algorithms (e.g., SHAP, LIME, Permutation Importance) often provide contradictory explanations for the same model prediction. Different algorithms may highlight different features or assign different importance magnitudes.
• Limitations of Current Consensus Methods: To resolve these conflicts, researchers have attempted to aggregate explanations using simple consensus functions (e.g., arithmetic mean, voting, geometric mean). However, these traditional methods suffer from critical flaws:
  • They frequently ignore model accuracy (a reliable prediction is treated the same as a random guess).
  • They fail to normalize assignment scales across different algorithms (e.g., SHAP values vs. Permutation Importance).
  • They do not distinguish between global (dataset-wide) and local (sample-specific) explanations, leading to biases during aggregation.
  • They struggle with negative values or zero assignments, which can distort results (e.g., the geometric mean collapses to zero if a value is zero).

2. Methodology

The authors propose a novel framework to evaluate and improve consensus in interpretability.

A. Experimental Setup

• Datasets: Six synthetic datasets with known ground truths (specific features used to generate the target variable) were created. This enables an objective assessment of whether an explanation correctly identifies the relevant features.
  • 2 binary classification, 2 multiclass classification, 2 regression.
  • The features included relevant variables and significant noise (irrelevant variables).
• ML Models: Six diverse models were trained: Logistic Regression (LR), K-Nearest Neighbors (KNN), Random Forests (RF), Support Vector Machines (SVM), Extreme Gradient Boosting (XGB), and Artificial Neural Networks (ANN).
• Interpretability Algorithms: A mix of global and local, model-agnostic methods was used:
  • Global: Mean Decrease in Impurity (MDI), Permutation Importance.
  • Local: LIME, SHAP, Integrated Gradients, Counterfactual Explanations.

B. The Proposed Solution: WISCA

The authors introduce WISCA (Weighted Scaled Consensus Attributions), a new consensus function designed to harmonize explanations by addressing the limitations of existing methods.

Key Components of WISCA:

1. Min-Max Scaling: All attributions from different algorithms are normalized to a range of $[0, 1]$ to ensure a fair comparison.
2. Differentiation Between Global and Local:
   • Global methods provide one value per feature.
   • Local methods provide values per sample. WISCA aggregates local attributions by dividing them by the total number of samples ($N$) to ensure they contribute proportionally to global methods.
3. Accuracy-Based Weighting (Correction Factors):
   • Classification: Attributions are weighted based on class probability ($\pi$). Predictions with high confidence (probabilities near 0 or 1) are heavily weighted, while uncertain predictions (probability near 0.5) are down-weighted. The authors chose a parabolic function ($1 - 4p(1-p)$) for this correction.
   • Regression: Attributions are weighted based on prediction error ($|y - \hat{y}|$). Predictions with low error (high reliability) retain their attribution weighting, while predictions with high error are penalized with an exponential decay factor ($e^{-\alpha|y-\hat{y}|}$).

Formulation:
$$\phi(f) = \sum_{s \in S} \frac{\phi'(f)}{N} \times \text{Correction Factor}$$
Where $\phi'(f)$ is the scaled attribution and the correction factor depends on the problem type (classification probability or regression error).

C. Evaluation Metrics

• Hit-Rate Metric: A custom metric to calculate the precision of feature ranking. It assigns higher scores when the "ground truth" features appear at the top of the consensus list.
• Separation Distance: Measures the gap between the minimum attribution of expected features and the maximum attribution of noise features.
• Statistical Correlation: Spearman correlation and Jensen-Shannon divergence were used to compare WISCA with individual algorithms.

3. Main Contributions

1. Novel Consensus Function (WISCA): A robust method that integrates model accuracy (via probability or error) and scales attributions, thereby overcoming the biases of simple averaging or voting.
2. Comprehensive Benchmarking: A systematic evaluation of five existing consensus functions (Arithmetic Mean, Harmonic Mean, Geometric Mean, Voting, Relative Position) against WISCA across diverse synthetic scenarios.
3. Identification of Flaws in Existing Methods: The study demonstrates that:
   • The Harmonic Mean is unsuitable because it inversely weights importance.
   • The Geometric Mean fails when an attribution is zero or negative.
   • Voting is heavily dependent on model accuracy and ignores the magnitude of contributions.
4. Validation with Real Data: Application of WISCA to real-world datasets (Cervical Cancer Risk, Wine Origin, Bike Sharing) to demonstrate logical consistency in feature selection.

4. Results

• Synthetic Datasets: WISCA consistently outperformed other consensus methods.
  • In regression datasets, WISCA was the only function that correctly identified all expected features across all models.
  • In classification datasets, both WISCA and Voting performed well, but WISCA showed lower standard deviation (more compact/consistent attributions) and better performance in complex multiclass scenarios.
  • Traditional methods like geometric and harmonic means performed poorly.
• Correlation with Ground Truth: WISCA showed high Spearman correlation with the "best" single interpretability algorithm for each specific model, indicating it effectively synthesizes the most reliable signals.
• Comparison with Linear Models: The feature rankings from WISCA were comparable to, and in some cases better than, those of a Logistic Regression model (which is inherently interpretable), validating its ability to extract true causal features.
• Real-World Application:
  • Cervical Cancer: Correct identification of the "Schiller" test as an important marker.
  • Wine: Correct identification of "Proline" as a marker for wine origin.
  • Bike Sharing: Correct identification of "Registered Users" as the main driver for rentals.

5. Significance and Conclusion

The paper states that consensus is essential for reliable XAI, but simple averaging is insufficient. By integrating model confidence and normalization, WISCA offers a more trustworthy explanation of black-box models.

• Impact: This approach reduces uncertainty for stakeholders in critical domains (medicine, finance) by providing explanations that are not only consistent across algorithms but also weighted by the reliability of the underlying prediction.
• Future Work: The authors suggest extending WISCA to handle class-specific probabilities and exploring the use of ML models to predict feature attributions based on existing interpretations.

In summary, WISCA represents a significant step toward harmonizing interpretability, moving away from subjective or contradictory explanations toward a robust, accuracy-weighted consensus that closely aligns with the true decision logic of ML models.