Does Graph Attention Improve Weighted-Polling Classification on the Pima Indians Diabetes Dataset? A Nested Cross-Validation Comparison with Standard Models

Technical Summary: Does Graph Attention Improve Weighted-Polling Classification on the Pima Indians Diabetes Dataset?

Problem Statement
The study addresses the question of whether Graph Attention Network (GATv2) encoders improve weighted-polling classification on small, structured clinical tabular datasets. While GATv2-based architectures have shown promise in agricultural morphology data (specifically the Dry Bean dataset), it is unknown if they generalize to clinical domains characterized by smaller instance counts, class imbalance, and data quality issues such as zero-encoded missing values. A critical methodological problem identified is "evaluation leakage" in transductive graph pipelines: if an encoder is trained on the labels of nodes that are subsequently scored, performance is artificially inflated. Furthermore, selecting hyperparameters on the same folds used for evaluation introduces optimism. This paper seeks to determine if GATv2 augmentation offers a genuine performance benefit over raw-feature weighted polling and standard classifiers when evaluated under a rigorous, leakage-free nested cross-validation protocol.

Methodology
The research employs a two-stage pipeline: a GATv2 encoder that generates relational embeddings from a k-nearest-neighbour (k-NN) similarity graph, followed by a weighted-polling classifier (KNN/Naive Bayes variants) that aggregates votes from nearest neighbours in the latent space.

• Dataset: The Pima Indians Diabetes Dataset (768 instances, 8 features, binary outcome). The data exhibits class imbalance (65% negative, 35% positive) and known data quality issues where missing values in five features are recorded as zeros.
• Preprocessing: Missing values (zeros) were imputed using the median of the training fold only. Features were normalized via z-score.
• Graph Construction: A k-NN graph was built using Euclidean distance on the feature vectors. Edge weighting (inverse-distance) was evaluated as a hyperparameter.
• Nested Cross-Validation: To prevent leakage and overfitting, a nested stratified cross-validation protocol was used:
  • Outer Loop: 10-fold cross-validation, repeated three times with different seeds.
  • Inner Loop: 5-fold cross-validation used exclusively for hyperparameter tuning. This included tuning the encoder (graph size, edge weighting, hidden width, attention heads, dropout, learning rate, weight decay, epochs) and the polling configuration (number of neighbours, voting scheme).
  • Constraint: The encoder was retrained within every fold using only training labels; test node features were visible for graph construction, but their labels were never used in the loss function.
• Comparators: The tuned GATv2 classifier was compared against:
  1. Raw-feature weighted polling (no encoder).
  2. GATv2 with the original (untuned) encoder.
  3. Five standard classifiers: Logistic Regression, RBF Support Vector Machine (SVM), Random Forest, Gradient Boosting, and k-Nearest Neighbours (k-NN).
• Metrics: Accuracy, macro-F1, sensitivity (recall of diabetic class), specificity, and Area Under the ROC Curve (AUROC). Statistical significance was assessed using 95% bootstrap confidence intervals for paired differences.

Key Contributions

1. Rigorous Evaluation Protocol: The study implements a nested cross-validation framework where the encoder, graph structure, and polling configuration are all tuned strictly within the inner loop, eliminating data leakage common in transductive graph evaluations.
2. Direct Within-Architecture Comparison: It provides a paired comparison of GATv2-augmented weighted polling against raw-feature weighted polling on identical folds, isolating the specific contribution of the graph attention mechanism.
3. Quantification of Evaluation Bias: The paper quantifies the performance inflation caused by the original protocol (scoring nodes used to train the encoder), demonstrating a significant gap between optimistic and realistic estimates.
4. Open Reproducibility: The code for the nested protocol, the original evaluation, and label-shuffle controls is publicly released.

Results

• Performance: The tuned GATv2 classifier achieved 75.22% accuracy and an AUROC of 0.808.
• Comparison to Raw Features: The tuned GATv2 did not outperform raw-feature weighted polling (75.35% accuracy). The paired difference in accuracy was −0.13 points (95% CI: −2.13 to +1.87). While a benefit of up to ~1.9 points cannot be excluded, differences larger than ~2 points in either direction are excluded.
• Comparison to Standard Models: The tuned GATv2 underperformed compared to standard classifiers. Logistic Regression achieved the highest accuracy (76.74%). The GATv2 AUROC was significantly lower (by 1.4 to 2.5 points) than Logistic Regression, SVM, Random Forest, and k-NN, with confidence intervals excluding zero for these comparisons.
• Sensitivity vs. Specificity: GATv2 variants showed higher sensitivity (58.58%–62.56%) but lower specificity compared to some standard models, indicating a shift in the operating point rather than improved discrimination.
• Impact of Tuning: Tuning the encoder did not improve accuracy over the original encoder configuration (difference of −0.04 points).
• Leakage Analysis: The original protocol, which scored nodes used to train the encoder, yielded 79.01% accuracy. This is approximately 3.8 percentage points higher than the nested cross-validation estimate, confirming that the previous evaluation overstated performance.

Significance and Claims
The paper concludes that, on the Pima Indians Diabetes Dataset, tuning a GATv2 encoder under nested cross-validation does not make weighted-polling classification more accurate than raw-feature weighted polling or standard classifiers.

The author explicitly states that these findings are limited to this single dataset and do not prove that graph attention cannot help on other clinical data. The study highlights that for small, noisy tabular datasets with substantial class overlap, the k-NN graph constructed from raw features may not contain sufficient additional information to justify the complexity of a graph attention encoder. The primary significance of the work lies in demonstrating the necessity of rigorous validation (nested CV) to avoid overestimating the utility of graph-based methods in tabular domains, as the "optimism" from leakage can mask the lack of genuine improvement.