🤖 machine learning

ASD-Bench: A Four-Axis Comprehensive Benchmark of AI Models for Autism Spectrum Disorder

This paper introduces ASD-Bench, a comprehensive four-axis benchmark evaluating diverse machine learning and foundation models on a curated dataset of 4,068 AQ-10 records across three age cohorts to reveal distinct age-specific diagnostic patterns, the limitations of single-metric evaluations, and the need for cohort-tailored deployment strategies in automated Autism Spectrum Disorder screening.

Original authors: Shubhankit Singh, Hassan Shaikh, Kuldeep Raghuwanshi, Keshav Bulia

Published 2026-05-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shubhankit Singh, Hassan Shaikh, Kuldeep Raghuwanshi, Keshav Bulia

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a "metal detector" to find a specific type of hidden treasure (Autism Spectrum Disorder, or ASD) in a crowd of people. For a long time, researchers have built these detectors, but they've mostly tested them in three ways that don't tell the whole story:

They only tested one type of detector at a time.
They only checked if it found the treasure, not if it was sure it found it, or if it would break if the ground shook.
They mostly tested it on adults, ignoring children and teenagers who might hide the treasure differently.

This paper, ASD-Bench, is like a massive, organized "Tough Guy Tournament" for 17 different AI detectors. The researchers wanted to see which one is the best at spotting ASD using a simple 10-question checklist (called AQ-10), but they did it in a much smarter way.

Here is the breakdown of their tournament, explained simply:

1. The Three Teams (Age Groups)

The researchers didn't just test everyone together. They split the crowd into three distinct teams, realizing that a child, a teenager, and an adult might "hide" their traits differently:

The Kids (1–11 years): The easiest group to spot.
The Teens (12–16 years): The hardest group. They are like "chameleons" who have learned to blend in, making the treasure harder to find.
The Adults (17–64 years): Surprisingly easy to spot, but only because the dataset was small and the patterns were very clear.

2. The Four Rules of the Game (The "Four-Axis" Benchmark)

Instead of just counting how many treasures the detectors found (Accuracy), the judges used four different scorecards:

Score 1: The Hit Rate (Performance): Did it find the treasure? (Standard accuracy).
Score 2: The Confidence Check (Calibration): If the detector says "90% sure," is it actually 90% sure? Some detectors were great at finding treasure but terrible at knowing how sure they were (like a person who guesses right but thinks they are a genius).
Score 3: The "Why" Factor (Interpretability): Can the detector explain which question on the checklist made it decide? (e.g., "I said 'Yes' because the kid hates social chit-chat").
Score 4: The Shake Test (Robustness): If you accidentally scribble over a few answers or the data gets a little noisy, does the detector still work, or does it panic and give a wrong answer?

3. The New Scorecard: "The HAP"

The researchers invented a new scoring system called HAP (Heuristic Aggregate Penalty).

The Analogy: Imagine a doctor's waiting room.
- False Positive: The detector says a healthy kid has ASD. The kid gets a second check-up. It's annoying, but not a disaster.
- False Negative: The detector says a kid with ASD is healthy. The kid misses out on help. This is a tragedy.
The HAP Rule: The HAP score treats missing a real case (False Negative) as 5 times worse than a false alarm. It also punishes detectors that are "moody" (inconsistent). If a detector is great on Monday but terrible on Tuesday, HAP gives it a bad score.

4. The Tournament Results

Here is what happened when they ran the 17 different AI models against the three age groups:

The Adults: It was a blowout. 10 out of 17 models got a perfect score. However, the researchers warn this might be because the adult group was small and the data was too easy, like a test with all the answers on the back of the page.
The Kids: The models did very well. The most important clue for kids was A9: "Do I enjoy social chit-chat?" Kids with ASD usually hate it, and the AI noticed this immediately.
The Teens: This was the boss battle. The models struggled the most here. The "social chit-chat" clue stopped working. Instead, the models started looking at A5: "Do I notice patterns in things all the time?" This suggests that as kids grow up, they learn to hide their social struggles (masking), but their obsession with patterns remains visible.
The "Confidence" Trap: One model, AdaBoost, got a perfect score for finding adults, but it was wildly overconfident and wrong about how sure it was. The paper says: "Don't trust a model that is confident but wrong."

5. The Winners (Who should you use?)

The paper doesn't say "Use this one forever." Instead, it says "Use the right tool for the right job":

For Adults (in a perfect, quiet room): A complex model called TabTransformer works best.
For Adults (in a noisy, messy room): A simpler model called MLP is more stable.
For Kids: A model called XGBoost is the champion.
For Teens: A "Foundation Model" called TabPFN is the best at finding the treasure, even though it's a bit fragile. If you need it to be tough against noise, use TabNet instead.

6. The Big Warning (The "Fine Print")

The authors are very careful to say: "This is a proof-of-concept, not a medical diagnosis tool."

The Data: They used a dataset where people filled out a questionnaire on an app. It wasn't a doctor confirming the diagnosis.
The Limit: Because the data came from one specific app and one specific time, we don't know if these models would work in a real hospital in a different country.
The Takeaway: This paper is a map showing us how to build better detectors and what to measure, but the detectors themselves aren't ready to replace a doctor yet.

In a nutshell: The researchers built a rigorous gym for AI models to test their strength, balance, and honesty. They found that children, teens, and adults are different puzzles, and the best AI for one group might be the worst for another. They also proved that being "accurate" isn't enough; a medical AI must also be honest about its confidence and tough enough to handle messy real-world data.

Technical Summary: ASD-Bench

Problem Statement

Automated screening tools for Autism Spectrum Disorder (ASD) currently suffer from three critical limitations:

Single-Axis Evaluation: Prior studies often rely on isolated metrics (e.g., accuracy or F1) and single-architecture comparisons, neglecting calibration, interpretability, and robustness.
Age-Agnostic Approaches: Most research focuses exclusively on either adult or child cohorts, obscuring age-specific diagnostic patterns. Adolescents, in particular, remain under-studied despite exhibiting distinct feature hierarchies and higher classification difficulty.
Clinical Misalignment: Existing benchmarks often fail to account for the asymmetric costs of clinical errors (where False Negatives are significantly more costly than False Positives) and lack stability metrics required for multi-site deployment.

Methodology

The authors introduce ASD-Bench, a systematic tabular benchmark evaluating 17 model configurations across four distinct axes.

Dataset (v3)

The benchmark utilizes a curated dataset of 4,068 records derived from two sources: the UCI Machine Learning Repository (v1) and a supplementary dataset from the University of Arkansas (v2). After deduplication and quality control, the data is stratified into three age cohorts:

Children (1–11 yr): 2,514 records.
Adolescents (12–16 yr): 818 records.
Adults (17–64 yr): 736 records.
The dataset consists of binary features derived from the AQ-10 (Autism-Spectrum Quotient 10-item) questionnaire and demographic variables.

Models Evaluated

The study evaluates four categories of models, each tested in baseline and hyperparameter-tuned configurations (via GridSearchCV):

Classical Ensembles: Logistic Regression, Random Forest, AdaBoost, XGBoost.
Neural Networks: Multi-Layer Perceptron (MLP).
Deep Tabular Architectures: TabNet, TabTransformer, FT-Transformer (all utilizing Monte Carlo Dropout for uncertainty estimation).
Foundation Model: TabPFN v2 (a prior-data fitted network performing in-context learning without gradient-based fine-tuning).

Four-Axis Evaluation Framework

Predictive Performance: Standard metrics (Accuracy, Precision, Recall, F1, AUC-ROC).
Calibration/Uncertainty: Expected Calibration Error (ECE), Brier Score, and entropy.
Interpretability: Feature importance via SHAP (Tree/Deep/Gradient Explainers), permutation importance, and built-in masks.
Adversarial Robustness: Performance under Feature Flips (10–30%), Gaussian Noise ( $\sigma \in \{0.1, 0.2, 0.3\}$ ), and Feature Removal (top-k).

The HAP Metric

The authors introduce the Heuristic Aggregate Penalty (HAP), a cost-sensitive metric designed for clinical deployment. It incorporates:

Asymmetric Penalties: A weight ratio of $w_{FN}:w_{FP} = 10:2$ (5:1), reflecting the higher clinical cost of missing a diagnosis.
Stability Penalty: An explicit variance term ( $\lambda \cdot \text{Var}(C_k)$ ) derived from a signal-to-noise criterion, penalizing models with inconsistent performance across cross-validation folds.

Key Results

Predictive Performance

Adults: The task is highly separable; 10 of 17 models achieved perfect F1 and AUC scores.
Children: F1 scores ranged from 0.864 to 0.915. TabTransformer (Tuned) achieved the highest F1 (0.915), while TabPFN v2 achieved the highest AUC (0.963).
Adolescents: This cohort presented the most difficult task, with an F1 ceiling of 0.837 (vs. 0.915 for children). TabPFN v2 led this cohort with an F1 of 0.837 and AUC of 0.900.

Calibration and Robustness

Dissociation of Metrics: High accuracy does not guarantee calibration. For instance, AdaBoost achieved F1 = 1.000 on adults but suffered severe miscalibration (ECE = 0.302).
Robustness vs. Accuracy Trade-off: Transformer-based models (TabTransformer, FT-Transformer) achieved perfect F1 on adults but showed significant degradation (~24%) under Gaussian noise. Conversely, tree-based models (Random Forest, XGBoost) demonstrated higher robustness.
Adolescent Robustness: Paradoxically, models were more robust on the adolescent cohort (e.g., TabNet Baseline R = 0.980) despite lower absolute accuracy, suggesting smoother decision boundaries under perturbation.

Feature Hierarchies

Feature importance shifted significantly across cohorts, supporting the need for age-specific models:

Children: Dominated by A9 (Social motivation/chit-chat), which ranked top in 11 of 17 models.
Adolescents: A complete hierarchy shift occurred; A5 (Pattern recognition) became the dominant feature, while A9 dropped to 8th place.
Adults: Exhibited a flat importance profile with no single dominant feature, consistent with developmental social masking.

Significance and Claims

The paper positions ASD-Bench as a proof-of-concept framework rather than a clinically validated diagnostic tool. Its primary contributions are:

Systematic Benchmarking: It is the first study to unify classical, deep tabular, and foundation models for ASD screening across three distinct age cohorts.
Clinical Metric Innovation: The introduction of HAP provides a principled method for evaluating models based on clinical cost asymmetry and deployment stability, moving beyond simple accuracy.
Cohort-Specific Insights: The findings demonstrate that a "one-size-fits-all" model is inadvisable. The distinct feature hierarchies (social vs. cognitive-perceptual) and varying difficulty levels (adolescents being harder) necessitate tailored deployment strategies.
Deployment Recommendations: The authors provide specific model recommendations based on the four-axis scorecard (e.g., TabTransformer for controlled adult settings, TabPFN for adolescents prioritizing accuracy, and TabNet for robustness).

Limitations Acknowledged by the Authors:

The dataset relies on self-reported AQ-10 scores, not clinician-confirmed diagnoses.
The adult cohort size (n=148 after deduplication) is small, potentially inflating performance metrics.
No external or multi-site validation was performed; results are internal benchmark scores.
The comparison between tuned models and TabPFN is inherently asymmetric due to the lack of fine-tuning for the foundation model.

The authors conclude that while the results are promising for algorithmic development, they represent internal benchmark performance and should not be interpreted as evidence of real-world clinical diagnostic readiness without prospective, multi-site validation.