🤖 machine learning

Neurai-VN Benchmark: Standardized Machine Learning Models for Multimodal Digital Phenotyping in Mental Health Classification

This paper introduces the Neurai-VN Benchmark, a standardized framework utilizing a high-resolution multimodal dataset from 100 Vietnamese adults to evaluate reproducible machine learning models for classifying mental health conditions such as depression, anxiety, and clinical disorders.

Original authors: Quoc-Cuong Pham, Hoang-Thuy-Duong Vu, Thi-Thanh-Huong Ha, Huy-Hieu Pham

Published 2026-07-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Quoc-Cuong Pham, Hoang-Thuy-Duong Vu, Thi-Thanh-Huong Ha, Huy-Hieu Pham

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine your phone and your smartwatch are like a pair of super-observant, silent roommates. They don't just tell time; they watch how you move, how fast your heart beats, how long you sleep, and even how often you unlock your screen. In the world of mental health, scientists call this "digital phenotyping." It's a fancy way of saying we can turn these digital habits into a map of how a person is feeling inside. For years, researchers have tried to use these maps to spot early signs of depression or anxiety, hoping to catch problems before they get too big. But here's the catch: every research team has been using different maps, different tools, and different rules. It's like trying to compare a recipe for cake written in French, one written in Japanese, and one scribbled on a napkin. You can't really tell which one is the best because the ingredients and instructions are all mixed up. Plus, most of these maps were drawn using data from people in Western countries, leaving us wondering if the same rules apply to people in Vietnam or other parts of the world.

This paper, titled "NEURAI-VN BENCHMARK," steps in to fix that messy kitchen. The authors created a standardized "test kitchen" using a brand-new dataset called NEURAI-VN, collected from 100 adults in Vietnam over two weeks. They gathered a massive amount of data, including 17 different types of signals from wearables and phones, along with self-reported mood logs. Instead of just guessing which computer program works best, they set up a strict, fair race. They tested four different types of machine learning models (think of them as different kinds of detectives: one is a simple logic expert, one is a tree-leaf analyst, one is a powerful boost-boosting engine, and one is a deep neural network) against four specific mental health challenges. The goal wasn't to invent a new magic cure, but to build a reliable ruler so future scientists can measure their own progress fairly.

The results of this race were clear, though not perfect. The researchers found that when they combined data from different sources—like mixing heart rate data with sleep logs and daily mood surveys—the "detectives" got better at their jobs. Specifically, the system was quite good at telling the difference between healthy people and those with depression (scoring a 0.71 on a scale where 1 is perfect) and between healthy people and those with any clinical mental health condition (also 0.71). It was slightly less sure when distinguishing between healthy people and those with anxiety (0.69), and the hardest job of all was telling the difference between someone with depression and someone with anxiety, where the score dropped to 0.56.

The paper suggests that while adding more types of data generally helps the models perform better, the "best" mix of data depends entirely on the specific question being asked. For instance, the best setup for spotting depression wasn't the same as the best setup for spotting anxiety. The authors explicitly ruled out the idea that just throwing any data at a model guarantees a win; they showed that how you organize and combine the data matters just as much as the data itself. They also emphasized that their results are based on a specific group of 100 Vietnamese adults, so these numbers are a solid starting point, but they aren't a universal law that applies to everyone everywhere yet.

In the end, this paper doesn't claim to have solved mental health diagnosis. Instead, it provides a reproducible "scoreboard." By establishing a standard way to test these digital tools, the authors hope that future researchers won't have to reinvent the wheel every time. They've built a common language and a fair playing field, allowing the scientific community to finally compare apples to apples, rather than apples to oranges, in the quest to understand mental health through our digital footprints.

Technical Summary: NEURAI-VN Benchmark for Multimodal Digital Phenotyping in Mental Health

Problem Statement

Digital phenotyping (DP) using smartphones and wearables holds significant promise for monitoring mental health by capturing day-to-day behavioral variations that periodic clinical assessments often miss. However, the field faces three critical barriers to progress:

Heterogeneity: Datasets vary widely in population, sensing modalities, and sampling regimes, making models difficult to transfer between cohorts.
Reproducibility: Feature extraction and preprocessing pipelines are often task-specific and poorly documented.
Evaluation Misalignment: Many studies use record-wise cross-validation (allowing data from the same participant in both training and test sets), which yields overly optimistic performance estimates compared to subject-wise validation. Additionally, existing resources are predominantly drawn from Western, English-speaking, university-affiliated populations, leaving a gap in understanding how these associations hold in other cultural and socioeconomic contexts.

Methodology

This paper establishes a reproducible benchmark using the NEURAI-VN dataset, a high-resolution, multimodal dataset collected from 100 Vietnamese adults over a two-week period. The dataset includes 17 modalities (13 passive sensing, 4 active assessments) and clinical diagnostic annotations provided by psychiatrists.

Data Representation and Feature Engineering

The benchmark organizes data into six modality groups based on source and temporal resolution:

Wearable (Wm, Ws): Per-minute signals (activity, heart rate) and daily summaries (sleep, HRV, SpO2, respiration, skin temperature).
Self-report (Sd, Sw): Daily logs/mood and weekly clinical scales (PHQ-9, GAD-7).
Smartphone (Pe, Pc): Event-based app states and continuous sensor data (accelerometer, gyroscope, network, battery).

Feature Extraction:

Continuous/Per-minute signals: Extracted daily channel-wise statistical descriptors (mean, min, max, skew, std).
Event-based: Daily event counts.
Self-reports: Last daily response retained; scores normalized to [0, 1].
Preprocessing: Features underwent mean imputation, min-max scaling, and univariate feature selection (ANOVA F-statistics) to retain the top-K features.

Benchmark Design

The study defines 15 benchmark configurations by exhaustively combining four feature groups: Wearable-minute ( $W_m$ ), Wearable-daily ( $W_s$ ), Self-report-daily ( $S_d$ ), and Smartphone ( $P$ ). Note that weekly self-reports ( $S_w$ ) were excluded to align temporal resolutions.

Classification Tasks:
Four clinically relevant binary classification tasks were defined:

B1: Healthy Control (HC) vs. Depression (Dep)
B2: HC vs. Anxiety (Anx)
B3: Depression (Dep) vs. Anxiety (Anx)
B4: HC vs. Clinical (Dep + Anx + Other Psychiatric Conditions)

Experimental Protocol:

Models: Four baseline models were evaluated: Logistic Regression (LR), Random Forest (RF), XGBoost (XGB), and Multi-Layer Perceptron (MLP).
Validation: Strict 5-fold subject-wise Group K-Fold cross-validation was used to prevent information leakage between training and test sets.
Metric: Subject-level Macro F1-score (mF1) was the primary evaluation metric to account for class imbalance.

Key Results

The benchmark results provide reproducible baselines for future research. Performance varied by task and feature configuration:

Top Performance:
- HC vs. Depression (B1): Achieved 0.71 ± 0.17 mF1 using the $W_m + W_s + S_d$ configuration with an MLP model.
- HC vs. Clinical (B4): Achieved 0.71 ± 0.06 mF1 using the $W_m + W_s + S_d$ configuration with an LR model.
- HC vs. Anxiety (B2): Achieved 0.69 ± 0.17 mF1 using the full multimodal configuration ( $W_m + W_s + S_d + P$ ) with an LR model.
- Depression vs. Anxiety (B3): Achieved 0.56 ± 0.22 mF1 using only the $W_m$ (Wearable-minute) configuration with an XGB model.
Multimodal Integration:
- Generally, configurations integrating multiple feature groups outperformed single-modality setups.
- The average mF1 increased progressively from single-modality (0.48) to all-group combinations (0.60).
- However, the optimal configuration was task-dependent; for instance, the most difficult task (Dep vs. Anx) performed best with a single modality ( $W_m$ ), while HC vs. Anxiety benefited most from full integration.

Significance and Contributions

The authors position this work as a foundational step toward standardized evaluation in digital mental health. The paper's primary contributions are:

A Reproducible Benchmark: It establishes a unified evaluation framework with standardized feature configurations and a strict subject-wise cross-validation protocol, addressing the lack of comparability in current literature.
Demographic Diversity: By utilizing a Vietnamese cohort, the study addresses the demographic gap in existing resources, testing whether behavior-symptom associations hold in non-Western, non-English-speaking settings.
Baseline Performance: The reported F1 scores serve as reference points for future methodological development, allowing researchers to quantitatively compare new algorithms against established linear, tree-based, and neural baselines.
Open Resources: The dataset and the code for preprocessing and evaluation are made available to facilitate further research, with the dataset accessible via Zenodo and the benchmark code via GitHub.

The paper concludes that while multimodal integration generally improves performance, the specific utility of different modalities varies significantly across diagnostic tasks, underscoring the need for task-specific feature engineering in future studies.