Predicting PANSS symptoms in schizophrenia spectrum disorders using speech only: an international, multi-centre, retrospective, computational study across multiple languages

This international, multi-centre study demonstrates that pretrained multilingual embeddings applied to speech recordings from 453 patients can accurately predict eight relapse-related PANSS symptoms in schizophrenia spectrum disorders with low error rates, offering a scalable, low-burden tool for objective symptom monitoring across diverse languages.

The Gist

Imagine you have a friend who is going through a tough time mentally. Usually, to understand how they are feeling, a doctor has to sit down with them, ask a long list of questions, and take notes. This is like a manual inspection of a house to find cracks in the foundation. It's necessary, but it's slow, expensive, and you can only do it once a year or so.

This paper is about building a smart, automated security camera that can watch your friend's voice and tell the doctor exactly how the "foundation" is holding up, without needing a human to sit there for hours.

Here is the breakdown of what the researchers did, using simple analogies:

1. The Big Problem: Too Many Languages, Too Few Data

For years, scientists have tried to build this "voice camera." But most of them only spoke English. Imagine trying to teach a dog to fetch a ball, but you only ever throw the ball in English. If you switch to Spanish or Turkish, the dog might not understand the command.

Previous studies were also like small fishing nets. They only caught a few fish (patients), so the net wasn't very strong. If you tried to use that net in a different ocean (a different country or language), it would likely tear.

2. The Solution: A Global "Voice Gym"

The researchers in this paper decided to build a massive, global gym for voices.

• The Players: They gathered 453 patients from 10 different countries (including the US, China, Turkey, Germany, Chile, and more).
• The Workout: They didn't just ask them to say "hello." They had them do various tasks: telling stories, describing pictures, talking about their dreams, and even reading a story about a crow.
• The Result: They created a library of 6,664 voice clips in 10 different languages. This is like training a dog not just in English, but in a dozen languages, so it understands the feeling behind the words, not just the words themselves.

3. The Magic Tool: Listening to the "Music," Not the Lyrics

Most previous attempts tried to transcribe the speech first (turning voice into text) and then analyze the text.

• The Analogy: This is like trying to judge a song by reading the sheet music. If the music is noisy or the singer has a unique accent, the sheet music might look wrong.
• This Study's Approach: They skipped the text entirely. They used a special AI (called mHuBERT) that listens to the sound waves directly. It's like listening to the tone, rhythm, and emotion of a song without reading the lyrics.
  • It can hear if a voice is flat (like a monotone robot).
  • It can hear if a voice is jumpy or disorganized (like a song with a broken beat).
  • It can hear if a voice is full of strange, unusual thoughts (like a song with weird, jarring notes).

4. The Goal: Predicting the "Relapse"

In schizophrenia, symptoms can flare up (a "relapse") before the patient even realizes it. The researchers wanted to see if their AI could predict the severity of specific symptoms (like hallucinations or lack of motivation) just by listening to these voice clips.

They treated the symptoms like a thermostat. The goal was to see if the AI could tell the doctor: "Hey, the temperature in the 'Delusion' room is rising," or "The 'Motivation' heater is turning off."

5. The Results: The AI Got It Right!

The results were surprisingly good.

• Accuracy: The AI could predict the severity of symptoms with an error margin of less than 1.5 points on a scale of 1 to 7.
  • Analogy: If you are guessing the weight of a person, and you are only off by about 3 pounds, that's pretty good!
• The Secret Weapon: The AI that listened to the raw sound (the "music") worked better than the ones that tried to analyze the "lyrics" (text) or just the basic pitch.
• Fairness: The AI worked well across different ages, genders, and education levels. It didn't seem to care if the person was from Turkey or Chile; it just listened to the voice.
• The One Weakness: The AI struggled a bit more when the symptoms were extremely severe.
  • Analogy: It's like a weather forecast. It's great at predicting a sunny day or a light rain. But when a massive hurricane is forming, it's harder to predict the exact wind speed. However, for catching the early signs of a storm (which is what doctors need), the AI is excellent.

6. Why This Matters

Imagine a future where a patient can talk to their phone for 5 minutes every day. The AI listens, and if it hears the "music" of their voice changing in a way that suggests a relapse is coming, it sends an alert to their doctor.

• No Transcripts Needed: It doesn't need a human to type out what was said.
• Works Everywhere: It understands many languages, not just English.
• Early Warning: It can catch the problem before it becomes a crisis.

In short: This paper proves that your voice is a powerful window into your mind. By teaching computers to listen to the music of our speech across many languages, we can build tools that help doctors catch mental health crises earlier, cheaper, and more fairly for everyone, everywhere.

Technical Summary

1. Problem Statement

Schizophrenia spectrum disorders (SSD) are characterized by fluctuating symptoms, where relapse is a critical clinical event. Current monitoring relies heavily on clinician-rated scales like the Positive and Negative Syndrome Scale (PANSS), which are time-consuming, labor-intensive, and prone to inter-rater variability. While spontaneous speech has shown promise as a biomarker for mental state, existing computational approaches face three major limitations:

• Data Scarcity and Monolingual Bias: Most studies use small datasets (<50 patients) and are predominantly English-centric, limiting generalizability to low-resource languages and diverse populations.
• Transcript Dependency: Many pipelines rely on converting speech to text (via manual transcription or ASR) before feature extraction. This introduces transcription costs, errors (especially in noisy clinical settings), and biases against underrepresented languages.
• Lack of Granularity: Previous work often dichotomized symptoms (high vs. low) or predicted total scores, rather than modeling specific, relapse-relevant PANSS items with regression.

The study aims to address these gaps by building a large-scale, multilingual, speech-only (transcript-free) model to predict specific PANSS symptom scores with high accuracy.

2. Methodology

Dataset and Participants

• Scale: The study aggregated data from 453 patients with SSD across 10 international centers (Czech Republic, USA, Spain, Chile, France, Switzerland, Netherlands, Turkey, Germany, China).
• Languages: The corpus covers 10 language variants (e.g., Czech, English, Spanish, Chilean Spanish, French, Swiss German, Dutch, Turkish, German, Mandarin Chinese).
• Data Volume: Recordings were processed into 6,664 speech segments (clipped to ≤60 seconds).
• Tasks: Speech was elicited via diverse tasks including free speech, picture description, dream reports, reading, and recall.
• Labels: The target variables were eight specific PANSS items selected as relapse markers:
  • Positive: P1 (Delusions), P2 (Conceptual Disorganization), P3 (Hallucinatory Behavior).
  • Negative: N1 (Blunted Affect), N4 (Passive/Apathetic Social Withdrawal), N6 (Lack of Spontaneity).
  • General: G5 (Mannerisms/Posturing), G9 (Unusual Thought Content).

Feature Extraction (Speech-Only)

Crucially, the study avoided text transcripts. Features were extracted directly from the audio signal:

1. Acoustic-Prosodic Profile: Extracted using OpenSmile (eGeMAPSv02 feature set) for 88 features (energy, pitch, spectral balance) and Prosogram for 31 prosodic variables (rhythm, intonation).
2. Pretrained Multilingual Embeddings: Audio was encoded using mHuBERT-147 (a self-supervised, multilingual speech representation model trained on 90,000 hours of data). These embeddings capture joint patterns of articulation, prosody, and segmental context without explicit linguistic knowledge.
3. Concatenation: A third feature set combined both acoustic-prosodic and embedding features.

Computational Modeling

• Split Strategy: Data was split by participant (not segment) into Training (80%), Validation (10%), and Test (10%) sets, stratified by language and a binary severity label to prevent data leakage.
• Algorithms: The study benchmarked 16 algorithms:
  • Traditional ML: 14 regressors including Ridge, Lasso, Elastic Net, SVR, Random Forest, XGBoost, and Gaussian Process Regression.
  • Deep Learning: Two Multi-Layer Perceptrons (MLPs) with 2 and 3 hidden layers.
• Training Protocol: Features were standardized and reduced via PCA. Models were tuned via grid search with early stopping and dropout regularization. Performance was evaluated using Root Mean Squared Error (RMSE) at both the segment level and the participant level (median aggregation).

3. Key Results

Prediction Performance

The best-performing models achieved RMSE ≤ 1.5 for all eight PANSS items, which is considered clinically meaningful given the 1–7 scale of PANSS.

• Best Overall Feature Set: Pretrained multilingual embeddings (mHuBERT) consistently outperformed acoustic-prosodic features and their concatenation for most items.
• Specific Item Performance (Segment Level RMSE):
  • N6 (Lack of Spontaneity): 0.860 (Best)
  • G5 (Mannerisms): 0.850
  • N1 (Blunted Affect): 1.029
  • P2 (Conceptual Disorganization): 1.318
  • P1 (Delusions): 1.494
  • P3 (Hallucinatory Behavior): 1.407
  • N4 (Social Withdrawal): 1.452
  • G9 (Unusual Thought): 1.213
• Model Selection: Support Vector Regressors (SVR) and MLPs were the top performers depending on the specific item.

Generalizability and Bias

• Cross-Language: Models performed comparably across high-resource (English, Chinese) and low-resource (Czech, Turkish) languages, demonstrating successful transfer learning.
• Demographic Bias: No significant bias was found regarding age, education, or sex, with one exception: the model for N4 (Social Withdrawal) showed reduced accuracy for male participants.
• Symptom Severity: Prediction error increased with higher symptom severity, suggesting models are less accurate for patients with very severe symptoms (though this range is often clinically obvious).
• Out-of-Sample Validation: A pilot test on 5 new patients (including Catalan speakers, a language not in the training set) yielded surprisingly low errors for most items, confirming external validity.

4. Key Contributions

1. Largest Multilingual SSD Corpus: This is the largest speech-based study on schizophrenia to date, encompassing 453 patients and 10 languages.
2. Transcript-Free Approach: Demonstrated that high-accuracy symptom prediction is possible using audio-only inputs, eliminating the need for costly transcription or error-prone ASR.
3. Granular Symptom Modeling: Moved beyond classification or total scores to predict specific, relapse-relevant PANSS items with regression, achieving sub-1.5 point errors.
4. Feature Superiority: Established that self-supervised multilingual speech embeddings (mHuBERT) are superior to hand-crafted acoustic features for this task, capturing linguistic and semantic nuances directly from audio.
5. Open Science: The authors have released all scripts and trained models (excluding raw data due to privacy) to facilitate further research.

5. Significance and Implications

• Clinical Utility: The study validates speech as a scalable, low-burden, and objective tool for monitoring symptom trajectories and predicting relapse in real-world clinical settings.
• Equity: By including low-resource languages and demonstrating cross-lingual transfer, the approach reduces the "English-centric" bias in AI psychiatry, making digital biomarkers accessible globally.
• Future Directions: The findings support the integration of speech-based monitoring into routine psychiatric care. Future work should focus on prospective studies to validate relapse prediction in real-time and improving interpretability of the "black box" models to understand the specific acoustic-linguistic markers driving the predictions.

Limitations: The study is retrospective, and the target PANSS labels themselves contain inter-rater noise. Additionally, performance degrades at the highest severity levels, and the external validation set was small (n=5).