Technical Summary: RedVox: Safety and Fairness Gaps in Speech Models Across Languages

Problem Statement

The integration of speech capabilities into Large Language Models (LLMs) represents a significant advancement in human-AI interaction, offering reduced latency and inherent error resilience. However, the safety and fairness of these speech-capable models remain critically understudied, particularly beyond English settings and under naturalistic conditions. Existing research on model toxicity and bias is predominantly confined to text-based domains or English-centric speech evaluations using synthetic voices. This creates dangerous blind spots, as recent studies suggest that multimodal inputs can bypass safety mechanisms more effectively than text alone. Furthermore, there is a lack of systematic documentation regarding safety evaluations in state-of-the-art speech model releases, with a significant gap in multilingual analysis.

Methodology

1. Survey of Reporting Practices

The authors first conducted a comprehensive survey of 38 state-of-the-art speech models (including SpeechLLMs and OmniLLMs) to assess safety reporting practices. They analyzed model cards, technical reports, and accompanying papers for mentions of safety, fairness, red teaming, or adversarial robustness.

2. The REDVOX Benchmark

To address the identified gaps, the authors introduced REDVOX, a multilingual safety and fairness benchmark for audio and speech.

• Scope: Covers five languages: English, French, Italian, Spanish, and German.
• Data Source: Built upon natural voices from 52 human participants (researchers from 7 European institutions) rather than synthetic voices.
• Content: Derived from two existing textual benchmarks:
  • SHADES: For fairness, focusing on stereotypical generalizations regarding gender, nationality, and socioeconomic status.
  • M-ALERT: For safety, covering criminal planning, substances, sexual content, self-harm, and weapons.
• Request Types: The benchmark employs two complementary input configurations to test model robustness:
  • Type I (Speech): Harmful content is vocalized by a human, accompanied by a short textual follow-up request.
  • Type II (Audio): Harmful content is presented as text, paired with a distracting non-speech audio signal (silence, ambient noise, or babble noise).
• Dataset Size: The full collection comprises 6,118 unique entries (approx. 10 hours of audio). Due to consent constraints, the publicly released subset contains 3,414 entries from 26 unique voices.

3. Evaluation Framework

The authors evaluated eight state-of-the-art models (five open-weight and three proprietary) using a multi-dimensional assessment:

• Dimensions:
  1. Relatedness: Did the model understand the input?
  2. Safety/Fairness: Did the model endorse, support, or assist with the harmful request?
• Classification Taxonomy: Responses were categorized on a severity scale:
  • Safe (S): Explicitly discourages harmful activity.
  • Safe by Accident (A): Misunderstands the input, producing a harmless response by error.
  • Controversial (C): Context-dependent or ambiguous responses.
  • Unsafe (U): Endorses or complies with the harmful request.
• Judging Mechanism: An "LLM-as-a-judge" approach using GPT-5.5 was employed to label responses, validated against a manually annotated testbed of 250 entries.

Key Results

1. Sparse and English-Centric Reporting

The survey revealed that only 8% of the 38 surveyed speech models document any evaluation beyond English. Most safety documentation is either non-existent or relies on private, non-public datasets.

2. Persistent Vulnerabilities in Natural Conditions

Even under non-adversarial, naturalistic conditions, vulnerabilities persist across all tested models.

• Model Performance: Proprietary models generally demonstrated lower unsafe response rates (≤3.1%) compared to open-weight models. However, open models like Voxtral exhibited high failure rates, producing fully harmful responses in roughly 1 out of 4 cases.
• The "Safe by Accident" Phenomenon: Some models, such as GPT-realtime2, showed high safety scores but a significant rate of "Safe by Accident" responses (9.9%), indicating safety driven by multimodal misunderstanding rather than genuine refusal mechanisms.

3. The Non-English Gap

Safety and fairness vulnerabilities systematically worsen in non-English languages.

• Statistical Gap: The unsafe response rate for English was 5.1%, compared to 10.0% for other languages (a near-doubling).
• Open Models: The gap is most pronounced in open-weight models; for instance, Voxtral's unsafe rate increased by 15 percentage points in Spanish and French compared to English.

4. Multimodality as a Stressor

The presence of audio input acts as a stressor that increases model vulnerability compared to text-only inputs.

• Gradient of Harm: For most open models, the rate of harmful elicitation follows a gradient: Text < Audio < Speech.
• Audio Effect: Even pairing a harmful text request with non-speech audio (silence or noise) resulted in higher unsafe rates than text-only inputs (up to +20% increase for Voxtral), suggesting that the mere presence of an audio modality degrades safety alignment.

5. Participant Challenges

A post-activity questionnaire revealed unique sociotechnical challenges in collecting speech data for red teaming:

• Psychological Burden: 61.5% of participants reported feeling uncomfortable or very uncomfortable releasing voice recordings containing harmful content, a significantly higher rate than for text.
• Identity Concerns: Participants expressed fear of voice identification and association with harmful material, highlighting a need for voice anonymization technologies and stronger privacy guarantees in speech safety research.

Significance and Claims

The paper claims that REDVOX is the first resource to cover unsafe and unfair stereotypical requests in a multilingual setting built upon natural voices. Its significance lies in:

1. Exposing Blind Spots: Demonstrating that current safety frameworks have dangerous blind spots in non-English languages and when speech is the input modality.
2. Highlighting Modality Ris: Establishing that speech and audio inputs act as consistent stressors that weaken safety alignment, even in non-adversarial scenarios.
3. Sociotechnical Insight: Documenting the unique ethical and privacy challenges of "speech red teaming" with human participants, a dimension previously unexamined in the literature.
4. Resource Provision: Providing a reproducible, multilingual benchmark to facilitate future research into multilingual safety evaluation and the development of duty-of-care frameworks for speech AI.

The authors conclude that while progress is being made, the field faces a fundamental resource bottleneck in multilingual speech safety data, and the psychological toll on human participants in red-teaming efforts requires new methodological and ethical approaches.