Protecting Bystander Privacy via Selective Hearing in Audio LLMs

1. Problem Statement

Audio Large Language Models (LLMs) are increasingly deployed in real-world environments (e.g., voice assistants, wearables). While these models excel at understanding the main speaker, they passively capture speech from unintended bystanders in the background.

• The Gap: Existing benchmarks and privacy defenses focus primarily on active users. They fail to address the privacy risks posed to bystanders whose speech is incidentally recorded without consent.
• The Risk: Audio LLMs can inadvertently memorize and reveal sensitive bystander information (identity, health, emotions) or be exploited via privacy attacks to extract this data.
• The Challenge: Can audio LLMs be designed to exhibit "Selective Hearing"—attending strictly to the intended main speaker while actively refusing to process or reveal information about incidental bystander speech?

2. Methodology

The authors propose a comprehensive framework consisting of a new benchmark, a novel evaluation metric, and a training pipeline.

A. SH-Bench (Selective Hearing Benchmark)

SH-Bench is the first benchmark designed to evaluate and improve bystander privacy in audio LLMs.

• Dataset Composition:
  • Total: 3,968 multi-speaker audio mixtures (~157.5 hours) involving 285 unique speakers.
  • Split: Divided into a Real-Scenario Partition (100 recordings from 5 everyday settings: coffee shops, gyms, shared living areas, public transit, waiting rooms) and a Synthetic-Scenario Partition (100 mixtures derived from the AMI Meeting Corpus).
  • Annotations: Each audio is paired with 10 multiple-choice questions (MCQs): 5 regarding the main speaker and 5 regarding the bystander.
  • Key Feature: Includes an "I Don't Know" (IDK) option. In selective mode, the correct answer for bystander questions is IDK.
• Construction Pipeline:
  • Real Data: Scripts generated by GPT-4o for main speakers (structured) and bystanders (informal/sensitive). Recorded by pairs of participants.
  • Synthetic Data: Main speaker segments from AMI mixed with bystander segments (attenuated by -10dB) with added reverberation.

B. Evaluation Framework & Metric: Selective Efficacy (SE)

The authors introduce Selective Efficacy (SE) as a unified metric to balance comprehension and privacy.

• Operating Modes:
  1. General Mode: The model answers all questions (main and bystander) correctly.
  2. Selective Mode: The model is instructed to listen only to the main speaker. It must answer main speaker questions correctly but select "IDK" for bystander questions.
• The Metric (SE): Calculated as the harmonic mean of four accuracy values:
  1. Main Speaker Accuracy (General Mode)
  2. Main Speaker Accuracy (Selective Mode)
  3. Bystander Accuracy (General Mode)
  4. Bystander Accuracy (Selective Mode)
  • Interpretation: A high SE requires high comprehension of the main speaker and successful refusal of bystander queries. A model that answers everything correctly in selective mode has low SE (privacy failure); a model that refuses everything has low SE (comprehension failure).

C. Bystander Privacy Fine-Tuning (BPFT)

To mitigate privacy leakage, the authors propose BPFT, a supervised fine-tuning pipeline.

• Training Data: A large dataset (3,768 mixtures, 75k questions) similar to SH-Bench but designed for training.
• Instruction Strategy: Each training sample includes paired instructions:
  1. "Answer the question" (General).
  2. "Refuse if the question is about the bystander" (Selective).
• Implementation: Applied to open-source models (Qwen-2.5-Omni, Step-Audio-2-mini) using Low-Rank Adaptation (LoRA) on the LLM backbone while freezing other components.

3. Key Contributions

1. SH-Bench: The first benchmark specifically for evaluating selective hearing and bystander privacy in audio LLMs, covering both real-world and synthetic scenarios.
2. Selective Efficacy (SE): A novel metric that quantifies the trade-off between multi-speaker comprehension and privacy protection, preventing models from gaming the system by either answering everything or refusing everything.
3. BPFT Pipeline: A training methodology that teaches models to distinguish between main and bystander speakers and refuse privacy-invasive queries without significantly degrading main-speaker comprehension.

4. Results

The paper evaluated state-of-the-art open-source and proprietary models (including Gemini 2.5 Pro, GPT-4o, Qwen, Llama, Step-Audio).

• Baseline Performance: Existing models show substantial bystander privacy leakage.
  • In Selective Mode, models often answer bystander questions correctly (e.g., Gemini 2.5 Pro achieved 59.2% accuracy on bystander questions in selective mode, where the target is IDK).
  • Strong audio understanding does not automatically translate to selective privacy protection.
  • Gemini 2.5 Pro was the best performer among unmodified models with an SE of 75.8%.
• BPFT Performance:
  • Models fine-tuned with BPFT showed dramatic improvements.
  • Step-Audio-2-mini + BPFT achieved an SE of 91.7% (an absolute 15.9% gain over the best unmodified model).
  • Bystander Accuracy in Selective Mode: Improved from ~30-50% to 96.1% (effectively choosing "IDK" correctly).
  • Main Speaker Accuracy: Remained high (94.3%), demonstrating that privacy protection did not significantly degrade core comprehension.
• Ablation Studies:
  • Speaker Description: Providing a description of the main speaker significantly helps models distinguish targets, but BPFT reduces reliance on this description by learning to identify bystanders directly from audio cues.
  • Refusal Mechanism: Without the "IDK" option, models struggle to refuse. BPFT-trained models learned to exhibit high entropy (uncertainty) or explicit refusal even without the explicit IDK option, and achieved >90% refusal rates on open-ended bystander questions.

5. Significance

• Systematic Framework: This work provides the first systematic approach to measuring and solving the "bystander privacy" problem in audio AI, a critical gap in current AI safety research.
• Practical Deployment: The findings suggest that current audio assistants are privacy risks in public/shared spaces. The proposed BPFT offers a viable, scalable solution to deploy these models safely.
• Balancing Act: The study proves that it is possible to enhance privacy protection (refusal) without sacrificing the model's ability to understand the primary user, challenging the notion that privacy and utility are mutually exclusive in this context.
• Future Direction: It sets a precedent for future research into multi-speaker privacy, moving beyond simple speaker diarization to semantic privacy protection.