Technical Summary: PitchBench

Problem Statement

Audio-language models (ALMs) are increasingly deployed in real-world applications requiring music understanding, such as tutoring, transcription, and recommendation systems. While frontier models have demonstrated proficiency in high-level semantic tasks (e.g., genre classification, instrument identification), their ability to perform reliable, fine-grained musical perception remains unverified. Specifically, pitch hearing—the ability to resolve the fundamental frequency of a sound—is a prerequisite for melody recognition, harmony analysis, and transcription.

Existing benchmarks fail to adequately assess this capability for three primary reasons:

1. Indirect Measurement: Prior work evaluates pitch hearing only through higher-level tasks (e.g., chord recognition or melody extraction), conflating the ability to perceive pitch with the ability to perform complex reasoning.
2. Methodological Flaws: Many evaluations rely on multiple-choice formats, which allow models to succeed via informed guessing or by exploiting textual distractors rather than genuine auditory perception.
3. Lack of Granularity: Current benchmarks do not isolate specific perceptual variables (e.g., note duration, acoustic noise, sound source, or notation format), leaving a gap in understanding why models fail.

Consequently, it remains unclear whether ALMs possess stable, perceptually grounded pitch perception or if they merely hallucinate musical structures based on training data priors.

Methodology: PitchBench

To address these gaps, the authors introduce PitchBench, a systematic evaluation suite comprising 28 experiments designed to decompose pitch perception into controlled, atomic, and contextual components.

Experimental Design

PitchBench evaluates ALMs across three hierarchical levels of difficulty:

• Level 1: Atomic Pitch Perception: Models identify single pitches in isolation. This isolates core frequency identification across variations in loudness and note duration.
• Level 2: Contextual Pitch Perception: Models identify pitch within temporal, structural, or acoustic contexts. This includes:
  • Temporal Localization: Identifying pitch at specific timestamps or within silence.
  • Sequential Structure: Determining intervals, contour, ranking, and counting in note sequences.
  • Chordal Structure: Identifying pitches and relationships within dyads and chords.
  • Acoustic Variations: Testing robustness against effects (reverb, distortion), background noise, time-stretching, vibrato, and detuning.
• Level 3: Melodic Pitch Perception: Models track a specific melodic line within polyphonic textures, specifically testing voice separation in four-part Bach chorales.

Stimulus Generation and Data

• Synthetic Generation: All stimuli are generated deterministically using pitchbench.generation, a Python package released with the benchmark. This ensures ground truth is known by construction (no manual annotation required) and allows for arbitrary scaling across parameters.
• Sound Sources: The dataset spans 19 timbres, including four analytic waveforms (sine, sawtooth, square, triangle) and 15 General MIDI instruments (piano, violin, flute, etc.).
• Acoustic Conditions: Stimuli are processed with Pedalboard to apply effects, saturation, and time-stretching. Background noise includes additive synthetic textures and real-world recordings.
• Response Formats: To avoid bias toward specific training conventions, models are evaluated on four equivalent pitch representations:
  • MIDI (0–127)
  • Scientific Pitch Notation (SPN, e.g., F#4)
  • Fixed-do Solfège (e.g., fa#)
  • Frequency in Hertz (Hz)

The benchmark comprises 17,667 question-audio-answer tuples.

Key Contributions

1. A Hierarchical Evaluation Suite: PitchBench provides the first structured decomposition of pitch hearing into atomic, contextual, and melodic levels, allowing researchers to diagnose specific failure modes rather than relying on aggregate scores.
2. Systematic Empirical Analysis: The authors evaluated six frontier ALMs (Gemini 3.1 Pro, Gemini 3 Flash, GPT-4o audio, Qwen-3.5 Omni Plus, Qwen-3.5 Omni Flash, and Audio Flamingo Next Instruct). The study reveals that pitch hearing is highly unreliable across current systems, with performance degrading sharply under mild acoustic perturbations and varying by sound source and notation.
3. Open Generation Pipeline: The release of the pitchbench Python package enables reproducible stimulus generation across 19 sound sources and all experimental conditions, facilitating future work on pitch-aware audio-language modeling.

Results

The evaluation of six frontier models yielded the following findings:

• Overall Unreliability: No model achieved robust pitch perception. Mean accuracy across all models was low, with the best-performing model (Qwen-3.5 Omni Plus) achieving only 47.7% average accuracy.
• Model Disparities:
  • Qwen-3.5 Omni Plus dominated, particularly in structured tasks (e.g., 96.2% on reference pitch, 88.5% on time-stretching). However, it struggled significantly with untangling simultaneous pitches (polyphony).
  • Qwen-3.5 Omni Flash mirrored the Plus model but with reduced capacity.
  • Gemini 3.1 Pro showed uneven performance, strong in counting and reference tasks but near-zero in transformation-heavy settings (e.g., vibrato).
  • GPT-4o audio and Gemini 3 Flash performed substantially worse, failing to generalize beyond basic perceptual tasks.
• Failure Modes:
  • Polyphony: All models failed to track melodic lines in polyphonic textures (Level 3), with scores of 0% on Bach chorale voice identification.
  • Acoustic Robustness: Performance degraded significantly under short note durations (50ms), detuning, and background noise.
  • Notation Bias: Models exhibited strong biases toward specific notations. For instance, GPT-4o showed a preference for natural notes (A4) over accidentals, suggesting reliance on text-training priors rather than auditory resolution.
  • Multiple-Choice Inflation: Reformulating tasks as multiple-choice questions (MCQ) artificially inflated performance for weaker models (e.g., Gemini 3.1 Pro gained +38.7 percentage points), confirming that open-ended evaluation is necessary to measure genuine perception.
  • Relative vs. Absolute Pitch: Providing a reference tone (relative pitch) substantially improved performance for most models, suggesting they may encode relational information better than absolute pitch categories.

Significance and Claims

The paper posits that reliable musical perception is a critical prerequisite for the deployment of ALMs in music-related applications. The authors claim that current models cannot be trusted to reason about, teach, or transcribe audio in the real world because they lack stable pitch perception, even for controlled synthetic stimuli.

PitchBench serves as a diagnostic tool rather than a mere leaderboard. By isolating specific perceptual components, it reveals that high-level success in music tasks does not imply underlying perceptual competence. The authors conclude that while ALMs have made rapid progress in general audio understanding, they have not yet developed the "perfect pitch" required for robust musical cognition. The benchmark bridges the gap between broad audio-language model evaluation and the rigorous, pitch-centric analysis traditionally found in Music Information Retrieval (MIR).