One Prompt, Many Sounds: Modeling Listener Variability in LLM-Based Equalization

Technical Summary: One Prompt, Many Sounds: Modeling Listener Variability in LLM-Based Equalization

Problem Statement
Conventional audio equalization is a static, manual process that often fails to adapt to changing listening contexts (e.g., mood, location) or individual user preferences. A significant challenge lies in the "semantic gap" between natural language descriptors (e.g., "warm," "clear") and technical frequency response parameters. Existing approaches typically treat this mapping as a deterministic regression problem, predicting a single "correct" point estimate for a given prompt. However, the paper argues that audio perception is inherently subjective; a single prompt corresponds to a distribution of valid preferences across different listeners, not a universal truth. Current systems often collapse this diversity into a mean value or rely on expert-defined presets, failing to capture the nuance of human intent and the variability of user interpretation.

Methodology
The authors propose a framework that treats natural language-to-equalization mapping as a distributional modeling problem rather than a point-estimation task.

• Data Collection: A controlled listening experiment was conducted with 11 participants. The dataset consists of 120 unique prompt-audio pairs derived from a survey of audio experiences. Participants adjusted a simplified 2D "Beosonic" equalization controller (mapping to a "smile curve" and a linear spectral tilt) to match natural language prompts. This process generated 1,320 annotations (11 responses per prompt), creating an empirical distribution of user preferences for each input.
• Modeling Approaches: The study utilizes the Phi-3.5-mini-instruct LLM and explores two primary strategies:
  1. In-Context Learning (ICL): Without updating model weights, the authors tested zero-shot (Text2Beosonic), few-shot (Static-ICL), and Retrieval-Augmented Generation (RAG) approaches. A novel variant, RAG-QA, was introduced to generate a list of 11 predictions (mimicking the participant pool) rather than a single output, thereby introducing stochasticity to reflect human subjectivity.
  2. Parameter-Efficient Fine-Tuning (PEFT): The authors applied Prefix-Tuning and Low-Rank Adaptation (LoRA). Two output strategies were explored: (a) a regression head predicting a fixed array of 11 coordinate pairs simultaneously, and (b) a next-token prediction task generating the text representation of the full coordinate array. Data augmentation via synonym replacement and label blurring was used to expand the training set.
• Evaluation Metrics: To address the limitations of Mean Squared Error (MSE) in capturing distributional variance, the paper employs the Kantorovich-1 distance (Earth Mover's Distance). This metric compares the predicted distribution of settings against the ground-truth distribution of human responses. To handle the bounded nature of the equalization controller (preventing boundary bias in density estimation), the authors utilize Reflective Kernel Density Estimation (KDE) to compute a Reflective Kantorovich Distance.

Key Contributions

1. LLM-Based Recommender Framework: A novel system that interprets audio descriptions to generate plausible equalization adjustments as a distribution of settings, explicitly modeling the one-to-many relationship between language and acoustic parameters.
2. Evaluation Methodology: A rigorous assessment framework using distributional metrics (Kantorovich distance) to evaluate how well models capture the variance and spread of subjective human responses, moving beyond single-point error metrics.
3. Curated Dataset: A dataset of 120 prompt-audio pairs with 11 distinct human annotations each, gathered from a controlled listening experiment, specifically designed to capture listener variability.

Results
The evaluation demonstrates that LLM-based approaches significantly outperform random sampling and static preset baselines in terms of distributional alignment.

• Model Performance: Both ICL and PEFT methods successfully translated natural language prompts into plausible equalization settings. While PEFT methods (specifically LoRA with next-token prediction) achieved the lowest median distances, the improvements over optimized ICL methods (like RAG-QA) were not statistically significant given the limited dataset size.
• Metric Validation: The Reflective Kantorovich Distance proved effective in distinguishing between models that capture the diversity of user preferences and those that collapse to a "safe" average or hallucinate settings outside the zone of plausibility.
• Statistical Significance: All proposed methods showed statistically significant improvements over random guessing, validating the feasibility of using LLMs to model the subjective landscape of audio control.

Significance and Claims
The paper claims to establish the feasibility of using LLMs as "artificial equalizers" that function by modeling the distribution of likely settings rather than seeking a single deterministic solution. By acknowledging that natural language prompts are polysemous and subjective, the proposed framework offers a path toward more accessible, context-aware, and expert-level audio tuning.

The authors maintain a modest stance regarding their findings:

• They note that the dataset size (120 prompts) limits statistical power and that 11 responses may not fully capture global population norms.
• They clarify that the study utilized text-only conditioning to isolate the semantic gap, deliberately excluding audio signal analysis to avoid complexity and latency, though they acknowledge audio conditioning is necessary for a fully context-aware system.
• Crucially, they state that while the models successfully mimic human choice distributions (minimizing the Kantorovich distance), the work has not yet undergone perceptual validation with new human listeners to confirm actual user satisfaction. The primary contribution is the computational foundation for treating natural language as a window into a distribution of preferences.