⚡ electrical engineering

JASTIN: Aligning LLMs for Zero-Shot Audio and Speech Evaluation via Natural Language Instructions

JASTIN is a novel, instruction-driven framework that aligns large language models with frozen audio encoders to achieve state-of-the-art zero-shot evaluation of diverse audio and speech content by formulating assessment as a self-instructed reasoning task.

Original authors: Leying Zhang, Bowen Shi, Haibin Wu, Bach Viet Do, Yanmin Qian

Published 2026-05-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Leying Zhang, Bowen Shi, Haibin Wu, Bach Viet Do, Yanmin Qian

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a music producer or a voice-over artist. You've just created a new audio clip, and you need to know: "Is this good?"

In the past, you had two choices:

The "Human Panel": Pay a group of people to listen and give you a score. This is the gold standard, but it's expensive, slow, and hard to scale.
The "Robot Ruler": Use a computer program with a fixed set of rules (like a ruler) to measure the sound. The problem? These rulers are rigid. A ruler designed to measure the straightness of a pencil line might break if you try to use it to measure the curve of a smile. Similarly, old audio tools often fail when you switch from speech to music, or when the "rules" of what sounds "good" change.

Enter JASTIN.

What is JASTIN?

Think of JASTIN as a super-smart, adaptable audio critic that has been trained to listen like a human but work like a machine.

Instead of using a single, rigid ruler, JASTIN uses a "conversation." You tell it what you want to know (e.g., "Rate how natural this voice sounds on a scale of 1 to 10," or "Tell me if this music sounds too noisy"), and it listens to the audio and gives you a score based exactly on those instructions.

How Does It Work? (The "Translator" Metaphor)

The paper describes a three-part system that works like a team of specialists:

The Ears (The Frozen Encoder): Imagine a pair of super-sensitive ears that have already heard millions of sounds. They are "frozen," meaning they don't learn anything new; they just do what they were born to do: break down sound into tiny, detailed pieces.
The Translator (The Adapter): The "Ears" speak a language of raw sound waves, but the "Brain" only speaks human language. The Adapter is a translator that converts those sound waves into words the Brain can understand, without losing any of the nuance.
The Brain (The LLM): This is a Large Language Model (like a very advanced chatbot). It's great at understanding instructions, reasoning, and context. It takes the translated sound and the specific instructions you gave it, then uses its "common sense" to decide on a score.

The Secret Sauce: How They Trained It

The biggest challenge with AI critics is that they usually get confused if you change the wording of your question. If you ask, "How noisy is this?" it might give a different answer than if you ask, "Is this quiet?" even though you mean the same thing.

To fix this, the researchers built a training pipeline that is like a "drill sergeant" for the AI:

Multi-Source: They fed the AI audio from everywhere: human speech, music, and random sounds.
Multi-Task: They didn't just ask it to rate "quality." They asked it to rate "emotion," "distortion," "naturalness," and even made up fake tasks to teach it how to think.
The "Paraphrasing" Trick: This is the most creative part. They took one single question and had another AI rewrite it in 20 different ways (short, long, formal, casual, inverted logic). They taught JASTIN that "Rate the noise" and "Tell me how clear the signal is" are the same request. This makes JASTIN robust—it doesn't get tripped up by how you phrase your question.

What Did They Find?

The paper claims JASTIN is a game-changer for three main reasons:

It's a "Zero-Shot" Champion: Usually, if you want an AI to judge music, you have to train it specifically on music. If you want it to judge speech, you train it on speech. JASTIN is different. You can ask it to judge a song, a voice, or a sound effect it has never seen before, and it will still do a great job without needing extra training.
It Matches Human Taste: When the researchers compared JASTIN's scores to actual human listeners, JASTIN was much closer to human opinion than the old "robot rulers" or even other fancy AI models.
It's Flexible: You can tell JASTIN to use a 1–5 scale, a 1–100 scale, or even a "Pass/Fail" system, and it will adjust its scoring logic instantly.

Where Does It Struggle? (The Limitations)

Even the best critics have blind spots. The paper admits JASTIN isn't perfect at everything:

Speed Rater: It's not great at judging how fast or slow someone is speaking. It sometimes misses the rhythm.
Whispering (ASMR): When judging ASMR (whispering sounds), the AI gets confused. It often mistakes the "breathy" nature of a whisper for a technical glitch or noise, because it's used to judging clear, loud voices.

The Bottom Line

JASTIN is a new way to evaluate audio. Instead of forcing audio into a box with a single measurement, it treats audio evaluation as a conversation. By teaching an AI to understand instructions and adapt to different contexts, the researchers have created a tool that is more flexible, more accurate, and more like a human listener than anything that came before it.

Technical Summary: JASTIN – Aligning LLMs for Zero-Shot Audio and Speech Evaluation via Natural Language Instructions

1. Problem Statement

The rapid advancement of generative audio models (e.g., text-to-speech, music generation) has outpaced the development of robust evaluation methodologies. Current evaluation approaches face three critical bottlenecks:

Narrow Domain Applicability: Traditional signal-processing metrics (e.g., PESQ, STOI) and specialized neural metrics (e.g., NISQA, UTMOS) are often domain-specific. For instance, PESQ is unsuitable for music, while Fréchet Audio Distance (FAD) struggles with speech, leading to fragmented evaluation pipelines.
Inconsistent Performance of General MLLMs: While general-purpose Multimodal Large Language Models (MLLMs) like Gemini 3 Pro or GPT-4o show promise in zero-shot tasks, their performance in specialized audio evaluation remains inconsistent and often lacks the precision required for rigorous assessment.
Lack of Instructional Flexibility: Specialized LLM-based judges often rely on rigid prompt templates and fixed scoring scales (e.g., 1–5). They struggle with zero-shot generalization to new tasks and cannot dynamically adapt to alternative user requirements (e.g., changing the scale to 1–100 or rephrasing instructions).

Existing solutions fail to provide a unified, instruction-driven framework that can generalize across speech, music, and sound effects without task-specific retraining.

2. Methodology: The JASTIN Framework

JASTIN (Instruction-driven Audio and Speech Evaluation via Natural Language Instructions) reframes audio assessment as a self-instructed reasoning task. The framework bridges high-resolution acoustic representations with the linguistic reasoning capabilities of an LLM.

Architecture

The architecture consists of three main components:

Frozen Audio Encoder: A pre-trained, high-performance encoder (specifically PE-A-Frame-base) extracts robust, frame-level acoustic features. This encoder is kept frozen to preserve its learned representations of temporal dynamics and acoustic events.
Trainable Audio Adapter: A lightweight projection network (linear projection followed by a bottleneck residual adapter with a 4:1 compression ratio) maps the continuous audio embeddings from the encoder into the discrete text token space of the LLM.
Fine-tuned LLM Backbone: A language model (Llama-3.2-3B) serves as the core reasoning engine. It processes interleaved multimodal inputs (task instructions and audio embeddings) to generate scores.

Input Formulation

Audio evaluation is formulated as a context-dependent scoring task: $s = f(T, A)$ , where $s$ is the predicted score, $T$ is the natural language task description, and $A$ is the input audio. The input is structured as a unified conversational sequence where task instructions and audio embeddings are interleaved within a chat template. The model is trained autoregressively to predict the target score sequence, minimizing negative log-likelihood only on the response tokens.

Data Preparation Pipeline

To ensure robust zero-shot generalization, JASTIN employs a comprehensive, heterogeneous data preparation strategy involving four key dimensions:

Multi-Source: Aggregates data from human-annotated ground truth (e.g., QualiSpeech, SpeechEval), pseudo-labeled data (generated by the AES model on public datasets like LibriTTS and MusicCaps), and proxy-task data (e.g., emotion detection, reverberation detection) to prevent overfitting to narrow definitions of "quality."
Multi-Task: Incorporates 24 diverse tasks covering naturalness, prosody, emotion, distortion, and production quality across speech, music, and sound.
Multi-Calibration: Applies template-based augmentations to training data, including dynamic rescaling of scores (e.g., mapping 1–5 to 1–100), inverting semantic logic (e.g., "rate noise" vs. "rate clarity"), and converting continuous scores to binary classifications.
Multi-Description: Utilizes a teacher LLM to paraphrase task descriptions, transforming formal requests into casual questions or technical rubrics. This ensures the model learns the underlying intent rather than memorizing specific keywords.

3. Key Contributions

Unified Generalization: JASTIN is a single model capable of evaluating speech, music, and sound effects in a zero-shot, single-turn setting without task-specific retraining.
Comprehensive Data Strategy: The introduction of a multi-source, multi-task, multi-calibration, and multi-description data pipeline significantly enhances the model's ability to generalize across unseen tasks and domains.
Instructional Robustness: By employing a self-instructed training paradigm, JASTIN achieves a balance between semantic sensitivity and lexical robustness. It adapts to changes in evaluation rules and scoring scales while maintaining consistent scoring when prompts are rephrased.
Human-Centric Alignment: The framework is designed to align closely with human subjective ratings, moving beyond static metrics to dynamic, instruction-driven evaluation.

4. Experimental Results

JASTIN was evaluated against traditional non-LLM metrics (e.g., AES, UTMOS, NISQA), general-purpose MLLMs (e.g., Gemini series, Qwen series, AudioFlamingo3), and specialized LLM baselines across five datasets: QualiSpeech, SpeechEval, AES (Speech, Sound, Music), AudioMOS2025, and DeepASMR.

Speech Evaluation: On the QualiSpeech and SpeechEval datasets, JASTIN achieved state-of-the-art (SOTA) Pearson (PCC) and Spearman (SRCC) correlations with human ratings, consistently outperforming both traditional metrics and general MLLMs. For example, on the SpeechEval Overall Quality task, JASTIN achieved a PCC of 0.662, surpassing the best baseline (UTMOS at 0.748 on Intelligibility but lower on Overall Quality) and specialized models.
Sound and Music Evaluation: On the AES dataset, JASTIN matched or exceeded the performance of the specialized non-LLM AES baselines across Content Enjoyment, Usefulness, Production Complexity, and Production Quality metrics, while significantly outperforming speech-centric metrics like UTMOS and NISQA which failed on non-speech tasks.
Zero-Shot Generalization: In out-of-domain (OOD) tasks (e.g., Music-Textual Alignment, ASMR MOS, Synthesized Speech MOS), JASTIN demonstrated superior generalization capabilities. It significantly outperformed general LLMs and traditional metrics, which showed fragmented performance or failed to adapt to unseen task descriptions.
Prompt Robustness: Sensitivity analyses showed that JASTIN maintains stable performance across diverse prompt structures (shortened, restructured, detailed, inverted), whereas baseline models exhibited significant fluctuations.

5. Significance and Claims

The paper claims that JASTIN represents a paradigm shift from static, fixed-metric regression to a dynamic, instruction-driven evaluation framework. Its significance lies in:

Eliminating the Need for Task-Specific Retraining: A single model can handle diverse evaluation scenarios (speech, music, sound) and adapt to new instructions without fine-tuning.
Bridging the Gap between Objective and Subjective: By leveraging the reasoning capabilities of LLMs, JASTIN better captures the nuance of human preference and contextual flexibility compared to traditional signal-processing metrics.
Scalability and Flexibility: The framework is model-agnostic, allowing for the substitution of better audio encoders or LLM backbones in the future, and supports dynamic scoring scales and custom rubrics.

The authors acknowledge limitations, noting that the model still struggles with temporally sensitive metrics like speaking speed and specialized aesthetic domains like ASMR, where the distinction between artistic style and technical degradation remains challenging. However, the work establishes a robust foundation for the next generation of automated audio assessment.