💬 NLP

Alignment Is All You Need: Instruction-Free Training for General Audio-Language Models

This paper introduces an instruction-free, alignment-only framework for training general audio-language models that keeps both the audio encoder and LLM frozen while learning only a lightweight projector on self-generated data, demonstrating that competitive multimodal performance can be achieved without extensive task-specific supervision or fine-tuning.

Original authors: Xuanru Zhou, Yiwen Shao, Jiahong Li, Dong Yu

Published 2026-08-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xuanru Zhou, Yiwen Shao, Jiahong Li, Dong Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving world of artificial intelligence, a specific type of computer program has emerged that can read text, look at images, and listen to sounds, then answer questions about them. These systems, known as multimodal large language models, are built by taking a powerful text-processing engine and teaching it to understand other forms of data, like audio. Traditionally, this teaching process has been long and labor-intensive. It usually involves three distinct steps: first, connecting the new sound data to the text engine; second, training the system on thousands of specific tasks with human-written instructions; and third, fine-tuning its behavior based on human preferences to make it sound more natural. The prevailing belief has been that without this heavy, multi-stage supervision, the system would not learn to listen effectively.

However, a new study challenges this assumption by asking a simpler question: if the text engine already knows how to follow instructions and reason, does it really need to be retrained from scratch to understand sound? The researchers propose that the heavy lifting of teaching the model specific tasks might be unnecessary. Instead, they suggest that simply aligning the sound data with the text engine's existing knowledge might be enough. This approach treats the audio not as a new language to be learned, but as a different way of presenting information that the engine is already capable of understanding, provided the connection between the two is clear.

The researchers set out to test this idea by building a system that skips the traditional training stages entirely. They took a pre-existing, powerful text model and a pre-existing audio encoder, which is a tool that converts raw sound waves into a format the computer can process. Crucially, they froze both of these components, meaning they did not change a single internal parameter of the text engine or the audio tool during the training process. The only part of the system they allowed to learn was a small, lightweight connector, or projector, that sits between the audio and the text engine. This connector's sole job was to translate the audio features into a language the text engine could understand, without ever being told what specific task to perform.

To train this connector without human instructions, the team developed a method called self-generated data construction. Instead of asking humans to write questions and answers for every audio clip, they used the text engine itself to create the training material. They took simple descriptions of audio clips, such as a sentence stating "a dog barking," and fed them into the frozen text engine without any prompt or instruction. The engine, acting as a creative writer, would then expand that simple description into a long, free-form response, imagining what it would say if it were actually hearing the sound. These generated responses became the target answers. The system then learned to make the audio sound like those descriptions by adjusting only the small connector, effectively teaching the audio to speak the same language as the text engine's internal thoughts.

When tested against a wide variety of benchmarks covering speech, music, and environmental sounds, this instruction-free approach performed remarkably well. The model matched or even surpassed systems that had undergone the traditional, expensive multi-stage training, despite using significantly less data. On tests measuring the ability to follow complex instructions, the new model outperformed many open-source competitors, retaining the original text engine's natural ability to follow commands because that engine was never altered. The study suggests that the bottleneck in building these audio systems is not the lack of massive instruction sets, but rather the quality of the connection between the sound and the text engine.

The researchers also explored how different factors influenced the results. They found that the performance of the system was limited by two main things: how much information the audio tool could extract from the sound, and how capable the text engine was at reasoning. If the audio tool was good at capturing sound details but the text engine was weak at reasoning, the system would not perform well, and vice versa. They discovered that simply adding more data did not always help; for tasks that required recognizing specific sounds, the system hit a performance ceiling quickly. However, for tasks requiring open-ended reasoning or creative descriptions, more data continued to improve the results. This indicates that the system's potential is defined by the capabilities of its core components rather than just the volume of training examples.

One interesting finding was that the system worked best when the text engine used to generate the training answers was the same one used for the final listening task. If the researchers used a different engine to create the training data, the performance dropped significantly. This suggests that the system is not just learning to recognize sounds, but is learning to align with the specific internal logic of a particular text engine. Furthermore, the study showed that the system could adapt to newer generations of text engines simply by retraining the small connector, without needing to retrain the massive engines themselves. This implies that as text models improve, audio models could be upgraded instantly by just swapping the connector, rather than rebuilding the entire system.

The study concludes that building a competitive audio-language model does not require the complex, multi-stage pipelines that have become standard. By freezing the core components and focusing solely on aligning the audio with the text engine's existing knowledge, researchers can create systems that are efficient, adaptable, and highly capable. This approach preserves the original strengths of the text engine, such as its ability to follow instructions, which are often lost when models are heavily fine-tuned for specific tasks. The results suggest that the path to more advanced audio understanding may lie not in teaching the model new skills, but in finding the most efficient way to let it use the skills it already possesses.

Technical Summary: Alignment Is All You Need for General Audio-Language Models

Problem Statement

Current Multimodal Large Language Models (MLLMs), including Large Audio-Language Models (LALMs), typically rely on a multi-stage training pipeline: cross-modal alignment, Supervised Fine-Tuning (SFT) with task-specific instructions, and Preference Optimization (e.g., DPO). This paradigm assumes that adapting a pretrained Large Language Model (LLM) to a new modality requires extensive, task-specific supervision. However, this approach has two primary drawbacks:

Erosion of Generalization: Extensive SFT and preference optimization adapt the LLM to specific instruction distributions, potentially eroding its native, universal instruction-following capabilities and causing the model to overfit to training templates rather than the underlying input signal.
Inefficiency and Scalability: Every new generation of LLMs requires rerunning expensive post-training pipelines on a fresh backbone, making the process computationally heavy and slow to adapt.

The central question posed by the authors is whether alignment alone is sufficient to build a competitive multimodal model, leveraging the LLM's pre-existing reasoning and instruction-following abilities without retraining the LLM or the audio encoder.

Methodology

The paper proposes an Instruction-Free Alignment-Only training framework. The core philosophy is that a well-aligned projector can connect a frozen audio encoder to a frozen LLM, allowing the LLM to interpret audio as it interprets text, without needing to be retrained on specific tasks.

1. Architecture

The model consists of three components:

Frozen Audio Encoder ( $E$ ): Maps raw audio $x$ to feature vectors. The paper evaluates five encoders across three paradigms: discriminative (AudioSet-Zipformer), ASR-supervised (Whisper-large-v2), and joint audio-language pretraining (Qwen family).
Frozen LLM ( $L$ ): A pretrained autoregressive model (e.g., Qwen2.5-7B-Instruct) that acts as a universal decoder.
Trainable Projector ( $P_\theta$ ): A lightweight, learnable module (a simple two-layer MLP) that maps the encoder's output into the LLM's input embedding space. Only the projector parameters are updated.

2. Self-Generated Data Construction

To avoid instruction-induced bias, the authors eliminate explicit task instructions and QA pairs. Instead, they utilize a Self-Generated Data Construction pipeline:

Semantic Surrogate: A natural language caption $c$ (either ground-truth or synthetically generated by a captioner) serves as a semantic surrogate for the audio $x$ .
Response Expansion: A frozen LLM expander $g$ takes the caption $c$ and generates a free-form response $r = g(c)$ as if it were directly perceiving the audio.
Training Target: The resulting dataset consists of pairs $(x, r)$ , where $r$ is the target response generated from the caption surrogate.

3. Instruction-Free Alignment Training

The training objective minimizes the causal language modeling loss on the response tokens $y$ (derived from $r$ ), given the audio prefix $h = P_\theta(E(x))$ :
$\mathcal{L}_{align}(\theta) = -\mathbb{E}_{(x,y) \sim \mathcal{D}_{align}} \sum_{t=1}^{n} \log p_L(y_t | y_{<t}, P_\theta(E(x)))$
Crucially, no instructions are prepended to the input during training. The audio prefix alone must trigger the desired response. The LLM instance used for training must be identical to the one used as the expander $g$ to ensure distributional consistency.

4. Theoretical Basis

The approach relies on the hypothesis that modern LLMs are universal decoders. If the projector achieves complete alignment (where the audio prefix elicits the same response distribution as the textual caption in the absence of instructions), the model generalizes to any test-time instruction. The performance is theoretically bounded by $\min(I(E), C(L))$ , where $I(E)$ is the information retained by the encoder and $C(L)$ is the response space of the frozen LLM.

Key Contributions

Instruction-Free Alignment-Only LALM: The authors demonstrate that a competitive LALM can be built by training only a lightweight projector while keeping both the audio encoder and LLM fully frozen. This approach matches or surpasses heavily post-trained baselines using substantially less data.
Preservation of Instruction Following: By freezing the LLM, the model retains its native instruction-following competence, achieving state-of-the-art performance on instruction-following benchmarks among open-source LALMs without SFT or preference optimization.
Fully Automatic Pipeline: The work extends the Self-Generated Data Construction framework from speech to general audio, enabling a training pipeline that requires no manually curated QA pairs, task templates, or instruction taxonomies.
Ablation and Scaling Insights: The study provides extensive ablations showing that performance is jointly constrained by encoder representations and LLM capability. It finds that scaling alignment data primarily benefits open-ended reasoning tasks, while closed-set recognition saturates quickly based on encoder limitations.

Experimental Results

The model was evaluated on four benchmarks: MMAU, MMAR, MMSU, and MMAU-Pro.

Performance vs. Baselines: The AudioSet-Zipformer variant (7B parameters, trained on ~576k samples) achieved 80.8/77.4 on MMAU (Sound), exceeding the previous open-source best (Audio-Flamingo 3) which used 46 $\times$ more data. On MMAU-Pro, it scored 52.8 (average), outperforming other open-source models (e.g., Qwen2.5-Omni: 52.2, Kimi-Audio: 46.6) and matching proprietary systems like GPT-4o-Audio (52.5).
Instruction Following: The model achieved 62.9 on the instruction-following split of MMAU-Pro, the highest among listed open-source LALMs, validating the preservation of the frozen LLM's capabilities.
Encoder Sensitivity: The discriminative AudioSet-Zipformer excelled in sound and music tasks, while the ASR-supervised Whisper-large-v2 performed better on speech-centric tasks (MMSU, MMAU Speech).
Scaling Behavior: Increasing training data from 400k to 4M samples showed diminishing returns on closed-set benchmarks (MMAU), which saturated quickly. However, open-ended reasoning tasks (MMAU-Pro Open-ended, MMAR) continued to improve with more data.
Caption Sources: Synthetic captions generated by LLMs were competitive with ground-truth captions, particularly for speech-centric tasks, though ground-truth captions held a slight edge for some encoders.

Significance and Claims

The paper claims that alignment alone is sufficient to build competitive multimodal models, challenging the necessity of extensive SFT and preference optimization. By reducing multimodal extension to a lightweight projector-training problem, the approach offers:

Efficiency: Rapid convergence and significantly lower computational costs compared to full fine-tuning.
Portability: The recipe can be seamlessly transferred to new LLM generations by simply retraining the projector, preserving the LLM's universal decoding behavior.
Generalization: The model avoids instruction-induced bias, allowing it to generalize to arbitrary instructions at inference time without further adaptation.

The authors acknowledge limitations, noting that performance is fundamentally bounded by the base encoder and LLM capabilities, and that the current study is limited to the audio domain and 7B-scale models. They suggest that while alignment captures semantics well, richer super-semantic signals might still require specialized supervision.