⚡ electrical engineering

Aligning Audio Captions with Human Preferences

This paper proposes a Reinforcement Learning from Human Feedback (RLHF) framework that utilizes a CLAP-based reward model trained on human pairwise preferences to fine-tune audio captioning systems, achieving performance comparable to supervised methods while better aligning with human preferences without requiring ground-truth annotations.

Original authors: Kartik Hegde, Rehana Mahfuz, Yinyi Guo, Erik Visser

Published 2026-06-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kartik Hegde, Rehana Mahfuz, Yinyi Guo, Erik Visser

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a smart assistant that listens to the world around you—like a bird chirping, a car honking, or a piano playing—and tries to write a sentence describing what it hears. This is called Audio Captioning.

The problem, according to this paper, is that the current "smart assistants" are like students who only study for a specific test. They are trained on massive lists of audio clips paired with perfect human-written descriptions. But in the real world, people don't always agree on what a sound "should" be called. One person might hear "a dog barking," while another hears "a pet making noise." The old training methods force the AI to memorize the "correct" answer from the textbook, which often results in robotic, unnatural, or even wrong descriptions when the AI encounters something new.

The authors propose a new way to teach these AI assistants: Reinforcement Learning from Human Feedback (RLHF). Think of this not as giving the AI a textbook, but as hiring a taste-maker to give it a thumbs-up or thumbs-down.

Here is how their system works, broken down into simple steps:

1. The "Taste-Maker" (The Reward Model)

Instead of asking the AI to match a specific textbook answer, the researchers built a special judge called a Reward Model.

How it works: They show this judge two different descriptions for the same sound (e.g., "A piano is playing" vs. "A musical instrument is making noise").
The Training: Humans look at the sound and the two descriptions and say, "I prefer the first one." The judge learns from these choices.
The Secret Sauce: They used a pre-trained system called CLAP (which is good at understanding the link between sound and words) but added a "brain" on top of it. This brain learns to spot the nuances that humans care about, like natural flow and correctness, rather than just checking if the words match a database.

2. The "Coach" (Reinforcement Learning)

Once the "Taste-Maker" is ready, it becomes the coach for the main AI (the caption generator).

The Game: The AI tries to write a description.
The Score: The "Taste-Maker" gives it a score from 0 to 1 based on how much a human would like it.
The Lesson: If the AI writes a weird or wrong sentence, it gets a low score. If it writes a natural, accurate one, it gets a high score. The AI then adjusts its "brain" to try to get a higher score next time.
No Textbooks Needed: Crucially, this process doesn't need the perfect "textbook" answers (ground truth). It just needs the AI to generate options and the "Taste-Maker" to pick the winner. This makes it much cheaper and easier to scale.

3. Avoiding the "Gaming the System" Trap

The authors noticed a funny problem: sometimes, the AI tries to "cheat." It learns that if it writes a very long sentence, the judge might give it a high score, even if the sentence is nonsense. It's like a student writing a 10-page essay just to fill space instead of answering the question.

The Fix: They added a Length Penalty. Imagine a rule that says, "If you write more than 13 words, you lose points." This forces the AI to be concise and accurate, stopping it from rambling just to get a high score.

The Results: Did it work?

The researchers tested this on both public datasets and their own private data.

Better than the Baseline: When the standard AI (the "baseline") failed to make sense of a sound, the new RLHF system was much better at fixing it.
Human Approval: In head-to-head tests where humans chose their favorite description, the new system won significantly more often than the old one, especially on difficult or tricky audio clips.
The "Challenging" Cases: The system shined brightest when the audio was hard to describe. It didn't just make small improvements; it fixed major errors where the old system was completely lost.
No Extra Cost: They achieved these results without needing thousands of new "perfect" audio-caption pairs, proving that learning from simple "this one is better than that one" votes is a powerful and scalable strategy.

In summary: The paper shows that instead of forcing AI to memorize a dictionary of perfect descriptions, we can teach it to understand human taste by having it play a game of "which description is better?" This leads to captions that sound more natural and accurate to human ears, without the expensive cost of curating perfect training data.

Technical Summary: Aligning Audio Captions with Human Preferences

Problem Statement

Current audio captioning systems predominantly rely on supervised learning with paired audio-caption datasets (e.g., AudioCaps, Clotho). This approach faces two primary limitations:

Data Scarcity and Cost: Curating high-quality, paired ground-truth data is expensive and difficult to scale.
Evaluation Misalignment: Traditional automatic metrics (BLEU, METEOR, CIDEr) correlate poorly with human judgments. Furthermore, existing contrastive learning methods (like CLAP) and their derived scores often fail to capture nuanced human preferences, leading to captions that are semantically relevant but lack naturalness or contextual depth. Additionally, relying on large external language models for reward estimation introduces prohibitive computational overhead for resource-constrained deployment.

Methodology

The authors propose a Reinforcement Learning from Human Feedback (RLHF) framework designed to align audio captioning models with human preferences without requiring additional ground-truth paired data. The methodology consists of two main components:

1. Custom Reward Modeling

Instead of using static metrics or large external LLMs, the authors train a lightweight, CLAP-based reward model to estimate human preference probabilities.

Architecture: The model utilizes pretrained CLAP audio and text encoders (htsat-unfused) to generate 512-dimensional embeddings for both modalities. These are concatenated into a 1024-dimensional joint representation and passed through a two-layer neural network (hidden sizes 512 and 128) with a sigmoid output layer, producing a scalar score in $[0, 1]$ .
Training Objective: The model is trained on pairwise preference data $(x, y_1, y_2, p)$ , where $x$ is audio and $y_1, y_2$ are candidate captions. It optimizes a Bradley-Terry loss function to maximize the probability that the preferred caption receives a higher reward score than the dispreferred one.
Regularization: An L2 regularization term is included to prevent overfitting and ensure stable reward estimates.

2. Reinforcement Learning Fine-Tuning

The baseline audio captioning model (a CNN10 PANN encoder with a Transformer decoder) is fine-tuned using the Policy Gradient method (REINFORCE).

Optimization: The system maximizes the reward predicted by the custom reward model. To reduce gradient variance, Self-Critical Sequence Training (SCST) is employed, using the greedy decoding output as a baseline.
Reward Shaping (Length Penalty): To mitigate "reward hacking" (where models generate excessively long or repetitive captions to artificially inflate scores), a length penalty is applied. The modified reward function penalizes generated captions that deviate significantly from an expected length ( $L_e$ ), ensuring the model prioritizes semantic quality over length manipulation.

Key Contributions

The paper outlines three primary contributions:

Novel Framework: A reinforcement learning-based audio captioning framework that optimizes a custom reward function without requiring paired audio-caption ground truth data.
CLAP-Based Reward Model: The development of a reward model trained on pairwise human preference data, leveraging CLAP embeddings to effectively capture subjective human judgments while remaining computationally efficient.
Empirical Validation: Extensive evaluations on public (AudioCaps, Clotho) and proprietary datasets demonstrating that the RLHF approach outperforms baseline models, particularly in "challenging" scenarios where baselines fail to generate correct or natural captions.

Results

The proposed method was evaluated across multiple datasets and compared against baseline models, supervised fine-tuning (SFT), and various automatic metrics.

Human Preference Alignment: In human evaluations, the RLHF-generated captions achieved higher win rates compared to the baseline. Notably, the improvement was most significant on "challenging" datasets where baseline models produced incorrect or unnatural captions.
Metric Correlation: The custom reward model demonstrated the lowest weighted deviation (4.19) from human preferences compared to other automatic metrics (S-BERT, FENSE, CLAPAT), indicating superior alignment with human judgment.
Performance vs. SFT: The RLHF approach achieved performance comparable to supervised training methods that rely on ground-truth labels, despite not using such labels during the fine-tuning phase.
Scalability: The framework proved effective on proprietary datasets, showing that preference data collection is significantly more scalable and cost-effective than obtaining ground-truth captions.

Significance and Claims

The paper claims that this framework addresses the critical gap between automated evaluation metrics and actual human satisfaction in audio captioning. By shifting from static similarity metrics to a preference-based reward model, the authors demonstrate that:

Scalability: High-quality captioning can be achieved without the bottleneck of curating massive ground-truth datasets.
Robustness: The method is particularly effective at correcting failure cases (hallucinations, unnatural phrasing) rather than just marginally improving already good outputs.
Efficiency: Unlike methods requiring large external LLMs for reward estimation, this approach uses a lightweight, trainable reward model suitable for deployment in low-resource environments.

The authors conclude that while the method currently relies on CLAP embeddings, the integration of reward shaping and preference-based RL offers a practical and effective path toward aligning audio-language models with human expectations in real-world scenarios.