🤖 AI

Auto-AEG: Scalable Data Construction for Open-Vocabulary Audio Event Grounding

This paper introduces Auto-AEG, a scalable pipeline that addresses the data scarcity bottleneck in Open-Vocabulary Audio Event Grounding by combining programmatically synthesized clips with multi-model pseudo-labels and reinforcement learning to significantly enhance the temporal localization capabilities of Large Audio-Language Models.

Original authors: Zihan Zhang, Xize Cheng, Wenhao Yan, Tong Zhang, Dongjie Fu, Boyun Zhang, Yongbo He, Tao Jin

Published 2026-07-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zihan Zhang, Xize Cheng, Wenhao Yan, Tong Zhang, Dongjie Fu, Boyun Zhang, Yongbo He, Tao Jin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a super-smart robot to listen to the world. You've already taught it to be a great storyteller; if you play a recording of a storm, it can tell you, "That sounds like heavy rain and thunder." But there's a tricky part: the robot is terrible at knowing exactly when things happen. It might say, "I hear rain," but it can't tell you if the rain started at 2:03 or 2:15. On the other hand, old-school sound detectors are like strict librarians: they can pinpoint the exact second a sound starts and stops, but they only recognize a tiny, pre-approved list of words. If you ask them about a "squeaky door," they might just say, "I don't know that word," even if the sound is right there.

This paper tackles the messy middle ground: teaching a smart robot to not only understand what a sound is but also to point its finger at the exact moment it happens, even if the sound is something it has never heard of before. The big problem is that teaching this skill is incredibly hard because it takes forever to hire humans to listen to hours of audio and mark down every single second a sound starts and stops. It's like trying to teach someone to play the piano by having them watch a million hours of videos without ever letting them touch the keys. The researchers wanted to find a way to teach the robot without needing a million human teachers.

Enter Auto-AEG, a new method that acts like a clever, automated tutor. Instead of waiting for humans to label every sound, the researchers built a pipeline that creates its own practice material. First, they use a computer program to mix and match sounds like a DJ, creating thousands of fake audio clips where the computer knows exactly when every sound starts and stops because it put them there. This gives the robot a "cold start"—a perfect, no-mistakes foundation to learn the basics of timing.

But the robot needs to learn from real life, too, where sounds are messy and overlap. Here's where the magic happens: the researchers let the robot practice on real-world audio (like recordings from the internet) where the timing labels are a bit fuzzy. Instead of just correcting the robot like a teacher, they use a technique called Reinforcement Learning. Think of this like a video game: the robot tries to guess the timing, and if it gets it close enough, it gets a "reward point." If it guesses wildly wrong, it gets no points. Over time, the robot learns to aim for those reward points, figuring out how to be precise even when the training data isn't perfect.

The paper shows that this two-step approach works wonders. When they tested their trained robots on a brand-new, difficult test set called AEGBench (which they also created to make sure the test was fair and covered tricky situations like overlapping sounds or very long noises), the results were impressive. The robots improved their ability to pinpoint sound timing by huge margins—up to 73.9% better than before for the larger model. They didn't just get better at guessing; they learned to be precise.

Crucially, the authors found that simply showing the robot more real-world audio wasn't enough. If they just kept teaching the robot with standard lessons on real audio, it didn't get much better. It was the "reward game" (the reinforcement learning) that turned the messy, imperfect data into a superpower. This suggests that the key to teaching robots to understand time in sound isn't just having more data, but having a smarter way to learn from it. They also proved that this method works on different sizes of robots, from smaller ones to massive ones with 30 billion parameters, and that it helps them stay good at understanding sounds in general, not just at finding them.

In short, this paper suggests that we don't need to wait for humans to label every sound in the universe to build better audio robots. By mixing perfect synthetic practice with a "reward-based" learning game on real-world data, we can teach these models to hear the world with much sharper timing, opening the door for them to understand complex, real-life soundscapes much more effectively.

Technical Summary: Auto-AEG: Scalable Data Construction for Open-Vocabulary Audio Event Grounding

Problem Statement

Large Audio-Language Models (LALMs) have demonstrated remarkable capabilities in audio understanding and reasoning, bridging acoustic and linguistic domains. However, they struggle to precisely localize sound events in time, often producing coarse or hallucinated temporal boundaries. Conversely, classical Sound Event Detection (SED) systems achieve frame-level precision but are constrained by closed-label vocabularies that cannot generalize to the richness of natural language queries.

This gap defines the task of Open-Vocabulary Audio Event Grounding (AEG): given an audio clip and a natural language query describing a target sound event, the model must predict all onset/offset intervals where that event occurs, supporting multiple occurrences and an open vocabulary. Progress in this field is fundamentally bottlenecked by data scarcity. Existing datasets are small, lack label diversity, and manual annotation of fine-grained temporal boundaries for arbitrary events is prohibitively expensive.

Methodology: Auto-AEG Pipeline

To address data scarcity without architectural changes, the authors introduce Auto-AEG, a scalable, annotation-free pipeline that constructs open-vocabulary onset/offset supervision through automatic data construction and model fine-tuning. The pipeline operates in two distinct stages, each tailored to a specific training objective:

1. Stage 1: Programmatic Synthesis for SFT Cold-Start

Objective: Provide exact ground-truth (GT) intervals to establish a stable cold-start for Supervised Fine-Tuning (SFT).
Process: The system programmatically synthesizes 10,000 training clips by composing segments from a pre-screened audio pool (derived from WavCaps).
- A target label is selected, and 1–5 non-overlapping occurrences are laid down.
- Distractor events and Gaussian background noise are added at controlled signal-to-noise ratios.
- Because the composition is scripted, the ground-truth intervals are exact and noiseless.
Training: Models are trained on these synthetic clips using a "List-All" query format, encouraging the model to output a JSON array of intervals. This stage instills the response format and basic acoustic recognition before exposure to real-world variability.

2. Stage 2: Multi-Model Annotation for GRPO

Objective: Leverage real-world audio to refine temporal localization using Reinforcement Learning, tolerating the inherent noise of automatic annotations.
Process: Real-world clips (10–30s) from FreeSound are annotated via a multi-model collaboration pipeline:
- Label Identification: Gemini processes 10-second chunks to identify audible events, pooling labels into a semantically orthogonal inventory.
- Event-Type Classification: Labels are classified as continuous (e.g., rain) or discrete (e.g., a knock) to determine span-merging logic.
- Temporal Localization: PE A-Frame localizes labels by thresholding per-frame audio-text similarity, merging adjacent spans only for continuous events.
- Global Cleaning: CLAP embeddings and an LLM are used to cluster near-synonyms and canonicalize labels, preventing vocabulary fragmentation.
Training: The 2,000 real-world clips (yielding 5,244 queries) are used for Group Relative Policy Optimization (GRPO).
- Reward Function: A composite reward ( $r$ $r$ ) combines:
  - $r_{iou}$ : F1-IoU@0.5 (penalizing both missed detections and hallucinations).
  - $r_{fmt}$ : Format compliance (presence of reasoning trace and valid JSON).
  - $r_{nem}$ : Non-empty reward (penalizing over-rejection).
  - $r_{prec}$ : Precision penalty (suppressing spurious short intervals).
- An L2 regularizer prevents the policy from drifting too far from the SFT initialization.

Key Contributions

Auto-AEG Pipeline: A scalable, two-stage framework that automatically constructs open-vocabulary onset/offset supervision. It uniquely pairs exact-GT synthetic data for SFT cold-starts with noisy real-world pseudo-labels optimized via GRPO.
AEGBench: An independent, quality-filtered evaluation benchmark constructed from four diverse sources (AudioSet, FSD50K, BBC, YouTube). It features difficulty stratification across six hard-case categories (Polyphonic Overlap, Gradual Onset/Offset, Repeated Occurrence, Low-Contrast, Semantic Ambiguity, and Long Duration) to enable diagnostic analysis.
Two-Stage Fine-Tuning Framework: A methodology that couples instruction-tuned cold starts with interval-aware GRPO. The authors demonstrate that this approach converts imperfect pseudo-labels into significant gains in temporal grounding without architectural modifications.

Experimental Results

The authors evaluated the pipeline on Qwen3-Omni-30B and Qwen2.5-Omni-7B models using the AEGBench (3,427 items, 9,790 queries) and the standard DESED SED benchmark.

Performance on AEGBench:
- Qwen3-Omni-30B: Achieved a mean Intersection-over-Union (mIoU) of 0.480 after SFT+GRPO, a +73.9% relative gain over the zero-shot baseline (0.276) and a +29.4% gain over SFT alone. It surpassed the strongest external zero-shot baseline, Gemini-3-Pro (0.323), on all metrics.
- Qwen2.5-Omni-7B: Showed nuanced results. While overall mIoU slightly decreased after GRPO (0.424 $\to$ 0.399), the Event F1 score increased significantly (0.416 $\to$ 0.474), and onset precision rose substantially (0.411 $\to$ 0.594). This suggests the precision-oriented reward successfully suppressed over-prediction in the smaller model, trading recall for sharper boundary estimates.
- Hard-Case Analysis: The models showed significant improvements on Gradual Onset/Offset and Low-Contrast categories, though Long Duration events remained challenging due to encoder window limits.
Performance on DESED (SED Task):
- Training on Auto-AEG data improved performance on the standard closed-set SED task (DESED).
- Qwen3-Omni-30B: PSDS1 increased from 0.153 (zero-shot) to 0.170 (SFT+GRPO).
- Qwen2.5-Omni-7B: PSDS1 increased from 0.120 to 0.136.
- Ablation studies confirmed that the gains stem from the GRPO objective rather than mere exposure to real audio, as continued SFT on the same real data yielded no improvement.

Significance and Claims

The paper claims that the gap between LALM reasoning and precise temporal localization can be closed on the data side without architectural changes. The core finding is that reinforcement learning (GRPO) is the critical mechanism that converts automatically constructed, imperfect pseudo-labels into effective training signals.

The authors assert that:

Data Construction is Sufficient: Automatically constructed data, when paired with interval-aware reward design, is an effective route to expanding the temporal localization capability of LALMs.
GRPO is Key: The improvement is driven by the RL objective, not just the volume of data. Supervised learning on noisy pseudo-labels (continued SFT) failed to improve performance, whereas GRPO succeeded.
Scalability: The Auto-AEG pipeline offers a scalable, annotation-free foundation for advancing LALM temporal capabilities, addressing the prohibitive cost of manual temporal annotation.

The work positions itself as the first to couple annotation-free data construction with reinforcement learning specifically for open-vocabulary audio event grounding, providing a new paradigm for training LALMs on temporal tasks where high-quality labeled data is scarce.