🤖 machine learning

Label Curation Using Agentic AI

The paper introduces AURA, an agentic AI framework that coordinates multiple agents to generate and validate multi-modal labels without ground truth by adapting a probabilistic model with Expectation-Maximization to infer true labels and annotator reliability, achieving significant accuracy improvements over baselines in both standard and challenging annotation scenarios.

Original authors: Subhodeep Ghosh, Bayan Divaaniaazar, Md Ishat-E-Rabban, Spencer Clarke, Senjuti Basu Roy

Published 2026-02-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Subhodeep Ghosh, Bayan Divaaniaazar, Md Ishat-E-Rabban, Spencer Clarke, Senjuti Basu Roy

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to organize a massive library of books, but you don't have a librarian. Instead, you have hired 50 different people to read the books and tell you what genre they belong to. Some are experts, some are tired, some are guessing, and some are just plain wrong. If you just take a "majority vote" (asking, "What do most people say?"), you might end up with the wrong genre if the majority of your helpers are actually bad at their jobs.

This is the problem the paper AURA solves.

The Problem: The "Noisy Crowd"

In the world of Artificial Intelligence, computers need labeled data to learn (like a student needing an answer key). Usually, humans do this labeling, but it's expensive, slow, and humans make mistakes. Recently, we've started using AI models to label data for us. But here's the catch: AI models are also imperfect. Some are smart, some are confused, and some are biased.

If you just let a bunch of AI models vote on the answer, the "wrong" ones can drag the whole group down.

The Solution: AURA (The Smart Manager)

The authors created a system called AURA (Agentic AI for Unified Reliability Modeling and Annotation Aggregation). Think of AURA as a super-smart manager who doesn't just count votes; it figures out who is telling the truth.

Here is how AURA works, using a simple analogy:

The Team of Detectives: AURA gathers a team of different AI "detectives" (agents). One might be great at spotting cats, another at spotting dogs, and a third might be terrible at both.
No Answer Key Needed: Usually, to know who is a good detective, you need to know the right answer beforehand (the "ground truth"). AURA is special because it doesn't need the answer key. It figures out the truth by looking at the patterns of disagreement.
The "Trust Score" Game:
- Imagine the detectives are arguing about a photo. Detective A says "Cat," Detective B says "Dog," and Detective C says "Cat."
- If Detective A and B have been proven to be liars in the past, AURA ignores them.
- If Detective C has been reliable, AURA listens to C.
- AURA does this mathematically. It runs a loop where it guesses the answer, then checks who voted for that answer, then updates its "Trust Score" for each detective, and repeats until everyone agrees on the most likely truth.

The Secret Sauce: The "Expectation-Maximization" Loop

The paper uses a fancy math trick called Expectation-Maximization (EM). Think of this like a game of "Hot and Cold":

Step 1 (Guess): AURA makes a guess about what the true label is for an image.
Step 2 (Check): It looks at the detectives. "Oh, the detectives who usually get things right voted for this guess. The ones who usually get things wrong voted for the other one."
Step 3 (Update): AURA updates its list of who is trustworthy.
Repeat: It does this over and over again. With every round, the "Trust Scores" get more accurate, and the final labels get more accurate.

What Did They Find?

The researchers tested AURA on four different types of data (videos of people doing sports, pictures of food, pictures of birds, and general images).

Beating the Crowd: In every test, AURA was better than just taking a simple majority vote. In some cases, it was 50% more accurate than the baseline methods, especially when the team included some very bad AI models.
Spotting the Fakers: AURA was excellent at figuring out which AI models were reliable and which were useless. For example, it correctly identified that one specific AI model was only right about 6% of the time and stopped trusting it.
No Training Required: You don't need to teach AURA how to do this. It works with "off-the-shelf" AI models right out of the box. You don't need to re-train them or give them special instructions.
Handling Imbalance: Even if the dataset has 100 pictures of dogs and only 1 picture of a bird, AURA doesn't get confused. It treats the rare bird just as carefully as the common dogs.

The Bottom Line

AURA is a system that turns a chaotic group of imperfect AI workers into a highly accurate labeling team. It does this by constantly asking, "Who is telling the truth?" and "What is the real answer?" without ever needing to see the correct answers beforehand. It's like having a manager who can instantly tell which employees are doing a good job and which are slacking off, ensuring the final report is always high-quality.

Technical Summary: AURA (Agentic AI for Unified Reliability Modeling and Annotation Aggregation)

1. Problem Definition

The paper addresses the persistent challenge of producing accurate, unbiased, and scalable labels for supervised learning as datasets grow in size, diversity, and modality. Traditional human-centric pipelines are described as costly, slow, and prone to variability due to subjective interpretation and fatigue. While active learning reduces the annotation burden, it remains constrained by human availability and latency.

Recent advances in Large Language Models (LLMs) and autonomous agents have enabled fully automated annotation pipelines. However, existing approaches largely overlook the systematic modeling of uncertainty, noise, and bias inherent in these agents. Autonomous agents are heterogeneous in reliability, with performance varying across tasks and contexts. The core problem identified is the lack of a statistically grounded, generic framework that can:

Infer high-confidence latent true labels in the absence of ground truth.
Explicitly model and quantify the reliability of individual AI agents.
Aggregate conflicting predictions from heterogeneous agents without requiring domain-specific fine-tuning or in-context learning.

2. Methodology: The AURA Framework

The authors propose AURA (Agentic AI for Unified Reliability Modeling and Annotation Aggregation), a fully automated framework that coordinates multiple off-the-shelf AI agents to collaboratively generate and validate labels.

Core Algorithm: AEML

At the heart of AURA is AEML (Agentic Expectation Maximization Labeling), an adaptation of the classical Dawid-Skene probabilistic model [Dawid and Skene, 1979]. AEML utilizes an Expectation-Maximization (EM) technique to resolve the circular dependency between label correctness and annotator trustworthiness.

The process operates as follows:

Inputs: A set of $n$ instances and $M$ annotators (AI agents) providing noisy labels $\ell^{(a)}_i$ . No ground truth is required.
Initialization: Class priors ( $p_{y^*}$ ) are initialized uniformly. Annotator reliability matrices (confusion matrices $\Theta^{(a)}$ ) are initialized with a hyperparameter $\lambda$ representing an assumed baseline reliability.
E-Step (Expectation): The model estimates the posterior probability of the latent true label $y^*_i$ for each instance. This is calculated by weighting the observed annotations from all agents according to their current reliability estimates and class priors.
$w_{iy^*} = Pr(y^*_i = y^* | \{\ell^{1:M}_i\}, p_{y^*}, \{\Theta^{1:M}\})$
M-Step (Maximization): Using the posterior label estimates from the E-step, the model updates the annotator reliability matrices ( $\Theta^{(a)}$ ). This involves counting the frequency of correct and incorrect annotations weighted by the posterior probabilities, effectively learning how reliable each agent is for specific classes. Class priors are also updated based on the aggregated posterior weights.
Convergence: The algorithm iterates until the difference in log-likelihood between consecutive iterations falls below a threshold $\gamma$ .

Key Design Principles

Off-the-Shelf Models: AURA operates using pre-trained models (e.g., GPT, Gemini, CLIP, Swin) without any fine-tuning or in-context learning.
Heterogeneity: The framework explicitly models agents as having distinct strengths and error characteristics via confusion matrices, rather than assuming uniform accuracy (as in majority voting).
Probabilistic Aggregation: Instead of simple voting, AURA performs principled aggregation by reconciling conflicting predictions based on inferred reliability.

3. Experimental Evaluation

The authors evaluated AURA on four real-world datasets spanning video and image modalities:

Kinetics-400: Video action recognition (1,000 videos, 60 classes).
ImageNet-ReaL: Image classification (4,271 images, 50 classes).
Food-101: Food recognition (1,000 images, 101 classes).
CUB-200: Bird species classification (1,018 images, 17 classes).

Annotators: The experiments utilized a diverse set of 18 heterogeneous agents, including LLMs (GPT-4o, GPT-5, Gemini, Grok), specialized vision models (Swin, CLIP), and domain-specific systems (Pegasus-1.2, QwenVL). A synthetic random annotator was also included to test noise handling.

Baselines: The primary baseline was Majority Voting. Other related approaches were deemed inapplicable as they required fine-tuning or in-context learning.

Metrics: Accuracy, Precision, Recall, F1-score, and Pearson Correlation Coefficient (to measure the alignment between estimated reliability and actual accuracy).

4. Results

Performance Improvement: AURA consistently outperformed Majority Voting across all datasets and metrics. Accuracy improvements ranged from 2.4% (Kinetics-400) to 5.8% (Food-101). In scenarios with poor-quality annotators (simulated by removing top performers), improvements reached up to 50% over the baseline.
Reliability Estimation: The framework accurately estimated annotator reliability. The estimated reliability scores closely tracked the true accuracies of the agents (typical discrepancies of 2-4%), successfully distinguishing high-performing agents (e.g., GPT-5-mini) from low-performing ones (e.g., MoonDream2).
Robustness to Class Imbalance: AURA demonstrated insensitivity to class imbalance, with a low Pearson correlation (0.169) between F1-score and class support, indicating it does not systematically favor majority classes.
Convergence: The model showed stable convergence behavior, with log-likelihood increasing monotonically until reaching a local optimum.

5. Significance and Claims

The paper claims that AURA represents a significant step toward scalable, reliable data annotation for next-generation AI applications. Its primary contributions are:

First Fully Automated Framework: It is presented as the first agentic AI framework for large-scale annotation that relies exclusively on off-the-shelf models without fine-tuning.
Statistical Grounding: Unlike heuristic aggregation methods, AURA provides a statistically grounded probabilistic framework that explicitly models agent uncertainty and bias.
Reliability-Aware Automation: It enables the assessment of annotator quality and the inference of high-confidence labels even in the absence of ground truth or pre-validation steps.
Generalizability: The framework demonstrates adaptability across diverse modalities (video, image) and architectures (LLMs, vision models) without task-specific calibration.

The authors conclude that AURA is particularly valuable in scenarios where annotator quality is unknown or heterogeneous, allowing for the utilization of a mix of costly, high-quality agents and cheap, low-quality alternatives without compromising overall annotation quality. The paper notes ongoing work to incorporate heterogeneous costs to design a framework with cost-performance trade-offs.