🤖 machine learning

CatalogAgent: A Supervisor-mediated Self-Learning System Enabling Context Engineering for GenAI Models

The paper introduces CatalogAgent, a supervisor-mediated self-learning system that resolves conflicts between generator and evaluator LLMs for e-commerce catalog enrichment by leveraging a memory base to aggregate supervisor decisions into context-engineered insights, thereby autonomously improving model accuracy by over 13%.

Original authors: Zhu Cheng (Xuan), Zhenming Wang (Xuan), Yu (Xuan), Tang, Dan Liu, Bryan Zhang, Athanasios N. Nikolakopoulos, Pranav Souri Itabada, Jing Zhang, Chih-Chi Chou, Peng Gao, Fatemeh Mansoori, Bharat Bojja

Published 2026-07-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhu Cheng (Xuan), Zhenming Wang (Xuan), Yu (Xuan), Tang, Dan Liu, Bryan Zhang, Athanasios N. Nikolakopoulos, Pranav Souri Itabada, Jing Zhang, Chih-Chi Chou, Peng Gao, Fatemeh Mansoori, Bharat Bojja, Sarath Chander, Sameer Thombare, Umit Batur, Tarik Arici

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where your favorite online store is run by a team of very smart, but slightly scattered, robot assistants. These robots are tasked with organizing millions of products, from t-shirts to toaster ovens, into neat digital shelves. To do this, they need to fill in specific details like "color," "material," or "size." This is the job of Generative AI (or GenAI), a type of computer program that can read text and images to create new information. But here's the catch: these robots aren't perfect. Sometimes they guess wrong, and sometimes they argue with each other about what the right answer is. If the store keeps these mistakes, customers get confused, and sellers get frustrated. The big question for scientists in this field is: How do we teach these AI robots to spot their own errors and get better at their jobs without needing a human to hold their hand every single time?

Enter CatalogAgent, a new system designed by researchers at Amazon that acts like a super-smart "boss robot" to solve this exact problem. Think of the system as a three-person team working in a busy warehouse. First, you have the Generator, a robot that looks at a product's title and description and guesses what its attributes are (like guessing a shirt is "cotton"). Second, there's the Evaluator, a skeptical robot that double-checks the Generator's guess to see if it makes sense. Usually, they agree, and the job is done. But when they disagree, or when a human seller says, "Hey, that's wrong!", a third robot steps in: the Supervisor.

The Supervisor is the star of the show. It's like a detective with a magnifying glass and a library of tools. When the Generator and Evaluator fight, or when a seller complains, the Supervisor investigates. It doesn't just pick a winner; it figures out why the mistake happened. Did the Generator hallucinate a fake brand name? Did the Evaluator get too strict? Did the seller confuse "fit type" with "inseam length"? The Supervisor writes down the lesson learned from every single fight.

But here is the magic trick: the system doesn't just file these lessons away. It has a Memory Summarizer, a clever librarian that reads thousands of these detective reports and finds patterns. Maybe it notices that for "Cell Phone Cases," the robots keep making up model names, or that for "Handbags," they always need to look at the second photo to see the inside. The Summarizer turns these patterns into new instructions, or "context engineering," and feeds them back to the Generator and Evaluator. It's like the robots are reading a study guide written by their own boss, helping them learn from their past mistakes so they don't make them again.

The paper shows that this self-learning loop works incredibly well. By using this method, the system improved the Generator's accuracy by 15.24% and the Evaluator's accuracy by 13.98%. However, the researchers were careful not to let the robots get too confident. They built a safety net called Regression Constraints. This is like a quality control check that ensures the robots don't get so focused on fixing their new mistakes that they start messing up the easy ones they already knew how to do. They tested this on 4.89 million product pairs and found that while the system aggressively fixed difficult cases, it kept its performance on the general population stable.

In short, CatalogAgent proves that AI systems can learn to police themselves. By having a smart supervisor mediate conflicts and turning those conflicts into teachable moments, the system creates a cycle where the robots get smarter every day, all without a human needing to rewrite their code. It's a shift from "fixing the robot" to "teaching the robot how to fix itself."

Technical Summary: CatalogAgent

Problem Statement

Product catalogs are fundamental to e-commerce, yet they frequently suffer from missing Structured Attributes (SAs) such as material, color, and shape. While Large Language Models (LLMs) have been adopted to predict these values from unstructured product data (titles, descriptions, images), existing generator-evaluator frameworks face significant limitations. Specifically, when the Generator (predictor) and Evaluator (verifier) produce conflicting outputs, it is unclear which component is at fault. Furthermore, standard LLM-as-a-judge paradigms struggle to identify high-level error patterns across diverse product types (PTs) and attribute groups without human intervention. Current production-scale requirements also demand lightweight, cost-effective models, making systematic context engineering essential to maximize performance without sacrificing scalability.

Methodology: CatalogAgent System

The authors propose CatalogAgent, a novel supervisor-mediated self-learning system designed to continuously improve Generator and Evaluator models through systematic context engineering. The system operates as a closed-loop cycle comprising four core components:

1. Worker Models (Generator and Evaluator)

The system employs lightweight LLMs (specifically Mistral-NeMo) for high-volume batch processing:

Generator: Infers specific SA values from product titles, descriptions, and structured data.
Evaluator: Verifies the accuracy of generated values based on available evidence.
Conflict Detection: The system flags cases where the Generator and Evaluator disagree, or where seller feedback contradicts the model-approved values.

2. Supervisor Agent (Investigator and Arbitrator)

A high-reasoning agent (powered by Claude Sonnet) intervenes when conflicts arise. It acts as an arbitrator and investigator using the ReAct methodology (Reasoning + Acting) with a comprehensive toolbox:

Arbitration: It analyzes reasoning chains from both the Generator and Evaluator to determine the correct attribute value.
Conflict Resolution: It resolves Generator-Evaluator disagreements and Model-Seller disagreements.
Tool Usage: It utilizes data access tools (catalog metadata, attribute schemas), domain-specific validators (e.g., size charts), and image analysis tools to verify visual attributes.
Output: It generates granular insights, including root cause diagnoses and specific recommendations for prompt or metadata refinement.

3. Memory Base and Memory Summarizer

To enable self-improvement without human intervention, the system aggregates individual case learnings:

Memory Base: A structured repository storing Supervisor Agent activities for individual cases, including case identification, arbitration results, and specific engineering insights (Prompt Improvements, Metadata Refinement, Image Usefulness Analysis).
Memory Summarizer: Processes records from the Memory Base to aggregate fine-grained insights into generalizable learnings at the Product-Type-Attribute (PT-A) level. It synthesizes common failure patterns into consolidated prompt instructions, attribute definition updates, and image input configurations.

4. Context Engineering under Regression Constraints

The system injects learnings back into the worker models via three types of context engineering:

Prompt Optimization: Refining instructions to improve clarity and reduce hallucinations (e.g., instructing the model to return "Not Obtainable" instead of fabricating a model name).
Metadata Refinement: Expanding value sets and clarifying ambiguous definitions (e.g., adding "button down shirt" to the valid values for shirt form type).
Image Usefulness Determination: Identifying whether main images or secondary images are predictive for specific attributes, optimizing input modalities to reduce token costs.

Crucially, these updates are governed by a constrained optimization problem. New configurations ( $\theta$ ) must minimize disagreement rates while ensuring that accuracy on a "Golden Dataset" ( $G$ ) does not degrade below a baseline ( $\theta_0$ ) by more than a tolerance threshold ( $\delta$ ). This prevents the system from overfitting to edge cases at the expense of general performance.

Key Contributions

The paper outlines three primary contributions:

A New Paradigm: Introduction of a supervisor-agent-mediated self-learning system that autonomously monitors operations and enhances attribute prediction quality through systematic context engineering without human intervention.
Novel Modules: The design of the Memory Base and Memory Summarizer, which create a self-improving feedback loop by capturing agent interactions and aggregating them into actionable insights for worker models.
Targeted Improvement Strategy: A unique approach to model improvement that mines difficult disagreement cases (between Generator/Evaluator and Model/Seller) to identify specific areas requiring context engineering, rather than applying broad, untargeted optimizations.

Experimental Results

The system was evaluated on a dataset of 4.89 million Product-Attribute pairs, with 50k pairs used for Supervisor investigation and 50k for testing the efficacy of the resulting context engineering.

Supervisor Performance: In human evaluations, the Supervisor Agent achieved 97.26% accuracy in generating SA values (acting as a Generator) and 94.56% accuracy in assessing correctness (acting as an Evaluator) for Generator-Evaluator disagreements. For Model-Seller disagreements, it achieved 85.65% precision and 94.28% recall.
Model Improvement: Through context engineering, the system successfully transferred the Supervisor's capabilities to the worker models:
- Generator Performance: Improved by 15.24%.
- Evaluator Performance: Improved by 13.98%.
Specific Gains:
- Metadata Updates: Expanding value sets (e.g., for "shirt form type") yielded accuracy improvements of up to 5%.
- Prompt Updates: Addressing hallucination issues (e.g., for "model name" in phone cases) led to a 15.01% improvement in accuracy for specific attributes.
- Regression Testing: The constrained optimization framework successfully validated that targeted improvements did not degrade performance on the general population, with most PT-As passing regression tests on the golden dataset.

Significance and Claims

The authors claim that CatalogAgent demonstrates a new paradigm for improving generative AI model accuracy in production environments. By leveraging a supervisor agent to mediate conflicts and a memory system to aggregate learnings, the system achieves autonomous self-improvement. The significance lies in the ability to:

Systematically address the "black box" nature of LLM disagreements by diagnosing root causes.
Transfer high-level reasoning capabilities from a powerful Supervisor to lightweight, cost-effective worker models via context engineering.
Maintain production stability through regression constraints, ensuring that optimizations for difficult edge cases do not compromise performance on well-understood cases.

The paper concludes that this approach offers a scalable, cost-effective alternative to human evaluation and manual tuning, enabling e-commerce catalogs to continuously evolve and enrich product data with minimal human intervention. Future work suggested by the authors includes using the high-quality labels generated by the Supervisor to fine-tune the worker LLMs directly.