Agentic Control Center for Data Product Optimization

This paper proposes an agentic control center that automates the continuous optimization of data products by employing specialized AI agents to generate supporting assets, monitor quality metrics, and integrate human-in-the-loop oversight to balance automation with trust.

The Gist

Imagine you have a massive, chaotic warehouse filled with boxes of data. Right now, it's just a storage room. You know the boxes are there, but if you ask a worker, "Where is the box about last year's sales?" they might shrug, or worse, they might pull out the wrong box.

The Problem:
To make this warehouse useful, you need to turn it into a Data Product. This means organizing the boxes, labeling them clearly, and creating a catalog of questions people can ask (like "Show me sales by region") along with the answers. Traditionally, you'd need a team of expert librarians (data engineers) to manually sort every box, write every label, and craft every answer. This is slow, expensive, and hard to scale.

The Solution: The "Agentic Control Center"
This paper introduces a new system that acts like a self-driving, self-optimizing warehouse manager. Instead of hiring a human to do the sorting forever, you hire a team of specialized AI robots (agents) that work together to constantly improve the warehouse.

Here is how it works, using simple analogies:

1. The Goal: The "Quality Contract"

Before the robots start working, you set a Quality Contract. Think of this like a checklist for a perfect warehouse:

• "We need 90% of the boxes to be labeled."
• "Answers must be found in under 5 seconds."
• "We need to be able to answer questions about every department."

The system doesn't just guess; it constantly measures itself against this checklist.

2. The Team of Robots (The Agents)

The system isn't just one big brain; it's a team of specialists, each with a specific job:

• The Planner (The General): This is the boss. It looks at the checklist, sees what's missing (e.g., "We have no labels for the 'Marketing' section"), and decides, "Okay, we need to generate 50 new questions about marketing."
• The Input Planner (The Tuner): This robot figures out how to do the job. If the warehouse is huge, it says, "Generate 80 questions." If it's small, it says, "Just 20." It adjusts the volume so the team doesn't get overwhelmed.
• The Specialists (The Workers):
  • Question Generator: Invents new questions people might ask.
  • SQL Translator: Turns those questions into the actual code (SQL) the database understands.
  • View Creator: Builds custom "windows" or dashboards so users can see data without getting lost.
  • Clustering Agent: Groups similar questions together (like putting all "Sales" questions in one folder) so the catalog isn't a mess.

3. The Loop: The "Try, Measure, Fix" Cycle

The magic happens in a continuous loop, like a video game where you keep leveling up:

1. Plan: The General spots a gap (e.g., "Column coverage is too low").
2. Act: The Specialists go to work, creating new labels and answers.
3. Measure: The system immediately checks the checklist again. Did the new labels help? Did the answers get faster?
4. Repeat: If the score is still low, the General picks a new task. If the score is high, it moves to the next problem.

4. The Human Safety Net (Human-in-the-Loop)

You might worry, "What if the robots go crazy and delete everything?"
That's why this system has a Control Center. It's like a glass-walled control room where humans can watch the robots work.

• Observability: You can see exactly what the robots are doing and why.
• Intervention: If a robot makes a weird choice, a human can step in, say "Stop," and fix it.
• Trust: Because everything is recorded (like a video replay in a Git repository), you know exactly who changed what and when.

5. The Result: A Living Knowledge Base

In the paper's test case, they used this system on three different "warehouses" (databases).

• On a small warehouse, the robots fixed it in minutes.
• On a huge, complex warehouse, the robots got smart. They realized, "We can't just ask simple questions; we need to ask complex ones that link different tables together." They automatically adjusted their strategy to handle the complexity.

In Summary:
This paper describes a system that turns a static, messy pile of data into a living, breathing product that gets smarter every day. It uses a team of AI agents to constantly ask, "How can we make this data more useful?" and then does the work to fix it, all while keeping a human in the driver's seat to ensure safety and trust.

It's the difference between having a library where books are thrown on the floor, and having a library where a team of robots constantly organizes the shelves, writes new summaries, and ensures you can find the book you need in seconds.

Technical Summary

1. Problem Statement

The proliferation of data within organizations has created a challenge: raw data is often difficult to discover, understand, and utilize effectively. While "data products" (reusable packages of data assets like views, query examples, and metadata) can bridge this gap, creating them is traditionally a manual, costly, and slow process reliant on domain experts.

Furthermore, emerging AI solutions using Large Language Models (LLMs) for data tasks often operate as "black boxes," lacking:

• Observability: Users cannot easily see how agents make decisions.
• Control: There is limited ability for human intervention or oversight.
• Trust: Subjective quality assessment and lack of continuous optimization mechanisms make it difficult to guarantee data product reliability.

The core problem is the need for a system that automates data product improvement while balancing high-level automation with human-in-the-loop (HITL) control, transparency, and measurable quality guarantees.

2. Methodology: The Agentic Control Center

The authors propose a framework called the Agentic Control Center, which operates on a continuous optimization loop. The system uses specialized AI agents to iteratively improve data products against user-defined "quality contracts."

A. System Architecture

The architecture is modular, consisting of four core components:

1. State Manager: Acts as the single source of truth, maintaining the DataProductState. This includes schema metadata, question-to-schema mappings, query versioning, answer confidence scores, and view definitions. It supports diverse databases (SQLite, MySQL, PostgreSQL, BigQuery) via a flexible abstraction layer.
2. Data Product Quality Metrics Module: A configurable engine that defines and tracks metrics such as table/column coverage, query complexity, execution speed, and response accuracy. It uses a dependency graph to trigger event-driven recalculation of metrics when the state changes (e.g., adding a new table).
3. Tool Registry: A flexible interface for registering external tools (agents or services). Tools are mapped to specific quality metrics (e.g., "View Creation" improves coverage but reduces complexity).
4. Agentic Orchestration Layer: The brain of the system that coordinates autonomous workflows.

B. The Agent Workflow

The optimization process follows a Plan → Parameterize → Execute → Update → Measure cycle:

• Planner Agent: The central decision-maker. It evaluates the current state against quality contracts, identifies gaps (e.g., low table coverage), and selects the highest-impact action using a tool precondition graph. It outputs a structured JSON payload with the rationale.
• Input Planner Agent: Translates high-level actions into precise tool parameters. It uses adaptive heuristics to scale generation (e.g., generating 80 questions for a large, under-covered database vs. 20 for a small one) to maximize efficiency.
• Specialized Agents: Modular agents execute specific tasks:
  • Question Generation: Creates novel queries.
  • Text-to-SQL: Generates executable SQL and query plans.
  • Follow-up Generation: Creates chained questions to enrich exploration.
  • Question Clustering: Groups queries into topics for better organization.
  • View Creation: Generates SQL views to optimize performance.
• Recalculation & Versioning: After execution, metrics are recalculated. All artifacts (SQL, views) are committed to an integrated Git repository for auditability and version control.

3. Key Contributions

1. Autonomous Data Product Improvement: Introduction of a framework that uses measurable quality contracts and optimization objectives to drive continuous, autonomous refinement of data products.
2. Multi-Agent Collaboration: Demonstration of a specialized multi-agent system where distinct agents (Planner, Input Planner, and Specialized Agents) collaborate to handle complex data tasks, separating strategic planning from tactical execution.
3. Human-in-the-Loop (HITL) Mechanisms: A design that prioritizes observability and control, allowing users to set targets, monitor metrics, and intervene, thereby addressing trust and safety concerns in agentic systems.

4. Results (Case Study)

The system was evaluated using three databases from the BIRD benchmark with specific quality targets (90% table coverage, 50% column coverage, <5s query execution).

• Adaptive Strategies: The system demonstrated context-aware optimization.
  • In small databases, it converged rapidly, quickly identifying and filling coverage gaps.
  • In complex databases, it employed sophisticated strategies, such as automatically increasing question generation volume and selecting tools for multi-stage question generation to handle complex joins and subqueries.
• Intelligent Parameterization: The Input Planner Agent successfully adjusted generation scales based on the state of the system (e.g., prioritizing large, unexercised tables).
• Diminishing Returns Detection: The Planner Agent showed meta-reasoning capabilities by detecting when further autonomous actions yielded minimal improvements, appropriately recommending manual review to avoid inefficient iterations.
• Organization & Auditability: The Question Clustering tool successfully grouped hundreds of generated queries into coherent topics. The Git integration provided a full audit trail of all autonomous decisions and modifications.

5. Significance

This work represents a significant step toward production-ready agentic data systems. By moving beyond simple query generation to a holistic optimization loop, the Agentic Control Center addresses critical gaps in current AI data tools:

• Scalability: It reduces the reliance on manual engineering for creating data assets.
• Trust & Safety: The explicit separation of planning, parameterization, and execution, combined with HITL controls and versioning, mitigates the risks of "black box" AI.
• Continuous Improvement: Unlike static data products, this system treats data assets as "living" interfaces that evolve and improve based on real-time quality metrics and user feedback.

The paper concludes that while this is a prototype, it lays the groundwork for a comprehensive framework capable of solving complex multi-objective optimization problems in data engineering.