💻 computer science

Declarative Data Services: Structured Agentic Discovery for Composing Data Systems

This paper introduces Declarative Data Services (DDS), a structured architecture that overcomes the convergence failures of unbounded agentic discovery in heterogeneous data systems by decomposing the search space into bounded, typed layers guided by declarative user intent and iterative runtime feedback.

Original authors: Shanshan Ye, Duo Lu

Published 2026-05-21

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Shanshan Ye, Duo Lu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Wild West" of AI Coding

Imagine you ask a very smart, well-read architect (an AI agent) to build a house. You say, "I want a 3-bedroom house with a solar roof, a pool, and a budget of $300,000."

In the past, if you asked an AI to write code for a single app, it usually did a great job. But when you ask it to build a complex data system (like a trading platform that needs to handle thousands of stock trades per second, store years of history, and stay fast), things go wrong.

Why? Because the AI is like a genius who has read every book in the library but has never actually built a house. It knows what a "Kafka" queue or a "ClickHouse" database is, but it doesn't know how they fit together in the real world. It might pick the wrong materials, forget to connect the pipes, or choose a design that collapses under pressure.

If you just tell the AI, "Fix it," and let it guess again and again (a process called "unbounded discovery"), it often spins in circles. It tries to fix the plumbing by changing the roof, or it keeps rebuilding the whole house from scratch every time it makes a small mistake. It never converges on a working solution.

The Solution: DDS (The "Structured Architect")

The authors propose a new system called Declarative Data Services (DDS). Instead of letting the AI wander aimlessly, DDS acts like a strict, organized construction manager who forces the AI to follow a specific, four-step blueprint.

Think of DDS as a four-story building where every floor has a specific job and a specific set of rules. The AI (the "sub-agent") does the work, but the Framework (the "manager") holds the blueprints and checks the work at every step.

The Four Floors of the DDS Building

Floor 1: The "What" (Intent)

The Job: Before building, you must define exactly what you want.
The Analogy: Instead of saying "I want a cool house," the user says, "I need 3 bedrooms, 2 bathrooms, solar panels, and a $300k budget."
The Magic: The system forces the AI to translate your vague wish into a strict, typed checklist (Data Model, Speed, Cost, etc.). If the request is impossible (e.g., "I want a mansion for $500"), the system stops you before any code is written.

Floor 2: The "How" (The Blueprint)

The Job: Design the flow of data without picking specific brands yet.
The Analogy: The architect draws a diagram: "Water comes in, goes to the filter, then to the tank, then to the shower." They don't say "We will use a Whirlpool filter" yet. They just decide the shape of the system.
The Magic: This separates the design from the products. If the design is wrong, you fix the drawing, not the specific brand of pipes. This prevents the AI from changing the whole house just because one pipe leaked.

Floor 3: The "Which" (The Skills)

The Job: Pick the actual products and configurations.
The Analogy: Now the architect picks the specific items: "We will use a Whirlpool filter and PVC pipes." But here's the secret sauce: The system uses a "Skill Book" (a persistent memory).
The Magic: This Skill Book is like a living manual. If a previous house had a problem because the Whirlpool filter needed a specific adapter, that fact is written in the Skill Book. The next time the AI builds a house, it reads the Skill Book and knows exactly which adapter to use. It doesn't have to "guess" or "re-learn" this mistake.

Floor 4: The "Check" (Runtime Attribution)

The Job: Build it, turn it on, and see what breaks.
The Analogy: You turn on the water. If the shower leaks, the system doesn't just say "The house is broken." It says, "The leak is in the PVC pipe connection (Floor 3)."
The Magic: This is the most important part. When the system fails, it traces the error back to the exact floor where the decision was made.
- If the budget is too low, it goes back to Floor 1.
- If the design is impossible, it goes back to Floor 2.
- If the specific product choice was bad, it updates the Skill Book on Floor 3.
- It does not make the AI guess the whole house again. It makes a tiny, targeted fix.

Why This Works (The "Aha!" Moment)

The paper tested this against "unbounded" AI agents (those that just guess and retry).

The Unbounded Agent: Like a student trying to solve a math problem by writing down random numbers until they get it right. It takes a long time, costs a lot of money (computing power), and often gives up.
The DDS Agent: Like a student using a textbook and a calculator. If they get a step wrong, the textbook tells them exactly which rule they broke. They fix that one rule and move on.

The Results:
In a test building a trading backend:

Unbounded Agents: Failed to build a working system 8 out of 10 times, even after many attempts. They got stuck in loops.
DDS: Built a working system 10 out of 10 times. It was faster, cheaper, and the system actually ran with real data.

The "Skill Book" Analogy

The paper emphasizes that knowledge needs a home.

In old AI methods, if the AI learned that "Kafka needs port 9092," it might forget that tomorrow. It has to re-learn it every time.
In DDS, that knowledge is saved in a Skill File (like a YAML file).
If the AI makes a mistake (e.g., "I forgot to open port 9092"), the system fixes the Skill File.
Crucially: The next time the AI builds a system, it reads the fixed Skill File. The mistake is never made again. The "fix" is permanent, not just a one-time patch.

Summary

The paper argues that to build complex data systems with AI, we cannot just let the AI "freestyle." We need a structured framework that:

Breaks the problem down into clear, typed steps (Intent -> Design -> Product -> Code).
Forces the AI to check its work at every step before moving on.
Remembers mistakes in a permanent "Skill Book" so they aren't repeated.
Pinpoints errors exactly so the AI doesn't have to guess what to fix.

It turns the chaotic process of "AI guessing" into a reliable, industrial assembly line for building software.

Technical Summary: Declarative Data Services (DDS)

Problem Statement

The paper addresses the challenge of agentic discovery for multi-system data backends. While Large Language Model (LLM) driven agents have shown success in discovering novel algorithms or optimizing single-system configurations (e.g., self-driving databases), they struggle when tasked with composing heterogeneous data stacks (e.g., combining queues, OLAP/OLTP stores, caches, and connectors) from natural language user intent.

The core difficulties identified are:

Heterogeneous Search Space: The solution space spans multiple distinct systems with different dialects, connectors, and operational quirks.
Verification Complexity: The "verifier" is not a benchmark answer key but whether a deployed stack actually runs and meets Service Level Objectives (SLOs).
Uneven Knowledge Capture: Composition knowledge (what works together, what anti-patterns to avoid) is sparse in pretraining data and changes with every release cycle.
Failure of Unbounded Discovery: Iterative coding agents that rely on failure-log feedback (unbounded discovery) fail to converge consistently. They often regress earlier successful tiers by rewriting architecture in response to downstream errors or get stuck in loops guessing at root causes without tracing signals to specific decision layers.

Methodology: The DDS Framework

The authors propose Declarative Data Services (DDS), a structured agentic discovery architecture that decomposes the global search problem into four bounded, typed sub-searches. The framework enforces a strict split between Framework (which owns contracts and validation) and Sub-Agents (which perform bounded search within those contracts).

The architecture consists of four layers (L1–L4), each carrying a typed contract:

L1: Intent Contract:
- Goal: Translate ambiguous natural language into a validated, typed declaration.
- Mechanism: A sub-agent elicits a draft intent, which the framework validates against a six-dimensional schema: data model, access pattern, scale, latency, consistency, and cost.
- Constraint: The framework rejects malformed inputs before any code generation occurs.
L2: Logical Operator DAG:
- Goal: Determine the topology (the "what" and "how" of data flow) independent of specific products.
- Mechanism: The framework defines a typed operator graph (e.g., INGEST → QUEUE → TRANSFORM → STORE).
- Constraint: The framework enforces reachability (every ingest must reach every serve) and composes edge guarantees into end-to-end SLOs. If a topology cannot meet the L1 budget or SLOs, it is rejected before product selection.
L3: Physical Skill Contract:
- Goal: Select specific products and configurations that fit the DAG.
- Mechanism: This layer utilizes Agent Skills, which are persistent, editable, structured artifacts (YAML) per system. Each skill contains four blocks: capabilities, compositions, anti_patterns, and operational.
- Constraint: The framework validates candidates against skill-defined anti-patterns (e.g., forbidden version ranges, incompatible operator pairings). Crucially, composition knowledge is stored here as a reusable artifact rather than transient prompt context.
L4: Runtime Attribution Loop:
- Goal: Deploy the stack and route runtime failures back to the correct layer.
- Mechanism: Upon deployment failure, the framework classifies the signal (e.g., image pull failure, port conflict, consumer lag) and routes it to the layer owning the violated decision.
- Constraint:
  - L4 errors (transient codegen/host issues) are auto-patched by the framework.
  - L3 errors (missing skills/anti-patterns) result in skill patches that are edited into the skill artifact, persisting for future deployments.
  - L1/L2 errors (intent or topology mismatches) are surfaced to the user for revision, as the framework cannot auto-revise these contract boundaries.

Key Contributions

Structured Architecture for Multi-System Discovery: A framework/agent split with four typed contracts (L1–L4) that decomposes the global search into bounded sub-searches, governed by ownership rules where the framework validates contracts and agents search within them.
Structured vs. Unbounded Discovery: Two architectural innovations:
- Typed Attribution: Routing every runtime signal to the specific layer owning the violated decision, preventing global search regression.
- Persistent Skill Memory: Storing composition knowledge in editable, citable skill artifacts (L3) rather than prompts, allowing fixes to accumulate and persist across deployments.
Empirical Validation (Trading Backend): A case study demonstrating that DDS converges to a working stack where unbounded discovery fails, with runtime failures becoming skill patches that improve subsequent iterations.

Results

The paper evaluates DDS against unbounded agentic discovery baselines (Conditions A, B, C) using a trading backend workload (50K ticks/sec, sub-10ms lookups, $300/mo budget) and the same underlying LLM (Claude Opus 4.6).

Convergence:
- Unbounded Baselines: Even with 5 iterations and explicit system names or skill content pasted as prose, unbounded agents failed to reach a working state (T1/T2) consistently. Condition A (NL only) reached T1 in 2/10 runs; Condition C (NL + skill prose) reached T1 in 6/10 runs.
- DDS: With the same 5-iteration loop, DDS reached T1 and T2 in 10/10 runs.
Efficiency:
- DDS required a median of 44 turns and $1.94 to converge.
- Unbounded baselines required significantly more turns (median 209–268) and cost ( $5.06–$ 6.23), often exhausting iterations without convergence.
Ablation Study: Removing the operational and anti_patterns blocks from the skill artifacts caused the success rate to drop to 0/3, demonstrating that specific operational knowledge (e.g., recommended images, port conflicts) is critical and not redundant with pretraining.
Attribution Accuracy: In a controlled fault-injection experiment (20 instances), the L4 attribution loop correctly routed signals to the owning layer 100% of the time.
Learning Loop: In a real failure scenario, four distinct runtime failures were attributed to L3 skill gaps. The framework applied skill-only patches (no code changes), and the second deployment fixed 4/4 failures with inline citations of the patched fields.

Significance and Claims

The paper positions DDS as an early prototype reporting lessons from real-world data-system composition. Its significance lies in demonstrating that structure, not just iteration or knowledge volume, is the key to reliable agentic discovery for complex systems.

The "Win" is Structure: The authors claim that the "typed channel" (the contract surface) is what turns iteration into a layer-bounded edit. Unbounded agents fail because they lack a mechanism to trace symptoms to root-cause decisions; DDS solves this by forcing failures to land on specific layers.
Persistent Knowledge: The paper argues that composition knowledge must be a "first-class, persistent artifact" (the skill) rather than transient context. This allows the system to learn from failures and apply those lessons to future deployments without re-discovering them.
Modest Scope: The authors explicitly state that the prototype runs on local Docker Compose and is a "proof of life," not a production-grade system. They acknowledge open problems such as attribution confidence for ambiguous signals (e.g., consumer lag), intent drift, and the need for an L5 evaluator layer to verify SLOs under sustained load. The work does not claim to solve all data engineering problems but offers a specific architectural pattern for bridging user intent to multi-system deployment.