WORKSWORLD: A Domain for Integrated Numeric Planning and Scheduling of Distributed Pipelined Workflows

This paper introduces WORKSWORLD, a new domain for numeric planners that automates the joint planning and scheduling of distributed data pipelines by allowing users to define high-level goals without specifying the entire workflow graph, demonstrating the ability to solve complex multi-site problems on commodity hardware.

The Gist

Imagine you are the Grand Conductor of a massive, global orchestra.

Your orchestra isn't made of musicians, but of computers, servers, and data centers scattered all over the world (some in big clouds, some in small foggy neighborhoods, and some right on the edge of town). Your job is to take raw ingredients (data) from various farms, cook them into delicious meals (processed information), and serve them to hungry customers (applications) in specific formats, all while keeping the bill low and the wait time short.

The paper you shared, "WORKSWORLD," is about building a super-smart AI Conductor that can figure out the perfect recipe for this orchestra, even when the kitchen is huge and the orders are complicated.

Here is the breakdown of the problem and the solution, using simple analogies:

The Problem: The "Too Many Choices" Dilemma

In the real world, companies spend billions on data, but often fail to get value because they can't figure out how to move data efficiently.

• The Old Way: Usually, a human engineer has to draw a map of exactly how every step should happen (e.g., "Take data from Site A, send it to Site B, compress it, then send to Site C"). This is like asking a chef to write down every single step of a recipe before they even know what ingredients are available. It's rigid, hard to change, and often leads to traffic jams (latency) or huge bills (cost).
• The Challenge: You have to decide:
  1. Where to cook the data (Cloud? Edge? Fog?).
  2. How to move it (Direct highway? A two-hop detour?).
  3. When to do it (Does it need to be instant, or can it wait?).
  4. What format it needs to be in.

Doing this math for a complex network is like trying to solve a Rubik's Cube while blindfolded. It's computationally impossible for humans to do perfectly.

The Solution: WORKSWORLD (The "Magic Recipe Book")

The authors created a new system called WORKSWORLD. Think of it as a universal translator and a master chef rolled into one.

1. The Input (The YAML Config):
   Instead of drawing a complex map, the human engineer just writes a simple shopping list in a file called YAML.

   • Analogy: You tell the AI: "I have apples at the farm (Source). I need apple pie at the bakery (Destination). I have ovens in three different cities. I want it done in under 10 minutes and under $50."
   • You don't tell the AI how to do it. You just tell it what you want.

2. The Translator (YAML to PDDL):
   The system takes your simple shopping list and translates it into a strict, mathematical language called PDDL (Planning Domain Definition Language). This is the language the AI Conductor speaks.

3. The Brain (The Numeric Planner):
   The AI (specifically a tool called ENHSP) looks at the "shopping list" and the "map of the world" (the resources). It runs a massive simulation to find the best path.

   • It asks: "Should I bake the pie at the farm and ship it? Or ship the apples to the city and bake there?"
   • It calculates: "If I bake at the farm, the shipping cost is low, but the oven is slow. If I bake in the city, the oven is fast, but shipping the apples costs more."
   • It finds the perfect balance between cost and speed.

4. The Output (The Plan):
   The AI produces a step-by-step instruction manual: "Go to Farm A, pick up apples. Send them to City B. Bake them. Send the pie to the Bakery." It builds the "pipeline" automatically.

Why is this a Big Deal? (The "Aha!" Moment)

Most existing tools are like specialized chefs who only know how to cook one specific dish on one specific stove. If you change the stove or the dish, they break.

WORKSWORLD is different because:

• It's Vendor-Agnostic: It doesn't care if you use Amazon, Google, or a local server farm. It just sees "Resources."
• It's "Numeric": It understands numbers like "Bandwidth," "Latency," and "Cost" in real-time, not just "Yes/No" logic.
• It Scales: The authors tested it on a computer you could buy at a store (commodity hardware). They successfully planned a pipeline with 14 steps across 8 different locations in just one hour.

The Real-World Examples

The paper gives three scenarios to show why this matters:

1. The Archivist (Slow & Cheap): You have old videos. You can compress them at the source (saving transfer costs) or send them raw to the cloud to compress (saving local storage). The AI picks the cheapest option.
2. The Wildfire Sensor (Fast & Critical): Sensors see smoke. They need to process the image immediately to trigger an alarm. The AI decides to process the data right at the sensor (Edge) because sending it to the cloud takes too long.
3. The Cybersecurity Guard (Instant & Strict): A hacker tries to break in. The system must block them in a split second. The AI places the security check right next to the firewall, even if it costs a bit more, because speed is the only thing that matters.

The Catch (Limitations)

The AI is smart, but it's not magic yet.

• The "Pre-Processing" Bottleneck: Before the AI starts solving the puzzle, it has to read the entire map of the world. If the map gets too huge (too many sites and links), this reading step takes a long time. It's like trying to read a whole encyclopedia before you can answer a single question.
• Linear Chains: Right now, the AI is best at "assembly lines" (Step A -> Step B -> Step C). It struggles a bit with complex, branching webs where data splits and merges in crazy ways.

The Bottom Line

WORKSWORLD is a new tool that lets companies say, "Here is my data and my goal," and lets an AI figure out the most efficient, cost-effective way to move and process that data across the globe. It turns a nightmare of complex math into a simple "set it and forget it" workflow, making AI and data pipelines accessible to everyone, not just super-computer experts.

Technical Summary

1. Problem Statement

The paper addresses the challenge of automated planning and scheduling for distributed data pipelines (workflows) that must run permanently across heterogeneous environments (Cloud, Fog, and Edge).

• The Core Challenge: Organizations struggle to derive ROI from AI and data platforms because they lack efficient mechanisms to move, process, and format data across distributed sites. Existing solutions often require a pre-defined workflow graph (DAG) and focus solely on scheduling tasks to resources.
• The Gap: There is a lack of systems that can jointly plan (determine the workflow structure, data conversions, and routing) and schedule (map components to specific compute/storage/network resources) while optimizing for multiple objectives (cost, latency, resource utilization).
• Specific Constraints:
  • Permanence: Unlike batch jobs, these pipelines are long-running (ingest pipelines).
  • Complexity: Data sources and sinks span multiple networks; consumers may require the same data in different formats.
  • Trade-offs: Decisions involve trade-offs between processing data at the source (saving transfer costs) vs. centrally (saving compute costs), and balancing strict latency requirements against cost.
  • Computational Hardness: The problem is structurally identical to the NP-hard Virtual Network Embedding problem.

2. Methodology

The authors propose a framework that converts a high-level YAML configuration into a Planning Domain Definition Language (PDDL) problem, solvable by state-of-the-art numeric planners.

A. The Framework (Figure 1)

1. Input: Users define the environment, available components, and goals (output formats/locations) via a YAML file.
2. Conversion: A Python/Jinja2 template engine converts the YAML into a PDDL 2.1 Level 2 problem instance.
3. Solving: An existing numeric planner (specifically ENHSP) solves the problem.
4. Output: A validated plan is generated and visualized, showing the constructed workflow graph and its mapping to the resource graph.

B. The WORKSWORLD Domain

The authors introduce a new PDDL domain called WORKSWORLD designed for numeric planning.

• Abstractions:
  • Sites: Cloud, Fog, and Edge locations with specific compute, storage, and network resources.
  • Interfaces: Two types of network-level resources:
    • Data Sharing Interfaces (DSI): For storage/streaming (e.g., Kafka, CEPH).
    • Data Processing Interfaces (DPI): For compute (e.g., Kubernetes, Slurm).
  • Workflow Components: Logical units of work categorized as Data Components (sources/sinks) or Processing Components (transformations).
• Key Actions: The domain defines six primary actions to build and schedule the workflow:
  • replicate_code: Copies component configurations (e.g., container images) to a site.
  • schedule_component: Assigns a component to a specific interface.
  • connect_direct_link / connect_composite_link: Establishes network paths between components.
  • propagate_input / propagate_output: Tracks data flow and provenance.
• Optimization Metrics:
  • Cost (Minimize): Weighted resource utilization (compute, storage, bandwidth) normalized by total capacity.
  • Latency (Constrain): "Absolute Latency" defined as the sum of traversal times for a single message across the workflow edges.

C. Planning Approach

• Joint Planning & Scheduling: The planner does not receive a fixed DAG. Instead, it must construct the DAG (deciding which conversions to apply and where to place them) while simultaneously assigning them to resources.
• Numeric Fluents: The domain uses numeric variables to track resource availability, bandwidth consumption, and accumulated costs/latency, allowing for realistic modeling of constraints.
• Solver: The authors utilize ENHSP (Expressive Numeric Heuristic Search Planner), chosen for its support of quantifiers, large problem files, and the Additive Interval-Based Relaxation (AIBR) preprocessing technique to prune the search space.

3. Key Contributions

1. WORKSWORLD Domain: A novel, open-source PDDL domain for the integrated planning and scheduling of permanent, distributed data pipelines. It decouples compute, storage, and network decisions.
2. Unified Framework: A toolchain (YAML $\to$ PDDL $\to$ Plan $\to$ Visualization) that allows data engineers to define goals without manually constructing complex workflow graphs.
3. Formal Problem Definition: A rigorous definition of pipelined workflow scheduling across related-heterogeneous resources, distinguishing it from classical batch scheduling by focusing on permanent mappings and network constraints.
4. Public Benchmarks: The release of the domain, benchmarks, and experimental data to facilitate future research in AI planning for data engineering.

4. Experimental Results

The authors evaluated the system using ENHSP on commodity hardware (CloudLab nodes) with a 1-hour CPU time limit and 30 GB RAM.

• Scalability (RQ1):
  • The system successfully solved linear-chain workflows with up to 14 components (7 processing + 7 data) distributed across 8 sites.
  • State Space Growth: While the raw state space grows exponentially with the number of components and sites, the AIBR preprocessing effectively prunes the space, allowing the solver to handle mid-sized enterprise scenarios.
  • Bottleneck: The primary scalability bottleneck identified is the AIBR preprocessing step, which grows exponentially with the number of interfaces and sites.
• Feasibility (RQ2):
  • The system demonstrated viability for centralizing the planning of workflows for a mid-sized organization (2 Cloud, 2 Fog, 4 Edge sites).
  • It successfully handled complex resource graphs (8 sites, 16 interfaces, 15 direct links, 14 composite links).
  • Failure Point: The system failed to solve workflows with 16 components within the time/memory limits, indicating a ceiling for current numeric planners on this specific domain formulation.

5. Significance and Future Work

• Significance: This work bridges the gap between AI Planning and Data Engineering. It moves beyond proprietary, platform-specific schedulers to a vendor-agnostic, declarative approach. It proves that general-purpose numeric planners can handle complex, real-world data pipeline problems that were previously considered computationally intractable for automated solvers.
• Implications: It enables organizations to automatically optimize data pipelines for cost and latency without manual intervention, a critical step for the effective deployment of AI tools.
• Future Directions:
  • State-Dependent Costs: Moving from static max-rate calculations to dynamic, state-dependent resource consumption models.
  • Non-Linear Workflows: Extending the domain to support series-parallel (non-linear) workflows, not just linear chains.
  • Hierarchical Planning: Exploring a hierarchy of planners to collaboratively solve scheduling for larger, more complex networks.
  • Solver Optimization: Investigating other planners (e.g., SMTPLAN+, GOOSE) and optimizing the AIBR preprocessing step to handle larger problem sizes.

In summary, WORKSWORLD provides a foundational framework for automating the design and deployment of distributed data pipelines, demonstrating that modern numeric planners can effectively solve joint planning and scheduling problems in realistic, multi-site environments.