⚡ electrical engineering

Outcome-as-a-Service for Multi-Objective Dask-Based Earth Observation Processing

This paper introduces an Outcome-as-a-Service framework for Dask-based Earth Observation workflows that utilizes a weighted multi-objective optimization approach to automatically select cluster configurations, achieving significant reductions in execution time, cost, energy consumption, and resource inefficiency compared to traditional fixed-heuristic provisioning strategies.

Original authors: Arthur Lalayan, Gregory Giuliani, Hrachya Astsatryan

Published 2026-07-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Arthur Lalayan, Gregory Giuliani, Hrachya Astsatryan

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a massive photo-editing studio. You have thousands of high-resolution satellite images of the Earth that need to be processed to find things like forest health or crop growth. To do this quickly, you need a team of computers (a "cluster") working together.

In the past, choosing the right team of computers was like guessing. You might say, "I need 100 computers with 8GB of memory each," hoping that would be fast enough without costing too much money or using too much electricity. But often, you'd either waste money on a team that was too big, or waste time waiting on a team that was too small.

This paper introduces a new way to think about this problem called "Outcome-as-a-Service" (OaaS). Instead of asking for specific hardware (like "give me 100 computers"), you simply tell the system what result you want. You might say, "I want this done as fast as possible," or "I want to spend the least amount of money," or "I want to use the least amount of electricity."

Here is how the system works, broken down into simple concepts:

1. The "Crystal Ball" (Surrogate Models)

Before the system can help you, the researchers had to do a lot of homework. They ran thousands of test jobs on real, bare-metal computers (not virtual ones) to see exactly how different combinations of computers performed.

They tested different sizes of data (from small to huge).
They tested different shapes of computer teams (e.g., a few powerful computers vs. many smaller ones).
They tested different speed settings (like putting the computer in "Eco mode" vs. "Turbo mode").

They used this data to train a "Crystal Ball" (a mathematical model). This model can now predict, with high accuracy, how long a job will take, how much it will cost, and how much energy it will use, without actually running the job first.

2. The "Smart Chef" (The Optimization Engine)

Once the Crystal Ball is trained, it acts like a smart chef.

The Ingredients: You give the chef your "recipe" (the type of image processing you need) and your "budget" (your priorities).
The Weights: You tell the chef how much you care about four things:
1. Speed: How fast it finishes.
2. Cost: How much money it costs (based on cloud pricing).
3. Energy: How much electricity it uses.
4. Frugality: How efficiently it uses resources (avoiding waste, like having a computer sit idle or memory sitting empty).

If you tell the chef, "I care 70% about speed and 10% about everything else," the system instantly calculates the perfect team of computers to get that job done. It doesn't just guess; it mathematically finds the best balance based on your specific priorities.

3. The "Menu" (The Service)

The researchers built a simple web service (a digital menu). You send a request, and within a fraction of a second, it gives you a ranked list of the best computer configurations for your needs.

It doesn't just say "Use 100 computers."
It says, "Use 128 small computers running in 'Eco mode' will get your job done in 463 seconds, cost $0.74, and use 35 watt-hours of energy."

The Results: Why It Matters

The researchers tested this system against three common ways people usually set up these computer teams:

The "Biggest is Best" approach: Just use the largest, most powerful cluster available.
The "Cheapest is Best" approach: Use the smallest cluster that might work.
The "Default" approach: Use the standard settings that come with the software.

The new system beat all of them. Depending on what you prioritized, it achieved:

39% faster completion times.
29% less cost.
37% less energy consumption.
20% better efficiency (less waste).

The Catch (What the paper actually says)

It is important to note the boundaries of this study, just like a recipe has specific ingredients:

The Hardware: The "Crystal Ball" was trained on a specific type of computer chip (Intel Broadwell). It works best for that specific hardware.
The Work: It is designed specifically for Earth Observation tasks (processing satellite images) using a specific software tool called Dask.
The Energy: The energy measurements only count the computer chip and the memory (RAM). It does not count the electricity used by the cooling fans, the power supply, or the building lights.

In Summary

This paper presents a tool that stops you from guessing how to set up your computer cluster. Instead of being a technician who has to know the ins and outs of hardware, you become a manager who simply states your goals (Fast? Cheap? Green?). The system then uses a "Crystal Ball" trained on real-world data to instantly tell you the perfect setup to achieve those goals, saving you time, money, and energy.

Technical Summary: Outcome-as-a-Service for Multi-Objective Dask-Based Earth Observation Processing

Problem Statement

Distributed Earth Observation (EO) workflows require significant computational and memory resources, often necessitating the use of frameworks like Dask on cloud or High-Performance Computing (HPC) infrastructures. However, selecting an optimal cluster configuration remains a complex combinatorial challenge. Practitioners typically rely on fixed heuristics (e.g., provisioning the largest available cluster or a minimum feasible one) that fail to balance competing objectives: execution time, monetary cost, energy consumption, and resource-use efficiency (frugality).

The search space for cluster configurations is vast, defined by the number of worker nodes ( $n$ ), node type ( $t$ ), and CPU frequency governors. Expanding computational capability to reduce execution time often increases cost and energy consumption, while energy-saving measures may extend runtime. Furthermore, the performance of EO tasks is highly sensitive to input data characteristics (size, resolution), algorithm complexity, and infrastructure variables such as CPU frequency governors and worker granularity (the ratio of vCPUs to memory per worker). Existing solutions often optimize for only one or two objectives (typically time and cost), target different frameworks (Spark/Hadoop), or fail to expose configuration selection as a deployable service.

Methodology

The authors propose an Outcome-as-a-Service (OaaS) framework that shifts the user interface from explicit resource requests to outcome requests. Users specify their priorities via a weight vector across four objectives, and the system autonomously recommends the optimal cluster configuration.

1. Experimental Infrastructure and Data Collection

The foundation of the framework is a comprehensive historical dataset ( $H$ ) generated through a systematic measurement campaign on homogeneous bare-metal infrastructure (CloudLab xl170 servers with Intel Xeon E5-2640 v4 processors).

Workloads: Three spectral vegetation indices were used as EO operators: Normalised Difference Vegetation Index (NDVI), Soil-Adjusted Vegetation Index (SAVI), and Enhanced Vegetation Index (EVI). These represent varying levels of arithmetic intensity and I/O dependencies.
Configuration Space: The study evaluated 15 cluster configurations (combinations of worker counts $n \in \{4, \dots, 256\}$ and vCPU counts $cput \in \{2, 4, 8\}$ ) across five input data sizes (64 GB to 1024 GB) and three CPU frequency governors (powersave, schedutil, performance).
Metrics: The system measured execution time ( $T$ ), monetary cost ( $C$ , derived from AWS c5 on-demand pricing as a proxy), energy consumption ( $E$ , measured via Intel RAPL for CPU and DRAM domains), and frugality ( $F$ ). Frugality is a composite metric of idle-node time and memory waste.
Dataset Size: The campaign resulted in 675 unique configuration cells (averaged over 3 repetitions each), totaling 2,025 experiments.

2. Surrogate Modeling and Optimization

The framework operates in two stages: offline training and online serving.

Surrogate Models: Independent regression models were trained using Histogram-based Gradient Boosted Regression Trees (HistGBM) to predict execution time, cost, energy, idle time, and memory waste. The frugality score is derived algebraically from the predicted idle and waste metrics to avoid overfitting.
Optimization Formulation: The problem is formulated as a weighted multi-objective optimization. Users provide a weight vector $W = (w_T, w_C, w_E, w_F)$ summing to 1. The system normalizes predicted objectives within the feasible candidate set and computes a scalar aggregated score $J$ .
Service Architecture: A lightweight FastAPI service enumerates all feasible configurations (up to 63 candidates per request), applies the surrogate models, calculates the weighted score, and returns the top-ranked configurations with sub-second latency.

Key Contributions

Four-Objective Framework: The paper presents a joint optimization framework for Dask-based EO processing that simultaneously considers execution time, monetary cost, processor+DRAM energy, and resource-use efficiency (frugality).
Comprehensive Measurement Campaign: A large-scale dataset was generated on bare-metal hardware covering three operators, five data sizes, fifteen cluster shapes, and three CPU governors, capturing complex interactions between these variables.
Surrogate-Assisted OaaS: The authors developed quantile surrogate models to accurately predict performance metrics for unseen workloads and implemented the optimization logic as a deployable HTTP service.
Empirical Validation: The service was evaluated against three deployable baselines (largest tier, smallest tier, and naive Dask default) and an oracle "best-in-H" configuration.

Results

The empirical evaluation demonstrates that the OaaS framework significantly outperforms common provisioning strategies:

Execution Time: When prioritizing time (T-heavy weights), the service achieved a 39% reduction in execution time compared to the largest-cluster baseline (B1) and was 3.5x faster than small-cluster baselines.
Monetary Cost: For cost-prioritized users (C-heavy), the service reduced costs by 29% compared to the naive default (B3) and 22% compared to the largest-cluster baseline.
Energy Efficiency: Under energy-focused weights (E-heavy), the service reduced energy consumption by 37% relative to the naive default and 16% relative to the largest-cluster baseline.
Frugality: The service improved frugality scores by 20% compared to the strongest baseline, effectively reducing resource over-provisioning.
Accuracy: The surrogate models achieved high predictive accuracy, with $R^2$ values ranging from 0.851 (idle time) to 0.977 (cost) and Spearman correlations above 0.91 for all targets.
Regret: The mean regret against the theoretical "best-in-H" configuration was 0.015, with exact matches to the optimal configuration in 50% of the test cases.

Significance and Claims

The paper claims that no single provisioning heuristic performs well across all objectives and workload conditions. By shifting from resource-centric to outcome-centric provisioning, the proposed OaaS framework allows users to explicitly trade off competing objectives without requiring expertise in cluster management.

The authors emphasize that their work is the first to address the intersection of service provision, multi-objective optimization (including energy and frugality), and Dask-based EO processing. The framework is designed to be modular, allowing surrogate models to be retrained independently as new hardware or operators are introduced. While the current scope is limited to homogeneous Intel Broadwell hardware and specific EO operators, the service architecture provides a reproducible proxy for cost and a rigorous method for energy measurement that can be adapted to other environments. The results suggest that automated, weight-driven configuration selection can yield substantial efficiency gains over manual or heuristic-based provisioning strategies.