💻 computer science

Overcoming Orchestration Bottlenecks at Exascale: A Decentralized, Policy-Driven Approach for Sim-AI Ensembles

This paper introduces EnsembleLauncher, a decentralized, policy-driven workflow orchestrator that overcomes orchestration bottlenecks on exascale systems like Aurora by successfully scaling to eight million tasks and significantly outperforming state-of-the-art tools while enabling flexible scheduling for heterogeneous simulation-AI ensembles.

Original authors: Harikrishna Tummalapalli, Christine M. Simpson, Riccardo Balin, Vitali A. Morozov, Thang D. Pham, Murat Keceli, Thomas D. Uram

Published 2026-07-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Harikrishna Tummalapalli, Christine M. Simpson, Riccardo Balin, Vitali A. Morozov, Thang D. Pham, Murat Keceli, Thomas D. Uram

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the director of a massive, chaotic movie set. You have millions of tiny, split-second tasks (like a camera flash) and a few huge, slow-moving tasks (like moving a giant castle set). In the past, trying to manage all these actors and props on a supercomputer the size of a city (like the Aurora supercomputer) was a nightmare. The "manager" in charge of handing out jobs got so overwhelmed by the sheer number of requests that the whole production ground to a halt.

This paper introduces a new way to run the show called EnsembleLauncher. Instead of having one single boss trying to talk to every single worker, EnsembleLauncher builds a fractal tree of managers. Think of it like a giant corporate ladder where the CEO doesn't call the interns; the CEO calls the VPs, the VPs call the managers, and the managers call the interns. This "recursive hierarchy" means the top boss never gets swamped with too many phone calls.

The Big Discovery: It's About the Shape, Not the Software

The authors ran a fascinating experiment to figure out why other tools were failing. They took two very different software tools (Dask and Parsl) and forced them to use the same "flat" management style (where everyone talks to the boss). Both tools crashed and burned at the same time.

Then, they gave those same tools a "hierarchical" shape (the tree structure). Suddenly, they worked much better.

The Finding: The paper proves that the shape of the management team (the topology) is the most important thing, not the specific software code being used. If you have a flat structure, even the best software will choke. If you have a tree structure, even different software can scale up.
The Proof: On the Aurora supercomputer, they scaled this new system to 8,192 nodes (which is the maximum they were allowed to use for the test) and ran 8 million serial tasks. The system was more than four times faster than the best tools currently available.

Why the Old Way Failed: The "Traffic Jam"

To understand why the old way failed, imagine a single traffic cop standing in the middle of a highway trying to direct millions of cars.

The Flat Problem: In the old "flat" systems, every single task had to ask the central scheduler for permission to run. At 128 nodes, the paper measured that for tiny 0.1-second tasks, the workers spent 27.6 seconds just waiting in line to talk to the boss, while the actual work only took 0.1 seconds. The workers were sitting idle, staring at their phones, because the boss was too busy.
The Tree Solution: EnsembleLauncher lets the local managers handle the small talk. The workers only talk to their local manager, who only talks to the next level up. This reduced the waiting time for those tiny tasks from 27.6 seconds down to just 1.9 seconds.

The "Smart" Scheduler: No One-Size-Fits-All

The paper also argues that you can't just use one rule to manage all jobs. Sometimes you need to do the biggest jobs first; sometimes you need to do the quickest ones.

The Experiment: They tested different "rules" for a mix of tasks that varied wildly in size and time (some took 0.1 seconds, others took over 1,500 seconds).
The Result: They found that for tasks with high variation (some very fast, some very slow), picking the right rule mattered a lot. Using a "Largest First" or "Longest First" rule cut the total time significantly compared to just doing tasks in the order they arrived (FIFO).
The Flexibility: EnsembleLauncher lets scientists plug in their own custom rules. In a test mimicking a real scientific workflow (called MOFA), a "smart" rule that routed jobs to the least busy team finished the work twice as fast as a rigid rule that forced all small jobs into one specific corner of the computer.

What They Didn't Solve (Yet)

The paper is very clear about what they haven't fixed.

The "Straggler" Problem: If one branch of the tree gets stuck with a slow job, the system can't easily steal that job and give it to a faster branch. They suggest this as future work.
The "One-Node" Crash: If a manager node dies, the system currently has to restart the entire branch of the tree under it, not just the single broken piece. They admit this is a limitation.
The Scale Limit: While they tested up to 8,192 nodes, the paper notes this was just the limit of the machine they were using, not necessarily the limit of their software. They suspect it could go even higher, but they haven't measured it yet.

The Bottom Line

The paper shows that to run the massive, mixed-up workflows of the future (where AI and super-simulations dance together), we need to stop using flat, single-boss systems. By building a deep, recursive tree of managers and letting scientists choose their own scheduling rules, we can keep the supercomputer's workers busy and the production moving. It's not just a new tool; it's a new way of thinking about how to organize a digital army of millions.

Technical Summary: Overcoming Orchestration Bottlenecks at Exascale

Problem Statement

Scientific computing is transitioning from monolithic applications to coupled Simulation-AI (Sim-AI) workflows. These workflows consist of highly heterogeneous tasks, ranging from short-running serial inference (sub-second) to long-running, multi-node MPI simulations (thousands of seconds). As these workflows scale to leadership-class exascale systems (e.g., Aurora, Frontier), they generate extreme ensemble sizes (millions of concurrent tasks) and high variance in task duration.

Current orchestration approaches face two primary bottlenecks:

System-Level Schedulers: Tools like Slurm and PBSPro are optimized for large allocations but struggle with the extreme throughput required for millions of small, dynamic tasks, often leading to scheduling bottlenecks.
User-Space Workflow Tools: Existing frameworks often rely on centralized control planes that create serialization bottlenecks at scale. While decentralized tools exist, many lack native support for multi-node MPI applications or impose steep learning curves. Furthermore, most tools lack programmable scheduling interfaces, preventing domain scientists from injecting custom runtime policies needed to address specific workflow bottlenecks.

Methodology: EnsembleLauncher

To address these challenges, the authors developed EnsembleLauncher, a recursively hierarchical workflow orchestrator designed for exascale Sim-AI ensembles.

Architectural Design

EnsembleLauncher employs a fully decentralized control plane structured as a recursive tree:

Managers: Govern subtrees (Global Manager as root, Sub-Managers as intermediate nodes). They handle coarse-grained task routing and resource partitioning.
Launchers: Serve as leaf nodes and primary execution engines. They handle fine-grained task ordering and execution.
Hybrid Push-Pull Mode: The system supports a hybrid architecture where upper levels use Static Partitioning (Push) to minimize control-plane communication, while leaf levels use Dynamic Load Balancing (Pull) to mitigate execution time skew via work-stealing.

Core Components

Programmable Scheduling Interface: The system exposes policy interfaces at two levels:
- Children Scheduler (Manager Level): Determines task routing and resource partitioning ( $S_{i+1} = f(S_i, T, N, d)$ ).
- Task Scheduler (Launcher Level): Determines task ordering in the dispatch queue ( $P = f(T, S)$ ).
- These interfaces allow users to implement custom heuristics (e.g., FIFO, Shortest First, Largest First) without modifying the core orchestrator.
Executor Abstraction: A stateless layer that decouples orchestration state from execution mechanisms, supporting Python process pools, thread pools, and MPI process pools. This enables the co-scheduling of serial and MPI tasks.
Fault Tolerance: Each orchestrator maintains a local checkpointer. Failures trigger localized teardown and restart of the affected subtree, with heartbeat mechanisms detecting unresponsive children.

Evaluation Setup

Experiments were conducted on the Aurora supercomputer (10,624 nodes, Intel Xeon Max CPUs, Intel Data Center GPUs). The evaluation compared EnsembleLauncher against state-of-the-art tools (Dask, Parsl) and a native MPI baseline across:

Micro-benchmarks: Single-node throughput and latency.
Scaling Tests: Weak scaling (workload proportional to nodes) and strong scaling (fixed workload, increasing nodes) up to 8,192 nodes.
Policy Sensitivity: Analysis of control-plane topology impact and the effect of scheduling policies on heterogeneous workloads.

Key Results

1. Scalability and Throughput

Exascale Scaling: EnsembleLauncher successfully scaled to 8,192 nodes with 8 million serial tasks.
Performance Gain: It outperformed state-of-the-art tools (Parsl and Dask) by more than 4× at 2,048 nodes, even for long-duration (60s) tasks.
Topology vs. Implementation: Controlled experiments demonstrated that control-plane topology is the primary determinant of throughput, not the specific framework implementation. When Dask and Parsl were configured with the same single-level hierarchy, they exhibited nearly identical scaling behavior and failure points.
Hierarchical Advantage: A two-level hierarchy (Depth=2) in EnsembleLauncher eliminated the control-plane saturation that starved workers in single-level configurations, maintaining near-ideal strong scaling.

2. Latency

Dispatch Latency: In single-node tests, EnsembleLauncher achieved a median round-trip latency of 3ms (Launcher-only) and 4ms (Manager-Launcher). This is significantly lower than Dask (10ms) and remains well below the duration of typical Sim-AI inference tasks (tens to hundreds of milliseconds).
Overhead: While each hierarchical level adds approximately 1ms of latency, this overhead is negligible compared to the execution time of scientific tasks.

3. Impact of Scheduling Policies

Heterogeneous Workloads: In experiments with a "bag-of-tasks" featuring high variance in duration and resource requirements, Largest First and Longest First policies significantly reduced total execution time compared to FIFO as variance increased.
Sim-AI Pipelines: In a MOFA workflow prototype, an Adaptive routing policy (dynamically routing to least-loaded children) achieved 2× higher GPU utilization and shorter execution times compared to static partitioned or uniform policies. Static partitioning caused bottlenecks where serial tasks saturated specific nodes, blocking subsequent pipeline stages.

Significance and Claims

The paper claims that EnsembleLauncher effectively overcomes orchestration bottlenecks in exascale Sim-AI workflows by shifting the architectural paradigm from centralized control to a recursively hierarchical, decentralized model.

Key contributions include:

Demonstrating Topology Dominance: Proving that the control-plane topology is the critical factor for scalability, not the underlying implementation language or specific algorithm of the scheduler.
Scalability: Achieving throughput on 8,192 nodes that exceeds current state-of-the-art tools by a factor of four, enabling the execution of ensembles with up to 8 million tasks.
Programmability: Providing a programmable interface that allows domain scientists to tailor scheduling and routing policies to specific workload characteristics (e.g., high variance, mixed MPI/serial tasks) without modifying the orchestrator code.
Co-Scheduling: Successfully integrating the execution of granular serial inference tasks and massive multi-node MPI simulations within a single workflow.

The authors conclude that while 8,192 nodes represents the maximum allocation on Aurora, the system's design suggests it is not the scalability limit of EnsembleLauncher. Future work aims to integrate Model Context Protocol (MCP) interfaces for LLM-driven agentic workflows and explore AI-driven scheduling policies that adapt heuristics at runtime.