Technical Summary: Delivering Science as a Service: Sci-Orchestra's Cloud-Native Approach to HPC

Problem Statement

Modern scientific discovery is increasingly reliant on advanced computational models, yet researchers face a critical bottleneck: the heavy technical burden of deploying and managing these environments. Traditionally, scientists are forced to act as system administrators, navigating complex infrastructure management, authentication protocols, and container deployments. This is particularly challenging in High-Performance Computing (HPC) environments at U.S. national laboratories (e.g., NERSC, ALCF, OLCF), which must balance open science with strict national security policies (e.g., Controlled Unclassified Information).

Existing solutions fail to fully address this gap:

• Traditional Workflow Managers (e.g., Nextflow, Snakemake) are optimized for static, non-interactive batch processing and lack the ability to host live, interactive dashboards or API-driven web applications.
• Cloud-Native Platforms (e.g., JupyterHub, BinderHub) offer interactive browser-based experiences but are often specialized for single-user notebooks and struggle to natively orchestrate multi-component service architectures or manage persistent, event-driven lifecycles across restrictive, hybrid HPC environments.
• Enterprise Orchestrators (e.g., Apache Airflow) are biased toward cloud environments and face significant engineering challenges when adapted to the rigid security and resource allocation rules of HPC centers.

There is a distinct lack of a platform that combines the interactive, browser-first user experience of cloud frameworks with the ability to dynamically deploy complex, interconnected microservices into highly restrictive HPC environments.

Methodology: The Sci-Orchestra Framework

Sci-Orchestra is a layered orchestration framework designed to fully automate experimental workflows, shifting the burden of resource allocation from the user to the system. It operates on a "Science-Platform-as-a-Service" model, utilizing a Kubernetes-based event management architecture.

Core Architectural Components

The system is built on three primary abstractions:

1. Service: A schema-driven definition of a specific task or application, including dependencies and policy constraints (e.g., a Jupyter container image with specific libraries and GPU requirements).
2. Event: An operational instance requiring computational allocation, dictating the timing, location, and parameters of the execution.
3. Entry: The concrete instantiation of one or more services injected into a specific Event, representing the live execution (e.g., running Kubernetes pods and a secure, dynamic URL).

Key Technical Enablers

• API-First Design: The framework abstracts local computational overhead, allowing researchers to provision, scale, and launch workflows via a single, streamlined API call.
• Containerization & Orchestration: Leveraging Kubernetes (with support for user-namespace runtimes like Singularity/Apptainer and Podman for HPC security), the system decouples software ecosystems from low-level toolchains. It automates the creation of namespaces, deployments, and services, including sidecar configurations for monitoring.
• FastMCP Integration: To facilitate seamless connectivity between diverse tools and AI agents, Sci-Orchestra incorporates FastMCP. This standardized protocol bridges data sources, scripts, and AI models, allowing the orchestration engine to dynamically recognize inputs, outputs, and capabilities without custom API wrappers.
• Federated Authentication & Security: The system integrates with identity providers (Google, institutional logins) to provide secure, event-specific URLs. Credentials are not persistently stored within computational environments, ensuring compliance with strict security policies.
• Infrastructure as Code (IaC): Resource management is treated as version-controllable source code, enabling reproducible, scalable, and automated deployment of network, storage, and compute assets on demand.

Key Contributions

1. Automated Workflow Orchestration: A framework that fully automates the lifecycle of experimental workflows, allowing scientists to prioritize discovery over backend operations.
2. Cross-Environment Interoperability: A modular infrastructure capable of spanning diverse environments, from generic servers to restricted HPC facilities (e.g., NERSC Spin and Perlmutter), without requiring code modifications for the underlying application.
3. Secure "Black-Box" Interoperability: An autonomous marketplace and execution environment that allows external collaborators to test and validate proprietary tools without source-code exchange, protecting intellectual property while enabling integration into scientific pipelines.
4. AI-Ready Infrastructure: A programmatic foundation enabling Machine Learning (ML) models, Large Language Models (LLMs), and autonomous agents to dynamically request HPC resources and orchestrate pipelines without human-in-the-loop bottlenecks.
5. Dynamic Service Discovery: A system that dynamically identifies, registers, and binds available computational and data services at runtime, bypassing the need for static infrastructure configurations.

Experimental Results

The authors validated the framework through comparative deployment experiments using two distinct services:

1. JupyterHub: The system successfully automated the provisioning of a secure, hardware-accelerated Jupyter Notebook environment on NERSC Perlmutter. The workflow demonstrated seamless federated login, automatic isolation of the software environment, and policy-compliant allocation of dedicated GPU resources, all accessible via a web browser.
2. Zenesis-web: A containerized computer vision application for zero-shot image segmentation was deployed to process high-resolution scientific datasets (thermal infrared and Synthetic Aperture Radar). The system handled computationally intensive tasks, including multi-stage processing and 3D rendering, leveraging HPC backend resources dynamically. The deployment confirmed the platform's ability to manage complex, non-optical sensor data formats within an interactive web environment without manual configuration.

These experiments demonstrated that a single service template could be generalized to deploy on virtually any backend, successfully bridging the gap between cloud-native dynamic resource allocation and rigid HPC infrastructure.

Significance and Claims

The paper posits that Sci-Orchestra fundamentally transforms the HPC landscape by bridging the architectural gap between traditional, command-based batch processing and modern, interactive, AI-driven scientific workflows.

• Acceleration of Discovery: By removing the technical barrier of infrastructure management, the framework accelerates the transition from laboratory prototypes to industrial-scale applications.
• Enabling Autonomous Science: The API-first, programmatic foundation is uniquely suited for "smart labs" and self-driving laboratories, where autonomous agents can compose, deploy, and monitor complex, multi-facility experiments in real-time.
• Secure Collaboration: The "black-box" approach facilitates industry partnerships by allowing proprietary tools to be tested and validated in secure execution environments without exposing source code.
• Scalability: The architecture supports a transition from individual experimental pods to high-throughput industrial deployments, offering a flexible ecosystem for distributed research teams.

In conclusion, Sci-Orchestra provides a comprehensive "Science-Platform-as-a-Service" that decouples scientific software stacks from low-level facility toolchains, equipping the scientific community with an agile, secure, and programmable ecosystem to address increasingly complex and time-critical research challenges.