The LCLStream Ecosystem for Multi-Institutional Dataset Exploration

The LCLStream ecosystem is a new end-to-end data streaming framework that integrates cloud microservices with HPC models to provide a secure, API-driven, and high-speed data request service for diverse scientific applications, such as AI training and X-ray analysis.

The Gist

Imagine you are a world-class chef working in a massive, high-tech kitchen (the X-ray beamline). You are preparing a dish that requires ingredients that are being harvested in real-time from a farm hundreds of miles away (the experimental data).

In the old way of doing things, you’d have to wait for the farmer to harvest the vegetables, pack them in heavy crates, drive them to your kitchen, and then finally unpack them before you could even start cooking. By the time you start, the ingredients might be old, or you might realize halfway through that you actually needed a different type of tomato.

This paper describes a new system called LCLStream. Instead of waiting for crates to arrive, LCLStream is like building a high-speed, ultra-secure pneumatic tube system that connects the farm directly to your kitchen. As soon as a tomato is picked, it’s sucked through the tube and arrives at your station in seconds, ready to be sliced.

Here is how the "LCLStream Tube System" works, broken down into simple parts:

1. The "Smart Slicer" (LCLStreamer)

Instead of sending the whole, bulky vegetable through the tube, we have a "Smart Slicer" at the farm. It takes the raw material, peels it, chops it exactly how the chef wants, and sends only the useful bits. This saves space in the tube and makes sure the chef doesn't waste time prepping.

2. The "Traffic Controller" (LCLStream-API & NNG-Stream)

If ten different chefs all try to use the tube at once, things could get messy.

• The API is like a digital waiter. You tell the waiter, "I need sliced carrots, medium thickness, every 5 seconds," and the waiter handles the request.
• NNG-Stream is like a "buffer tank" in the middle of the tube. If there’s a sudden burst of carrots, the tank catches them so the tube doesn't explode; if there’s a lull, the tank keeps a steady flow going so the chef never stops working.

3. The "Security Guard" (Certified)

Because this tube system is incredibly valuable, we can't have random people tapping into it. The Certified system acts like a high-tech security badge. Every person and every machine has a unique, unforgeable digital ID. If you don't have the right badge, the tube won't even open for you.

4. The "Remote Kitchen" (HPC/OLCF)

The most amazing part? The "kitchen" where the heavy cooking happens doesn't even have to be in the same building. The scientists can be at a massive, super-powered industrial kitchen (a Supercomputer) located in a completely different state. Because the "tube" is so fast, the data arrives almost as quickly as if the computer were sitting right next to the X-ray machine.

Why does this matter?

In science, speed is everything.

• AI Training: It’s like feeding a hungry robot (AI) fresh data constantly so it can learn to recognize patterns instantly.
• Real-time Adjustments: If a scientist sees through their "window" that the experiment is going slightly wrong, they can fix it while it's happening, rather than finding out a week later when the data is finally processed.

In short: LCLStream turns "collect data, then analyze it" into "analyze data as it is being born."

Technical Summary

Technical Summary: The LCLStream Ecosystem for Multi-Institutional Dataset Exploration

1. Problem Statement

Modern X-ray science experiments, particularly at facilities like the Linac Coherent Light Sources (LCLS), generate massive volumes of data at extremely high rates (gigabytes to terabytes per second). Traditionally, data analysis has been an "offline" process, occurring after the experiment has concluded. This creates a significant latency that prevents autonomous experiment steering—the ability to use real-time data to adjust experimental parameters (e.g., mechanical alignment or laser settings) to optimize results.

Furthermore, coupling High-Performance Computing (HPC) resources with on-site experimental data sources is technically challenging. It requires overcoming barriers in:

• Data Accessibility: Most data libraries (like psana) are restricted to local facility use for security and technical reasons.
• Data Heterogeneity: Different beamlines use different formats, metadata, and processing requirements.
• Security & Coordination: Securely transmitting data across institutions while managing remote HPC job scheduling and high-bandwidth network bursts.

2. Methodology

The authors propose LCLStream, an end-to-end, modular data streaming ecosystem designed to bridge the gap between experimental detectors and remote HPC facilities. The architecture merges cloud microservices (for flexibility and API-driven control) with traditional HPC batch execution models (for high-performance processing).

The framework consists of several integrated components:

• LCLStreamer: A high-performance engine that performs local data reduction, calibration, and formatting. It uses a modular "pipeline" approach (Event Source $\rightarrow$ Processing Pipeline $\rightarrow$ Serializer $\rightarrow$ Data Handler) to convert raw events into user-preferred formats (e.g., HDF5).
• LCLStream-API: A RESTful HTTPS API that allows remote users to request specific datasets using JSON-formatted queries.
• NNG-Stream: A high-speed message buffer based on the nanomsg library. It acts as a "cache" between parallel data producers (at the facility) and consumers (at the HPC center), smoothing out network bursts and providing resilience against individual process failures.
• Psi-k & Psik-API: A web-enabled interface for managing HPC batch jobs (e.g., SLURM) remotely, allowing users to submit analysis jobs as part of the data request.
• Certified: A security framework providing mutual TLS (mTLS) authentication via x.509 certificates, ensuring secure, encrypted communication between all microservices.
• Elog Integration: Leveraging the facility's electronic logbook to automate workflows based on experimental events (e.g., starting a stream automatically when a "run" begins).

3. Key Contributions

• Decoupled Data Pipeline: By performing data reduction and compression at the source (the facility), the framework reduces the bandwidth required for transmission and enables remote analysis of high-rate data.
• Modular Microservice Architecture: Instead of a monolithic program, the ecosystem uses small, interoperable services that can be independently scaled and updated.
• Unified API-Driven Workflow: It provides a single point of entry for researchers to request data, launch remote compute jobs, and monitor progress, regardless of the physical location of the data or the compute resources.
• Standardization: The framework promotes the use of open standards (REST, JSON, HDF5, mTLS) to ensure interoperability across different scientific institutions.

4. Results

The framework was validated through several high-impact scientific applications:

• AI/ML Training (MAXIE & PeakNet): Successfully streamed hundreds of terabytes of X-ray images to facilitate unsupervised training of masked autoencoders.
• High-Rate Time-of-Flight Analysis (TMO-prefex): Demonstrated the ability to process electron arrival times from the LCLS-II upgrade, enabling faster iteration in time-resolved experiments.
• Crystallography (CrystFEL): Achieved a latency of 15–25 seconds between data collection and remote processing, which is fast enough for researchers to manually steer experiments in real-time.
• Performance Metrics: Network round-trip latency between SLAC and Oak Ridge (OLCF) was measured at 33–36 ms. Data throughput was limited primarily by the local read/formatting speed (1–3 GB/sec) rather than the network buffer.

5. Significance

The LCLStream ecosystem represents a paradigm shift in Integrated Research Infrastructure (IRI). It moves X-ray science away from "collect then analyze" toward a "stream and analyze" model. This is critical for the next generation of experiments involving AI-driven optimization and extremely high-repetition-rate X-ray sources. By providing a secure, scalable, and flexible way to move data from the "edge" (the detector) to the "exascale" (the supercomputer), LCLStream enables more efficient use of beamtime and accelerates the pace of scientific discovery.