FAST-HEP: Compiling Declarative Analysis Workflows for High-Energy Physics and Beyond

In the vast, silent collisions of particles inside machines like the Large Hadron Collider, scientists are hunting for the fundamental rules that govern our universe. To find these rules, they must sift through mountains of data, looking for rare patterns hidden within billions of events. This process is not a single experiment but a long-term scientific journey that can last for decades, often outliving the specific computer programs and software tools used to create it. The challenge is not just in running the analysis once, but in ensuring that the same scientific question can be asked and answered years later, even as the underlying technology changes, the people involved move on, and the data formats evolve. If the instructions for how the data was processed are lost or become too tangled to understand, the scientific result becomes a black box, impossible to verify or rebuild.

To solve this problem of long-term scientific survival, a system called FAST-HEP, centered around a workflow engine named Flow, has been developed by Luke Kreczko at the University of Bristol. This system treats scientific analysis not as a rigid set of computer instructions, but as a clear, written description of what the scientist wants to achieve, separate from the specific code used to do it. By separating the "what" from the "how," the researcher has created a way to compile these descriptions into a universal plan that can run on different computers and with different software tools without needing to be rewritten. This approach ensures that the scientific intent remains transparent and reproducible, allowing the complex machinery of high-energy physics to evolve without breaking the experiments that depend on it.

For decades, physicists have written their data analysis as imperative code, a style where the computer is told exactly how to move step-by-step through the data, checking one event after another. While this works well for the moment, it tightly couples the scientific idea to the specific programming language and libraries used at the time. When those libraries change or the researchers who wrote the code leave, the analysis often becomes difficult to understand or impossible to rerun. The new Flow system changes this dynamic by introducing a declarative language. In this model, a scientist simply describes the data they need, the operations they want to perform, and the results they expect, without worrying about the underlying mechanics. It is like writing a recipe that lists the ingredients and the final dish, leaving the specific cooking tools and techniques to be decided later by the chef.

The heart of this system is a compiler that acts as a translator between the scientist's description and the computer's execution. When a scientist submits their workflow, the system does not run it immediately. Instead, it first normalizes the description, gathering all the scattered pieces of information—such as where the data lives, what corrections to apply, and how to handle different scenarios—into a single, complete document. It then builds a logical graph, a map that shows how every piece of data flows from the source to the final result, connecting inputs to outputs with clear lines of dependency. This map allows the system to check for errors before any heavy computing begins, ensuring that the required data exists and that the steps make sense together.

Once the map is built and validated, the system creates a backend-independent execution plan. This plan is a detailed set of instructions that describes the work to be done but does not specify which computer or software library will do it. This separation is crucial because it means the same scientific plan can be run on a laptop, a local cluster, or a massive distributed network without changing the core logic. The system can also handle variations, such as testing how results change if a specific measurement is slightly different, by expanding only the parts of the plan that are affected rather than rewriting the whole thing. This makes it easier to explore different scientific scenarios and understand how uncertainties affect the final answer.

The system also pays close attention to provenance, which is the record of exactly how a result was produced. Every time the workflow runs, it generates a detailed summary that links the final output back to the specific version of the software, the exact data files used, and the computer environment where it ran. This creates a permanent, traceable history for every scientific result. If a scientist needs to verify a finding years later, they can look at this record to see precisely what happened, rather than trying to reconstruct the process from memory or scattered notes. This level of detail turns the workflow from a black box into a transparent process where every step is visible and accountable.

The development of Flow was driven by real-world experiences where older systems struggled to adapt to new technologies. The researcher found that simply writing code in a different style was not enough; the underlying software architecture had to be designed to allow parts to be swapped out easily. In the past, changing a single library often required rewriting large chunks of the framework because the different parts were too tightly linked. Flow solves this by treating every component, from data sources to output formats, as a replaceable module that connects through clear, defined contracts. This means that as new, faster, or more efficient tools become available, they can be plugged into the system without disrupting the scientific analysis itself.

This approach has already been tested in real analyses for major experiments, including those at the CMS detector and the LUX-ZEPLIN experiment. The results show that a concise, declarative description can successfully guide complex computations across different data structures and experiments. The system successfully separates the scientific intent from the implementation, allowing the analysis to remain stable while the software ecosystem around it evolves. By making the workflow explicit and inspectable, the system reduces the burden on scientists to remember every detail of their code and provides a solid foundation for long-term preservation.

The ultimate goal of this work is to ensure that scientific analyses remain sustainable over time. In a field where data volumes are growing and computing resources are becoming more diverse, the ability to preserve and reproduce results is essential. Flow provides a way to do this by making the workflow a first-class object that can be compiled, analyzed, and executed independently of the tools used to create it. This allows the scientific community to evolve its software and hardware without losing the ability to understand or repeat the work of the past. The system does not just run the analysis; it documents the entire process, ensuring that the path from raw data to scientific discovery remains clear and accessible for future generations.

The success of this project relies on a shift in how scientific software is built. Instead of viewing workflows as temporary scripts that run once and are forgotten, the researcher treats them as programs that can be compiled and validated. This perspective allows for a level of transparency and flexibility that was previously difficult to achieve. The system records every decision, every dependency, and every variation, creating a complete picture of the scientific process. This not only helps with immediate debugging and validation but also builds a lasting record that can be used to verify results long after the original researcher has moved on.

In the end, the work presented in this paper offers a new way to think about scientific computing. It moves away from the idea that the code is the most important part of an analysis and instead focuses on the clarity of the scientific description. By separating the description of the science from the machinery that runs it, the system ensures that the science itself remains the priority. This allows the tools and technologies to change and improve without threatening the integrity of the research. The result is a more robust, transparent, and sustainable approach to scientific discovery, one that can adapt to the future while honoring the work of the past.