Jarvis-HEP: A lightweight Python framework for workflow composition and parameter scans in high-energy physics

This paper introduces Jarvis-HEP, a lightweight Python framework designed to streamline high-energy physics workflows by enabling YAML-based specification, dependency-aware execution, and modular integration of diverse computational tools for efficient parameter scans and multi-step studies.

The Gist

Imagine you are a detective trying to solve a massive mystery: What is the universe made of?

In the world of high-energy physics, scientists have a "Standard Model" (a rulebook of known particles), but they suspect there are hidden rules and secret characters (New Physics) lurking in the shadows. To find them, they have to test millions of different "what-if" scenarios.

The problem is that testing these scenarios is like trying to bake a cake where every ingredient requires a different oven, a different chef, and a different recipe book. Some tools calculate particle masses, others predict how dark matter behaves, and others simulate collisions. Usually, a scientist has to manually run Tool A, copy the results, paste them into Tool B, check if the numbers make sense, and then run Tool C. If they want to test 10,000 different scenarios, doing this by hand is impossible—it's slow, prone to errors, and exhausting.

Enter Jarvis-HEP.

Think of Jarvis-HEP as a super-smart, automated kitchen manager for these physics detectives. It doesn't bake the cake itself; instead, it organizes the whole kitchen so the chefs (the various physics software tools) can work together seamlessly.

Here is how it works, using simple analogies:

1. The Master Recipe (The YAML File)

Instead of writing complex computer code to tell the tools what to do, scientists write a simple, human-readable "recipe" called a YAML file.

• Analogy: Imagine writing a shopping list and a set of instructions on a notepad: "Buy 500 apples. If the apple is red, bake a pie. If it's green, make a salad."
• In the paper: This file tells Jarvis-HEP: "Here are the variables (like the size of the universe or the mass of a particle). Here is the rule for how to pick them (randomly or in a grid). Here is the order to run the tools."

2. The Assembly Line (The Workflow)

Once the recipe is written, Jarvis-HEP sets up an assembly line.

• Analogy: Imagine a factory where a robot arm grabs a raw material, passes it to a painter, then to a polisher, and finally to a quality checker.
• In the paper: Jarvis-HEP automatically links different software packages. It takes the output of one tool and feeds it directly into the next, ensuring that if Tool A needs a specific file format, Tool B gets it exactly right. It handles the "dependencies" (making sure the right tools are installed and ready).

3. The Busy Bee Swarm (Asynchronous Execution)

This is Jarvis-HEP's superpower. Traditional methods do things one by one (like a single person washing dishes, drying them, and putting them away). Jarvis-HEP uses asynchronous parallel processing.

• Analogy: Imagine a swarm of 16 bees working in a hive. While one bee is waiting for a flower to bloom, another is flying to a different field. They don't wait for each other. If one bee is slow, the others keep working.
• In the paper: The system runs many calculations at the same time. If one computer part is busy, the system instantly moves to the next available task. This makes the whole process incredibly fast and efficient.

4. The "EggBox" Test Drive

To prove it works, the authors tested Jarvis-HEP on a famous math puzzle called the "EggBox" potential.

• Analogy: Imagine a landscape full of hills and valleys (the "EggBox"). The goal is to find the deepest valleys (the most likely answers).
• In the paper: They showed that Jarvis-HEP could use different "search strategies" (like random walking, grid searching, or smart AI-guided searching) to explore this landscape. It successfully found all the important peaks and valleys without getting stuck or confused.

5. The Safety Net (Logging and Checkpoints)

If you are baking 10,000 cakes and the power goes out, you don't want to start over from scratch.

• Analogy: Jarvis-HEP is like a baker who takes a photo of the kitchen every 30 seconds. If the power fails, you can look at the last photo, pick up exactly where you left off, and keep baking.
• In the paper: The system automatically saves "checkpoints." If a scan is interrupted, you can restart it, and it resumes from the exact point it stopped, not from the beginning. It also keeps detailed logs (like a diary) of every single step, so if something goes wrong, you know exactly which "ingredient" caused the problem.

Summary

Jarvis-HEP is a lightweight, easy-to-use tool that lets physicists stop worrying about the messy logistics of connecting different software tools. It turns a chaotic, manual process into a smooth, automated assembly line.

• For the User: You just write a simple recipe (YAML file).
• For the Computer: It handles the heavy lifting, manages the tools, runs thousands of tests at once, and saves the results in an organized way.

The paper claims this makes it much easier for researchers (even small teams or individuals) to explore complex theories about the universe without needing to be expert programmers or spend weeks setting up their computer systems. It's a "workflow composition" tool designed to make the science faster and less prone to human error.

Technical Summary

1. Problem Statement

High-energy physics (HEP) phenomenology, particularly in the search for Beyond the Standard Model (BSM) physics, requires evaluating observables, likelihoods, and experimental constraints across complex, high-dimensional parameter spaces. This process typically involves:

• Multi-step Workflows: Linking disparate computational tools (e.g., spectrum generators, cross-section calculators, Monte Carlo simulators).
• Complex Dependencies: Managing external software packages (e.g., Pythia, HepMC) and their interdependencies.
• Scalability Issues: Performing parameter scans over millions of points often leads to cumbersome, error-prone manual management of execution, logging, and data recording.
• Accessibility Gap: Existing global fitting frameworks (e.g., GAMBIT, MasterCode) are powerful but often heavy and complex to set up. There is a lack of lightweight, flexible tools for individual researchers or small teams to conduct exploratory studies without significant overhead.

2. Methodology: Jarvis-HEP Framework

Jarvis-HEP is a pure Python framework designed to streamline these workflows through a declarative, project-based architecture.

Core Architecture

• YAML-Based Configuration: Users define the entire scan workflow (variables, likelihoods, dependencies, calculators) in a single human-readable YAML file.
• Worker Factory & Asynchronous Execution:
  • The framework employs a Worker Factory (a singleton) that manages global task scheduling.
  • It utilizes asynchronous task scheduling (asyncio) to execute multiple computational tasks concurrently. Unlike synchronous execution, this non-blocking approach allows faster nodes to process tasks independently of slower ones, optimizing resource usage.
  • The workflow is visualized as a directed hierarchical flow graph where packages are grouped into computing layers (L1, L2, L3) to minimize dependencies and maximize parallelism.
• Modular Design:
  • Calculators: Supports both external command-line tools and internal Python components.
  • Dependency Management: Automatically handles the installation and compilation of external libraries (e.g., HepMC, Pythia8) via shell commands defined in the YAML.
  • Data Recording: A standalone Data Recorder process writes results to HDF5 files in real-time (default interval: 30 seconds), ensuring data persistence even if the process is interrupted.

Sampling Capabilities

The framework decouples the sampling algorithm from the execution engine, implementing samplers as Python iterators for memory efficiency. Supported methods include:

• Deterministic/Independent: Random sampling, Grid sampling, and Bridson sampling (Poisson disk sampling for repulsive priors).
• Markov Chain Monte Carlo (MCMC): Metropolis-Hastings (MH) and Parallel Tempering MCMC (PT-MCMC) for exploring multimodal distributions.
• Nested Sampling: Integration with dynesty for Bayesian evidence calculation and posterior sampling.
• Machine Learning: A Deep Neural Network (DNN) sampler that learns the mapping from parameters to observables, followed by rejection sampling to correct biases.

User Interface

• Command-Line Interface (CLI): Operated via the Jarvis command with flags for debugging (--check-modules), monitoring (--monitor), data conversion (--convert), and visualization (--plot).
• Project Structure: Organizes tasks into self-contained directories containing configuration (bin/), dependencies (deps/), outputs, logs, and checkpoints.

3. Key Contributions

• Lightweight Workflow Orchestration: Provides a unified interface to link external HEP tools, manage dependencies, and execute complex multi-step calculations without writing custom glue code.
• Asynchronous Parallel Processing: Introduces a non-blocking execution model that significantly improves throughput for I/O-bound and compute-bound tasks compared to traditional sequential scanning.
• Robust Resilience: Features built-in checkpointing and resuming capabilities, allowing long-running scans to recover from interruptions (e.g., terminal disconnection, system crashes) without restarting from zero.
• Comprehensive Ecosystem: Includes integrated tools for:
  • Logging: Structured, point-level logging for debugging specific failures.
  • Visualization: Integration with Jarvis-PLOT for generating standard HEP plots (corner plots, likelihood surfaces).
  • Portability: Ability to pack projects into reproducible bundles for sharing or archiving.
• Symbolic Expression Engine: Supports symbolic math (via sympy) for defining likelihoods, selection cuts, and derived variables directly within the configuration.

4. Results and Validation

The paper validates Jarvis-HEP using the "EggBox" model, a synthetic test case with a complex, multimodal likelihood landscape ($z = (\sin(x\pi)\cos(y\pi) + 2)^5$).

• Performance: On a standard personal computer, the framework sustained a throughput of $3 \times 10^4$ to $6 \times 10^4$ completed lightweight tasks per minute. This demonstrates that the asynchronous scheduling layer introduces minimal overhead compared to the actual physics calculations.
• Sampling Efficiency:
  • PT-MCMC successfully identified multiple distinct high-likelihood regions in the EggBox model where standard MCMC chains got trapped.
  • Bridson sampling demonstrated superior uniformity and space-filling properties compared to random sampling.
  • DNN Sampling effectively learned the target distribution, concentrating samples in high-likelihood regions.
• Usability: The end-to-end workflow (fetching a project, running a scan, monitoring, visualizing, and archiving) was demonstrated to be seamless, reducing the barrier to entry for complex phenomenological studies.

5. Significance

Jarvis-HEP addresses a critical gap in the HEP software ecosystem by offering a flexible, low-overhead alternative to monolithic global fitting frameworks.

• Democratization of Analysis: It enables individual researchers and small teams to perform rigorous, reproducible parameter scans without needing extensive infrastructure or deep expertise in complex workflow management systems.
• Future-Proofing: Its modular architecture allows for the easy integration of new sampling algorithms (including AI-driven methods) and external tools.
• Reproducibility: By enforcing a project-based structure with automated dependency handling and checkpointing, it significantly enhances the reproducibility of numerical studies in high-energy physics.

In summary, Jarvis-HEP is a versatile tool that modernizes the workflow of BSM phenomenology, balancing the need for computational power with the practical requirements of ease of use and reproducibility.