OxyMake: A Formally-Specified, Content-Addressable Workflow Engine
OxyMake is a formally-specified, single-binary Rust workflow engine that replaces unreliable file-modification timestamps with content-addressed BLAKE3 hashing to ensure robust, cross-machine caching and eliminate spurious re-executions while maintaining compatibility with existing Make-style pipelines.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are running a massive, multi-day cooking competition. You have a recipe book (the workflow) that tells you how to turn raw ingredients into a final feast.
In the old days, the head chef (workflow engines like GNU Make or early versions of Snakemake) decided whether to re-cook a dish based on a very simple rule: "Did the clock on the ingredient jar change?"
This is like checking the "Last Modified" timestamp on a file. But this rule is flawed.
- The False Alarm: If you copy your ingredients to a new kitchen, or restore them from a backup, the jars get new timestamps even though the food inside is identical. The old chef sees the new time, panics, and re-cooks the entire meal from scratch, wasting hours of time and energy.
- The Silent Failure: Conversely, if you swap the food inside a jar but don't change the jar's label (timestamp), the chef sees the old time, assumes everything is fine, and serves you stale, rotten food without realizing it.
OxyMake is a new, super-smart kitchen engine built to fix this. Instead of looking at the clock on the jar, it tastes the food (checks the actual content).
Here is how OxyMake works, broken down into simple concepts:
1. The "Taste Test" (Content-Addressable Caching)
Instead of asking "When was this last touched?", OxyMake asks, "What is this made of?"
It creates a unique "fingerprint" (a cryptographic hash) for every single ingredient, every step of the recipe, and every tool used.
- If you copy your ingredients to a new kitchen, the fingerprints stay the same. OxyMake sees they haven't changed and says, "No need to cook; we already have the perfect dish."
- If you swap a tomato for a potato, the fingerprint changes instantly. OxyMake knows immediately that the dish is different and cooks only the specific part that needs changing.
This eliminates "phantom re-runs" (wasting time cooking what you already have) and "silent stale outputs" (serving bad food because the label looked old).
2. The "Blueprint vs. The Build" (Three-Graph Architecture)
OxyMake thinks in three layers, like an architect, a builder, and a foreman:
- The Blueprint (RuleGraph): This is your recipe book. It's abstract. It says, "Make a soup for every vegetable." It doesn't know which vegetables yet.
- The Build Plan (JobGraph): OxyMake takes the blueprint and expands it. "Okay, we have carrots, potatoes, and onions. That means we need to make three specific soups." It figures out exactly what needs to be done before it starts cooking.
- The Construction Site (ExecGraph): This is the actual cooking happening right now. It tracks which pot is boiling, which one is finished, and which one burned.
Because OxyMake plans everything out first (in a fraction of a second), it can handle huge recipes with thousands of steps much faster than other engines that try to figure things out as they go.
3. The "No-Boss" Kitchen (Daemon-Free Execution)
Most workflow engines require a central "boss" computer (a daemon) to watch over the kitchen 24/7. If the boss crashes, the kitchen stops.
OxyMake is different. It's like a team of independent cooks who can walk in and out of the kitchen at any time.
- Cooperative Claiming: If two cooks try to grab the same pot, they have a polite handshake protocol to decide who cooks it.
- No Single Point of Failure: If one cook leaves or their computer crashes, the others just pick up the slack. You don't need a central manager to keep the kitchen running.
4. The "Universal Translator" (Polyglot Execution)
In a normal kitchen, you might be forced to use only one type of knife or one language. OxyMake is a polyglot.
- You can write the recipe in TOML (a simple, clean text format), which is easy for humans to read and for computers to understand instantly.
- But the actual cooking can happen in Python, R, Shell scripts, or Julia. OxyMake doesn't care what language the cook speaks; it just passes the ingredients and collects the finished dish.
5. The "Honest Ledger" (Reproducibility)
OxyMake keeps a strict, unchangeable diary (a lockfile called ox.lock).
- It records exactly what ingredients were used, what the recipe said, and what the final dish tasted like.
- Even if you come back to this kitchen five years later, on a completely different computer, OxyMake can look at the diary and say, "Yes, if we follow this exact path, we will get the exact same result."
- It even uses formal math (TLA+) to prove that its "handshake protocol" for the cooks is safe, ensuring that two cooks never accidentally cook the same dish twice or step on each other's toes.
The Trade-Off: Speed vs. Safety
The paper is very honest about one downside:
- Cold Start: The very first time you run a recipe, OxyMake is slightly slower than the old engines because it has to "taste" (hash) every single ingredient to build its fingerprints.
- Warm Run: But once it has done that, the second time you run it (or if you run it on a different computer), it is massively faster and much more reliable. It skips the cooking entirely if the fingerprints match, whereas old engines might get confused by the timestamps and re-cook everything.
Summary
OxyMake is a workflow engine that stops guessing based on timestamps and starts checking the actual content of your data. It's built with a fast, modern language (Rust), doesn't need a central boss to run, and keeps a perfect, auditable record of everything it does. It trades a tiny bit of initial setup time for huge gains in speed, safety, and reliability when you run your workflows again and again.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.