How are MLOps Frameworks Used in Open Source Projects? An Empirical Characterization
This paper empirically characterizes the usage of eight popular open-source MLOps frameworks by analyzing their adoption in GitHub projects and correlating actual API usage patterns with community feature requests, revealing that developers predominantly customize these tools via APIs rather than using them out-of-the-box while seeking improvements in core functionality, API exposure, and CI/CD integration.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are a chef trying to run a massive, high-tech restaurant. You have a brilliant recipe (the Machine Learning model), but getting that recipe from the kitchen to the customer's table perfectly every single time is a nightmare. You need to track ingredients, monitor cooking times, ensure the food doesn't spoil, and automate the serving process.
MLOps frameworks are like the "smart kitchen systems" designed to help you manage this chaos. They are software tools that promise to handle the entire lifecycle of your AI recipes, from mixing the batter to serving the dish.
This paper is like a forensic investigation into how 969 real-world "restaurants" (open-source software projects) are actually using these smart kitchen systems. The researchers looked at eight of the most popular systems (like MLFlow, Wandb, and BentoML) to answer three big questions: How are they used? What parts are people actually using? And what are people complaining about or asking for?
Here is what they found, broken down into simple terms:
1. The "Out-of-the-Box" Myth
The Expectation: You buy a smart kitchen system, plug it in, and press a "Start" button. The machine does everything for you.
The Reality: The paper found that almost no one uses these tools like that. Developers rarely just run them from a command line or set them up in an automated workflow (like a robot arm that just turns on).
Instead, developers treat these frameworks like Lego sets. They don't just use the pre-built castle; they take the bricks (the code APIs) and build their own custom structures. They are writing their own scripts to connect the tools to their specific needs, rather than letting the tool do the heavy lifting automatically.
2. The "Swiss Army Knife" vs. The "Specialized Tool"
The researchers noticed that developers often use multiple tools at once, like a chef using a specific knife for fish, a different one for vegetables, and a thermometer for the oven.
- The Trackers: Tools like MLFlow and Wandb are the most popular. They are like the "scoreboards" and "notebooks" of the kitchen. Developers use them to write down what ingredients they used, how the dish tasted, and to keep a history of every version of the recipe.
- The Builders: Tools like Kedro and Prefect are like the assembly line managers. They help organize the steps of the cooking process.
- The Quality Checkers: Tools like Deepchecks and Evidently AI are the health inspectors. They check if the ingredients have gone bad or if the taste has changed over time.
The study found that people often combine these. For example, they might use a "scoreboard" (MLFlow) to track the recipe, an "assembly line" (Prefect) to run the cooking steps, and a "health inspector" (Evidently AI) to make sure the food is safe. They do this because no single tool does everything perfectly.
3. The "Missing Manual" Problem
The researchers also looked at the "complaint boxes" (issue trackers) where users tell the tool makers what they want. They found a clear pattern:
- People want better "scoreboards": Users are asking for better ways to track their experiments and monitor their models.
- People want easier "plug-ins": Since developers are building their own custom structures (using the APIs), they are asking for the tools to be easier to connect to their custom builds. They want the "Lego bricks" to snap together more easily.
- People want better automation: Even though they aren't using the tools "out-of-the-box" yet, they are asking for better ways to integrate these tools into their automated cooking lines (CI/CD pipelines).
The Big Takeaway
The paper concludes that while these "smart kitchen systems" exist and are powerful, they aren't being used as simple, ready-made solutions. Instead, developers are hacking them together to build custom solutions because the tools don't quite fit their specific needs yet.
The users are saying: "We love the core features (like tracking and monitoring), but please make it easier to connect these tools to our custom workflows, and please help us automate the boring parts better."
In short, the MLOps world is currently in a phase where everyone is building their own custom kitchen gadgets using the same set of tools, rather than just buying a pre-made appliance. The tool makers need to listen to the builders to make the next generation of gadgets easier to use.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.