🤖 machine learning

MinT: Managed Infrastructure for Training and Serving Millions of LLMs

MinT is a managed infrastructure system that enables efficient training and serving of millions of LoRA-based LLM policies by keeping a single large base model resident while dynamically managing lightweight adapter revisions, thereby achieving significant improvements in scalability, memory efficiency, and deployment speed across dense and MoE architectures.

Original authors: Mind Lab, :, Song Cao, Vic Cao, Andrew Chen, Kaijie Chen, Cleon Cheng, Steven Chiang, Kaixuan Fan, Hera Feng, Huan Feng, Arthur Fu, Jun Gao, Hongquan Gu, Aaron Guan, Nolan Ho, Mutian Hong, Hailee Hou

Published 2026-05-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mind Lab, :, Song Cao, Vic Cao, Andrew Chen, Kaijie Chen, Cleon Cheng, Steven Chiang, Kaixuan Fan, Hera Feng, Huan Feng, Arthur Fu, Jun Gao, Hongquan Gu, Aaron Guan, Nolan Ho, Mutian Hong, Hailee Hou, Peixuan Hua, Charles Huang, Miles Jiang, Nora Jiang, Yuyi Jiang, Qiuyu Jin, Fancy Kong, Andrew Lei, Kyrie Lei, Alexy Li, Lucian Li, Ray Li, Theo Li, Zhihui Li, Jiayi Lin, Kairus Liu, Kieran Liu, Logan Liu, Xiang Liu, Irvine Lu, Maeve Luo, Runze Lv, Pony Ma, Verity Niu, Anson Qiu, Vincent Wang, Rio Yang, Maxwell Yao, Carrie Ye, Regis Ye, Wenlin Ye, Josh Ying, Danney Zeng, Yuhan Zhan, Anya Zhang, Di Zhang, Ruijia Zhang, Sueky Zhang, Ya Zhang, Wei Zhao, Ada Zhou, Changhai Zhou, Yuhua Zhou, Xinyue Zhu, Murphy Zhuang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you run a massive, incredibly expensive library of books (the Base Model). These books are huge, take up a whole room, and are very hard to move.

In the old way of doing things, if you wanted to teach a book a new trick—like how to solve math problems or write legal contracts—you had to photocopy the entire book, write the new notes in the margins, and then move that whole new, heavy copy to a different room to show people. If you wanted to teach 1,000 different tricks, you'd need 1,000 heavy copies of the book, taking up a massive amount of space and time to shuffle around.

MinT (MindLab Toolkit) changes the game. Instead of moving the whole book, MinT says: "Let's keep the heavy book right where it is, locked in the library. We'll just write the new tricks on small, lightweight sticky notes (called LoRA adapters)."

Here is how MinT works, broken down into three simple ideas:

1. The "Sticky Note" System (Scale Down)

Instead of moving a 100-pound encyclopedia, you only move a 1-ounce sticky note.

The Old Way: To update a model, you merge the notes back into the book, creating a new heavy file. This takes forever to save and move.
The MinT Way: You keep the heavy book (the Base Model) sitting in the computer's memory. When you train a new skill, you just save the tiny sticky note.
The Result: Moving the "update" is incredibly fast. The paper found that moving just the sticky note was 18 times faster for a medium-sized model and almost 3 times faster for a giant model compared to moving the whole book. It's like sending a text message instead of mailing a brick.

2. The "Switching Stations" (Scale Up)

Imagine you have a giant, complex machine (a super-computer) that can only hold one heavy book at a time.

The Problem: Usually, if you want to train 5 different versions of a policy (e.g., one for math, one for coding, one for law), you need 5 different machines, each holding a copy of the heavy book. That's a waste of money.
The MinT Solution: MinT acts like a smart switchboard. It keeps the heavy book loaded in the machine. When it's time to train the "Math" version, it swaps in the "Math" sticky note. When it's time for "Coding," it swaps that note out and puts in the "Coding" note.
The Result: You can train many different "policies" on the same machine without needing more computers. The paper showed this made training 1.77 times faster for a 4-billion-parameter model and 1.45 times faster for a 30-billion-parameter model, all without needing extra memory.

3. The "Smart Catalog" (Scale Out)

Imagine a library with a million different sticky notes, but your reading desk only has room for a few at a time.

The Problem: If a user asks for a specific sticky note, and it's not on the desk, you have to go to the storage room, find it, and bring it back. If 1,000 people ask for 1,000 different notes at once, the storage room gets clogged, and everyone waits in a long line.
The MinT Solution: MinT organizes this like a smart delivery service.
- The Catalog: It can keep track of one million different sticky notes (policies).
- The Cache: It keeps the most popular notes on a shelf near the desk (CPU cache) so they are ready instantly.
- The "Packing" Trick: Sometimes, a sticky note is actually made of thousands of tiny, tiny pieces (like a puzzle). MinT glues these pieces together into a single, neat package before sending them to the desk. This makes the delivery 8.5 times faster because the delivery truck doesn't have to stop 37,000 times to pick up tiny puzzle pieces; it just picks up one box.

The Big Picture

MinT is essentially a manager for AI training. It stops you from wasting time and money by moving giant files around. Instead, it keeps the heavy "brain" (the Base Model) in one place and manages the thousands of tiny "skills" (Adapters) that get attached to it.

For Training: It lets you switch between different skills quickly without reloading the whole brain.
For Serving: It lets you serve millions of different skills to users, only loading the specific ones needed at that exact moment, and doing it in a way that doesn't clog the system.

In short: Don't move the elephant; just move the mouse. MinT makes it possible to run millions of different AI personalities on a shared, expensive foundation without the foundation ever having to move.

Technical Summary: MinT (MindLab Toolkit)

Problem Statement

As Large Language Models (LLMs) evolve toward trillion-scale parameters and lifelong learning from experience, the infrastructure required for post-training has become a critical bottleneck. Traditional workflows rely on materializing full fine-tuned checkpoints for every model variant, task branch, or tenant-specific adaptation. This approach creates significant scalability challenges:

Resource Inefficiency: Copying or serving full checkpoints for each policy variant consumes excessive storage and memory, making continuous training and deployment of agentic capabilities difficult.
Training-Serving Mismatch: Moving full models between training and inference engines introduces latency and complexity, particularly when managing version control, rollback points, and diverse policy populations over shared base models.
Scalability Limits: As models grow to 1T+ parameters (dense and Mixture-of-Experts/MoE), the overhead of managing full checkpoints for thousands of LoRA (Low-Rank Adaptation) policies becomes prohibitive.

Methodology: The MinT Framework

MindLab Toolkit (MinT) proposes a managed infrastructure system that treats exported LoRA adapter revisions as the fundamental unit of policy management, rather than full model checkpoints. The system is designed around a "resident base" architecture where the base model remains loaded in memory, while task behaviors, product branches, and experimental versions are represented by lightweight adapter revisions.

Core Architecture

MinT separates the policy record (service-owned lifecycle state) from the adapter revision (the executable payload).

Adapter-Centric Lifecycle: Instead of merging adapters back into the base model, MinT exports trained LoRA parameters as serving-compatible revisions. These revisions are checked for compatibility with the resident base and loaded directly into inference engines (e.g., vLLM) that already hold the base weights.
Time-Sliced Training: MinT supports multi-LoRA training on shared resident bases. A single trainer can switch between different policy records by swapping only the LoRA tensors, optimizer states, and rollout records, keeping the frozen base model resident in GPU memory. This supports both single-worker PEFT and distributed Megatron training paths.
Three-Tier Serving Cache: To manage large policy populations, MinT separates addressability from residency:
- Addressable Catalog: A durable store of all exported adapter revisions (up to $10^6$ scale).
- CPU Cache: Local to a serving actor, holding hundreds of adapters for quick promotion.
- GPU Batch: The active set of adapters resident in inference slots for the current decoding step.
Cold Loading as Service Work: First-touch adapter loads are treated as scheduled service work. MinT deduplicates load requests for the same missing policy and uses packed MoE LoRA tensors to reduce object fanout, significantly speeding up live engine loading.

Scaling Axes

MinT scales this adapter-revision path along three dimensions:

Scale Up: Supports LoRA RL on frontier-scale dense and MoE architectures (validated up to 1T parameters). It handles model-parallel training (Tensor/Expert Parallelism), sparse routing consistency (MoE router replay), and dynamic sparse attention (DSA) corrections.
Scale Down: Minimizes the training-serving handoff by moving only the adapter (often <1% of the base model size). This eliminates full-checkpoint materialization entirely.
Scale Out: Expands the policy namespace while bounding engine-local execution. It decouples durable policy addressability from hot working sets, enabling large catalogs without requiring all adapters to be resident simultaneously.

Key Contributions

Adapter Lifecycle Management: MinT defines a complete lifecycle from update to serving, including sharded-to-serving conversion, compatibility checks, rollout records, and rollback capabilities. The adapter revision becomes the fixed behavior selected by the system.
Large-Scale Multi-LoRA RL Training: The system demonstrates end-to-end LoRA RL on large dense and MoE deployments, including 235B-A22B scale training and a 1T-class MoE countdown-task path, supporting both single-worker and distributed Megatron training.
Policy-Population Serving: MinT serves exported adapters through shared-base vLLM engines, separating durable addressability from CPU/GPU hot sets. It introduces packed MoE LoRA representations that reduce small-object fanout, improving live engine loading speeds.
Public Reproducibility: MinT provides a Tinker-compatible API and uses "mint-cookbook" recipes to reproduce SFT, preference optimization (DPO), rollout-based RL (GRPO), and AutoResearch examples through the same adapter lifecycle.

Experimental Results

The paper validates MinT's capabilities through extensive measurements on Qwen3 (4B, 30B, 235B) and Kimi K2 (1T) models:

Handoff Efficiency (Scale Down):
- Adapter-only handoff reduces the measured handoff step by 18.3× on a 4B dense model and 2.85× on a 30B MoE model compared to merging full checkpoints.
- Concurrent multi-policy GRPO shortens wall time by 1.77× (4B) and 1.45× (30B) without increasing peak memory.
Training Performance:
- MinT successfully trained LoRA policies on a 235B-A22B MoE model and a 1.04T-parameter Kimi K2 model, maintaining consistency in MoE router replay and DSA correction.
- Learning traces show that the adapter lifecycle supports SFT, DPO, and GRPO paradigms with comparable performance to full fine-tuning baselines (e.g., GRPO accuracy rising from 0.11 to 0.47 on AIME24).
Serving Scalability (Scale Out):
- The system supports addressable catalogs of up to 100,000 entries in single-engine sweeps, with extrapolated support for $10^6$ policies at the fleet level.
- Cold Loading Optimization: Packed MoE LoRA tensors reduced tensor objects from 37,248 to 672, resulting in an 8.5–8.7× speedup in live engine loading (median <0.2s) compared to unpacked loads.
- CPU caches successfully held hundreds of adapters (e.g., 550 loaded adapters under weak locality), while GPU batches were bounded to ~64 distinct adapters per step.

Significance and Claims

The paper claims that MinT makes multi-tenant training services a practical reality for large-scale post-training. By shifting the managed unit from full checkpoints to adapter revisions, MinT enables:

Cost Reduction: Eliminating full-checkpoint materialization and reducing data movement overhead.
Reproducibility: Providing a consistent, service-level interface for training, evaluation, and serving that hides distributed complexity.
Scalability: Enabling the management of million-scale LoRA policy catalogs over shared 1T-class base models without requiring every policy to be a full-model server.

The authors position MinT not as a new training algorithm, but as an infrastructure layer that makes the "adapter-revision path" usable for frontier-scale dense and MoE architectures, addressing the specific systems challenges of continuous training, deployment, and agentic reinforcement learning.