Multi-Task GRPO: Reliable LLM Reasoning Across Tasks

Imagine you are training a super-smart robot assistant to solve all kinds of puzzles, from math problems to logic riddles. You want this robot to be a true generalist, capable of handling anything you throw at it without getting confused or failing at the hard stuff. In the world of artificial intelligence, this is called "reasoning," and the robots are called Large Language Models (LLMs). To make these models better, scientists use a technique called "Reinforcement Learning," which is like giving the robot a high-five (a reward) when it gets an answer right and a gentle "try again" when it gets it wrong. One popular way to do this is called GRPO, which is a clever method for figuring out exactly how much the robot should learn from each try. However, there's a catch: if you ask the robot to learn ten different types of puzzles at once, it often gets stuck. It might get really good at the easy puzzles but completely ignore the hard ones, or it might get stuck on a specific type of riddle while forgetting how to do the others. The big question is: how do we train one brain to be equally good at everything, ensuring no single skill gets left behind?

This is exactly the problem tackled by a new method called Multi-Task GRPO (MT-GRPO). Think of the standard training process like a teacher trying to teach a class of students with very different learning speeds. If the teacher just focuses on the "average" student, the smart kids might get bored while the struggling kids fall further behind. The standard approach often lets the easy tasks "hijack" the training, making the model look good on average while secretly failing at the difficult challenges. The authors of this paper realized that simply averaging the scores doesn't make a reliable robot; you need to make sure the worst performer in the class is also getting better.

To fix this, the researchers invented a two-part strategy. First, they created a dynamic "weighting system." Imagine a coach who constantly watches the scoreboard. If the robot is doing great at "Countdown" (a math game) but terrible at "Zebra puzzles" (logic riddles), the coach immediately shifts the focus. They start giving the robot more practice problems from the Zebra category and fewer from the math category. But here's the tricky part: sometimes the robot tries a Zebra puzzle and gets a "zero" score because it didn't even try to solve it properly, meaning it learns nothing from that attempt. If the coach just counts the number of problems, they might think they are teaching Zebra puzzles, but in reality, the robot is only learning from the math ones because the Zebra attempts were "zero-gradients" (empty learning).

To solve this, the second part of their invention is a "Ratio-Preserving Sampler." This is like a very strict librarian who ensures that even if the robot throws away the useless practice attempts, the final pile of useful practice problems still has the exact right mix of math and logic puzzles. If the coach wants 30% of the practice to be logic puzzles, the librarian makes sure that after filtering out the junk, 30% of the remaining good problems are still logic puzzles. This prevents the easy tasks from accidentally taking over the training session just because the hard tasks have more "failed" attempts.

The results of this new method are quite promising. When the team tested it on a mix of three different reasoning tasks, their new MT-GRPO method improved the accuracy of the worst-performing task by 16–28% compared to the standard method, and by 6% compared to another advanced method called DAPO. Even more impressively, it reached a 50% accuracy threshold on the hardest tasks using 50% fewer training steps than the competition. In a larger test with nine different tasks, the method continued to outperform others, showing that it can scale up to handle more complex and diverse challenges. The authors found that by adjusting a "knob" (a parameter called $\lambda$), they could control how much the model prioritizes fixing its weakest skills versus just getting better on average.

In short, this paper suggests that by being smarter about which problems the robot practices and ensuring those problems actually make it to the training session, we can build AI that is not just good at the easy stuff, but reliably competent across the board. It's a step toward creating AI that doesn't just specialize in one narrow field but can truly reason through a wide variety of real-world problems without leaving any of its skills behind.