Using dynamic loss weighting to boost improvements in forecast stability

Imagine you are a weather forecaster trying to predict the temperature for next week. Every morning, you get a new piece of data—a fresh temperature reading from yesterday. Naturally, you update your prediction for next Tuesday. But here's the catch: every time you update, your prediction for next Tuesday might jump up or down wildly. One day you say it will be 70°F, the next day 65°F, then 72°F. This constant wiggling is called "forecast instability." It's annoying for anyone trying to plan a picnic or a supply chain because they can't trust the numbers to stay put.

Scientists have been trying to build computer models that are not only accurate (getting the temperature right) but also stable (sticking to their story). Recently, researchers discovered a way to teach these models to be a bit more steady by adding a "stability penalty" to their training. Think of it like a teacher telling a student, "Don't just get the answer right; don't change your answer every time you see a new hint." However, the old method of teaching this was a bit rigid. It used a fixed rulebook that said, "Pay 10% attention to stability and 90% to accuracy," and that ratio never changed during the whole training process.

This paper asks a simple, curious question: What if we made the rulebook dynamic? What if the computer could learn to focus on accuracy first, and only start worrying about stability once it was good at the basics? The authors, a team from KU Leuven, tested this idea using a powerful forecasting model called N-BEATS. They tried out several "dynamic loss weighting" techniques—methods that let the computer shuffle the importance of accuracy versus stability as it learns. They found that by letting the model adjust its own priorities on the fly, they could get forecasts that were just as accurate but much less jittery than before. It's like teaching a student to stop second-guessing themselves once they've mastered the lesson, resulting in predictions that people can actually trust.

The Story of the Wiggly Forecast

In the world of time series forecasting, computers are like crystal balls that look at the past to guess the future. But these crystal balls have a bad habit: they are fickle. When new data arrives, the model might completely flip-flop on what it predicted for next week. This is known as rolling origin forecast instability. Imagine you are planning a party. If your friend tells you, "I think 10 people will come," and then five minutes later says, "Actually, maybe 20," and then an hour later, "Wait, maybe 5," you'd be pretty frustrated. You need a prediction that is reliable enough to order the right amount of pizza.

For a long time, the goal was just to make the prediction as accurate as possible. But in the real world, accuracy isn't the only thing that matters. If a forecast jumps around too much, people stop trusting it. They might start making their own "gut feeling" adjustments, which often makes the forecast less accurate. So, researchers wanted to teach models to be "stable" without losing their "accuracy."

The Old Way: A Static Rulebook

Previously, a method called N-BEATS-S was introduced to solve this. It worked by giving the computer a two-part homework assignment.

1. Part A: Get the numbers right (Accuracy).
2. Part B: Don't change your answer too much when you get new data (Stability).

The computer had to balance these two tasks. The old way of doing this was to use a static hyperparameter (let's call it $\lambda$). This is like a dial that the human trainer sets before the computer starts learning. If you set the dial to 0.025, the computer is told, "Hey, focus 97.5% on being accurate and only 2.5% on being stable." If you set it to 0.275, you tell it, "Okay, pay more attention to stability."

The problem with this static dial is that it's set in stone for the entire training process. The computer doesn't get to decide when to focus on what. It's like telling a student, "Spend 20% of your time on math and 80% on history," for the entire semester, even if they are already a math genius and just need to practice history a little bit. The researchers suspected that a rigid rulebook might be holding the computer back.

The New Idea: A Dynamic Dance

The authors of this paper wondered: What if the computer could change the dial while it was learning? This is called Dynamic Loss Weighting (DLW). Instead of a fixed rule, the computer gets to decide, "Okay, right now I'm terrible at accuracy, so I'll ignore stability for a moment. But once I'm good at accuracy, I'll start worrying about stability."

They tested several existing methods that do this kind of dynamic balancing, and they also invented a new one called Task-Aware Random Weighting (TARW).

• GradNorm: This method tries to make sure the computer learns both tasks at the same speed. If it's learning accuracy too fast, it slows that down to focus on stability.
• Uncertainty Weighting: This method asks, "How unsure am I about this task?" If the computer is very unsure about stability, it pays less attention to it.
• Random Weighting (RW): This is a bit chaotic. It just picks a random number between 0 and 1 for the stability dial every single time the computer learns a new batch of data. Surprisingly, this randomness often works well.
• TARW (The New Kid): The authors took the "Random Weighting" idea and tweaked it. Instead of picking a number between 0 and 1, they picked a number between 0 and a specific limit (let's call it $\kappa$). This ensures the computer never gets too obsessed with stability and forgets to be accurate. It's like saying, "You can be random, but don't go crazy."

What They Found: The Sweet Spot

The team ran these experiments on two massive datasets of monthly time series (the M3 and M4 datasets), which contain thousands of different time series like sales figures or weather data. They compared the new dynamic methods against the old static method (N-BEATS-S with a fixed dial).

Here is what the numbers told them:

1. Stability Boost: Almost all the dynamic methods made the forecasts much more stable. The "wiggles" in the predictions went down significantly. For example, on the M4 dataset, some methods reduced the instability metric (RMSSC) from around 0.537 (the old static method) down to 0.146 or 0.224. That's a huge drop in jitteriness.
2. Accuracy Held Steady: The big fear was that by focusing on stability, the computer would get worse at being accurate. The results showed that for the best methods, accuracy stayed almost exactly the same. In fact, on the M3 dataset, the new TARW high method actually produced forecasts that were more accurate than the old N-BEATS model, with a score (RMSSE) of 1.082 compared to 1.088.
3. The Winner: The TARW method (specifically the "high" version) stood out. It managed to improve stability significantly without hurting accuracy. It was like finding a way to make the forecast both rock-solid and super precise.
4. The "Why": When they looked under the hood, they saw that the dynamic methods were doing exactly what the authors hoped. In the beginning of training, the computer focused heavily on accuracy. Only after it got good at that did it start paying attention to stability. The static method, by contrast, was trying to do both from day one, which sometimes confused the model or forced it into a "local optimum" (a stuck spot where it's stable but not very accurate).

The Takeaway

The paper suggests that we don't need to force a computer to balance accuracy and stability with a rigid, pre-set rule. Instead, letting the model dynamically adjust its priorities—specifically by prioritizing accuracy first and then introducing stability—leads to better results.

The authors propose TARW as a simple, effective tool for this. It's essentially a "smart randomizer" that keeps the computer from getting too obsessed with stability too early. While the computational cost during training is slightly higher (the computer has to do a tiny bit more math to adjust the weights), the cost during actual use (inference) is exactly the same as the old methods.

In the end, this research suggests that if you want a forecasting system that people can trust, you shouldn't just ask it to be accurate. You should teach it to be consistent, but you have to let it learn that lesson at the right time. By using these dynamic methods, we can get forecasts that are not only right but also reliable, saving businesses and planners from the headache of constantly changing numbers.