Ensemble learning for time series forecasting using metaheuristic optimisation

Predicting the future is a human instinct, but predicting the future of numbers is a notoriously difficult science. In the world of finance, time series forecasting is the attempt to guess what a stock price will be tomorrow based on what it was yesterday, last week, and last year. The data is rarely a straight line; it is jagged, noisy, and constantly shifting, much like trying to forecast the weather in a storm. Because the market is so volatile and unpredictable, relying on a single method to make a prediction is often like trying to navigate a ship with only one compass. If that compass is slightly off, the entire journey is compromised. This is why researchers have turned to "ensemble learning," a strategy where multiple different prediction models are built and then combined. Instead of trusting one expert, the system listens to a panel of experts, hoping that their combined wisdom cancels out individual mistakes and reveals a clearer path forward.

The challenge, however, lies in how to combine these experts. If you simply ask every model to guess and take the average, you might miss the fact that one model is currently performing brilliantly while another is struggling. The real trick is to figure out, at any given moment, which models deserve more trust and which should be listened to less. This is the specific problem tackled by a team of researchers from India who developed a new way to manage these groups of prediction models. They created two distinct systems, one inspired by the way birds flock and another based on a mathematical approach to handling uncertainty, to dynamically decide how much weight to give each model's prediction. Their goal was to build a system that could adapt in real-time, shifting its focus to the most reliable predictors as market conditions changed.

To test their ideas, the researchers turned to the Indian stock market, a vast and complex financial ecosystem that has seen dramatic shifts over the last fifteen years, including major economic reforms and a global pandemic. They gathered historical data for seven different entities: two major market indices that track the overall health of the economy, and five of the largest companies listed on the exchange. They did not rely on just one type of prediction engine. Instead, they built a "panel" of three different artificial intelligence models, each with its own strengths. One model was designed to find patterns in static data, another was built to remember long-term trends over time, and a third was a streamlined version of that memory-based model, designed to be faster. These three models were trained on the past fifteen years of data to learn how to predict the next day's closing price.

Once these three models were ready, the researchers applied their two new management systems to combine their predictions. The first system, which they called Ensemble PSO, works by simulating a swarm of particles searching for the best solution. Imagine a group of birds flying together; each bird knows its own best position and the best position found by the whole flock. They constantly adjust their flight path to move toward the best spot. In this computer system, the "birds" are different combinations of weights assigned to the three models. The system runs thousands of simulations, constantly adjusting the mix until it finds the perfect balance that minimizes the error between the predicted price and the actual price. The second system, Ensemble ARO, takes a different approach. It treats the uncertainty of the market as a variable to be managed mathematically, finding a set of weights that remains robust even if the market behaves unexpectedly. It essentially asks, "What is the safest way to combine these predictions so that we don't get caught off guard?"

The researchers compared these two new systems against the individual models, a simple average of all three, and other advanced forecasting methods that have been used in the past. To make a fair comparison across all seven different datasets, they developed a new way to score the results. Instead of looking at just one number, like how far off the prediction was on average, they looked at four different measures of accuracy and combined them into a single, comprehensive score. This allowed them to rank the models not just on one dataset, but across the entire board of seven different stocks and indices.

The results were clear and consistent. The system using the swarm-based optimization, Ensemble PSO, consistently outperformed every other method, including the individual models and the other ensemble techniques. It proved to be the most reliable predictor across the different market conditions and time periods tested. The second-best performer was the system using the robust optimization approach, Ensemble ARO, which also significantly outperformed the standard models. The study found that while individual models like the long-term memory networks were good at their specific tasks, they often struggled when the market became too volatile or unpredictable. By dynamically adjusting the weights, the new ensemble systems were able to lean on the model that was performing best at that specific moment, effectively smoothing out the errors and providing a more accurate forecast.

The researchers concluded that their approach offers a powerful tool for financial forecasting. By using metaheuristic optimization to manage the weights of different models, they created a system that is more adaptable and accurate than relying on a single algorithm or a simple average. The findings suggest that in a world of noisy and changing data, the best way to predict the future is not to find the single perfect model, but to build a flexible team of models that can be guided by an intelligent system to work together. This work provides a concrete step forward in how we might use artificial intelligence to navigate the complexities of the stock market, offering a method that is both scalable and robust enough to handle the real-world unpredictability of financial data.