📈 economics

Beyond Persistence: Understanding Forecastability of Weekly Potato Prices Across Multiple Market Centres

This study of weekly potato prices across 17 Indian markets reveals that while simple persistence models outperform complex algorithms at short horizons, Exponential Smoothing becomes the superior method for longer forecasts, demonstrating that price predictability depends more on forecast horizon and seasonal strength than on model complexity.

Original authors: Muktinath Vishwakarma, Manish Kurhekar

Published 2026-10-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Muktinath Vishwakarma, Manish Kurhekar

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of agriculture, price is the heartbeat of a market. For a farmer, a trader, or a storage operator, knowing what a crop will cost next week or next month can mean the difference between profit and loss. This is especially true for potatoes, a crop that is harvested in bursts, rots quickly if not handled right, and moves through markets with a rhythm that changes with the seasons. Because these prices swing so wildly, experts have long tried to build mathematical models to predict them. The goal is simple: look at the past and guess the future. But there is a stubborn problem in this field. Often, the most complicated computer programs, filled with layers of artificial intelligence, fail to beat a much simpler trick. That trick is to assume that next week's price will be exactly the same as this week's. This assumption, known as persistence, works surprisingly well because prices tend to stick close to where they are from one week to the next. The big question for researchers has been whether adding more complexity to the prediction models actually helps, or if the simple guess is already as good as it gets.

A team of researchers set out to answer this question by looking at the weekly potato prices from seventeen different markets across India over a ten-year period. They did not just pick one market and one method; they tested twelve different ways of forecasting, ranging from basic statistical tools to advanced machine learning and deep learning systems. They asked these models to predict prices one week, four weeks, eight weeks, and thirteen weeks into the future. To make the test fair, they compared every single complex model against that simple "next week is the same as this week" benchmark. They wanted to see if the fancy models could find hidden patterns that the simple guess missed, and if those patterns changed depending on how far ahead they were looking.

The results revealed a clear story about time. When the researchers asked the models to predict just one week ahead, the simple benchmark won almost every time. In fact, for most of the seventeen markets, no complex model could improve upon the basic guess. The prices were so persistent that the most recent number held almost all the information needed for the immediate future. However, as the researchers pushed the prediction further out, the story changed. At four weeks, and even more so at eight and thirteen weeks, the simple guess began to lose its grip. The models that could recognize broader patterns, specifically those that understood how prices move in yearly cycles, started to pull ahead. One particular statistical method, known as exponential smoothing, which is designed to track levels, trends, and seasonal cycles, became the best performer at the longer horizons. It outperformed the simple guess in sixteen of the seventeen markets when looking thirteen weeks ahead.

What was most surprising was that the most advanced technology did not lead the pack. The deep learning models, which are often hailed as the future of prediction, and the machine learning algorithms, which are known for finding complex non-linear relationships, generally failed to beat the simple benchmark. They did not consistently find better answers than the statistical models, and in many cases, they performed worse. The researchers found that the ability to predict prices further into the future was not about having a more powerful computer or a more complex algorithm. Instead, it depended on the nature of the market itself. Markets where prices showed a strong, repeating pattern over the course of a year were much easier to predict at longer distances. In markets where prices moved up and down without a clear seasonal rhythm, the simple guess remained the best option.

The study also highlighted that this ability to predict is not a fixed trait of a market. A market that is easy to predict today might become harder to predict tomorrow, depending on how the prices behave over time. The researchers were careful to note that while the statistical improvements were clear, they were not always strong enough to be considered a guaranteed win for every single market on its own. However, when looking at the group of markets as a whole, the evidence was strong: as the prediction window stretched further, the value of understanding seasonal patterns grew, while the value of just copying the last price faded.

Ultimately, this research suggests that the key to better forecasting is not to build more complex machines, but to understand the specific rhythm of the data. For short-term predictions, the past is the best guide. But for longer-term planning, looking for the seasonal heartbeat of the market is far more useful than trying to force a complicated algorithm to find patterns that aren't there. The study concludes that the best approach is to match the tool to the task: use simple methods when the future is close, and turn to models that understand seasonal cycles when the future is distant, rather than assuming that more complexity always leads to better answers.

Technical Summary: Beyond Persistence: Understanding Forecastability of Weekly Potato Prices Across Multiple Market Centres

Problem Definition
This study addresses the challenge of forecasting weekly potato prices across 17 Indian market centres over the period 2005–2015. While agricultural price forecasting has increasingly adopted complex machine learning (ML) and deep learning (DL) architectures, it remains unclear whether these sophisticated models provide genuine predictive value over simple persistence benchmarks. The core problem is distinguishing between models that merely minimize error and those that extract useful information beyond the "Naive-1" forecast (which assumes the next week's price equals the current week's price). The study specifically investigates how forecastability varies across different forecast horizons (1, 4, 8, and 13 weeks) and whether differences in performance can be explained by the temporal characteristics of the price series, such as persistence, trend, and seasonality.

Methodology
The research employs a rigorous rolling-origin evaluation design to ensure out-of-sample validity.

Dataset: Weekly potato prices (INR/kg) from 17 market centres, comprising 539 observations each (Jan 2005–Apr 2015). The data is split chronologically (70% development, 30% evaluation).
Models: Twelve forecasting methods were evaluated:
- Benchmarks: Naive-1, Seasonal Naive, Drift.
- Statistical: Exponential Smoothing (ETS), ARIMA.
- Machine Learning: Ridge Regression, XGBoost.
- Deep Learning: LSTM, N-BEATS, PatchTST, iTransformer, TCN.
Evaluation Protocol: Forecasts were generated at 1-, 4-, 8-, and 13-week horizons. Performance was measured using Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), Symmetric MAPE, and Mean Absolute Scaled Error (MASE).
Skill Metric: Forecast skill ( $S$ ) was defined relative to the Naive-1 benchmark ( $S = 1 - \frac{MAE_{model}}{MAE_{Naive-1}}$ ). Positive values indicate improvement over persistence.
Statistical Testing: The Diebold–Mariano test (with Harvey–Leybourne–Newbold correction) was used for pairwise comparisons against Naive-1, with Holm's procedure applied to correct for multiple testing. Friedman and Wilcoxon tests were used for cross-market model ranking.
Explanatory Analysis: Five temporal characteristics (lag-1 autocorrelation, trend strength, seasonal strength, normalized volatility, spectral entropy) were calculated to explain cross-market differences in forecast skill.

Key Results

Horizon-Dependent Forecastability:
- Short-term (1 week): Naive-1 is the dominant method, achieving the lowest average error. Only 5 of 17 markets showed any numerical improvement over Naive-1.
- Long-term (4–13 weeks): As the horizon increases, the ability of models to outperform persistence improves significantly. At 13 weeks, 16 of 17 markets showed positive numerical skill.
Model Performance:
- Exponential Smoothing (ETS) emerged as the best-performing model on average for 4-, 8-, and 13-week horizons, significantly outperforming Naive-1 at the 13-week horizon in cross-market statistical tests.
- Machine Learning and Deep Learning: Neither ML (Ridge, XGBoost) nor DL models (LSTM, N-BEATS, Transformers, TCN) provided consistent gains over Naive-1. In fact, most neural architectures performed worse than the simple persistence benchmark across all horizons.
Drivers of Forecastability:
- Persistence: While all markets exhibited strong short-term persistence (lag-1 autocorrelation > 0.90), the degree of persistence did not explain why some markets were more forecastable than others at longer horizons.
- Seasonality: Seasonal strength was the only characteristic significantly associated with forecast skill at longer horizons. The correlation between seasonal strength and forecast skill increased from weak at 1 week ( $\rho \approx 0.16$ ) to very strong at 13 weeks ( $\rho \approx 0.83$ ). Markets with strong seasonal patterns (e.g., Delhi) showed the largest gains over persistence, while those with weak seasonality (e.g., Ahmedabad, Shillong) did not.
Statistical Significance: While numerical improvements were observed at longer horizons, no individual market-horizon pair remained statistically significant after Holm correction for multiple comparisons. However, the cross-market analysis confirmed that ETS performs significantly better than Naive-1 at the 13-week horizon.

Key Contributions and Significance
The paper reframes potato-price forecasting not as a competition to find the most complex model, but as an investigation into the forecastability of the underlying data relative to a strong benchmark.

Beyond Model Complexity: The study demonstrates that for univariate weekly potato prices, increasing model complexity (via ML/DL) does not automatically yield better forecasts. Parsimonious statistical models (ETS) are sufficient to capture the exploitable temporal structure present in these series.
The Role of Horizon: The utility of forecasting models is intrinsically linked to the forecast horizon. Short-term forecasting is dominated by persistence, whereas longer-term forecasting benefits from modeling recurring seasonal structures.
Market Heterogeneity: The study highlights that forecastability is not a fixed property of a market but depends on the interaction between the forecast horizon and the market's specific temporal structure (specifically seasonality).
Methodological Rigor: By employing rolling-origin evaluation and strict statistical testing against a persistence benchmark, the study provides a robust framework for distinguishing between numerical error reduction and genuine, reproducible predictive value.

Conclusion
The authors conclude that the primary challenge in weekly potato-price forecasting is not the lack of powerful algorithms, but the mismatch between model complexity and the information structure of the data. For short horizons, simple persistence is nearly optimal. For longer horizons, the gains are driven by seasonal structure rather than model complexity, and these gains are best captured by standard statistical models like ETS. The study suggests that future improvements in this domain may depend more on incorporating external variables (e.g., weather, arrivals, storage) than on developing more complex univariate architectures.

More like this