When accuracy does not improve decisions: a decision-outcome validation framework for cross-regional box office forecasting
This paper introduces a decision-outcome validation framework demonstrating that minimizing predictive error does not guarantee optimal market selection decisions in cross-regional box office forecasting, as models optimized for decision outcomes often outperform accuracy-focused models in generating revenue.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the world of film distribution, a distributor's job is a high-stakes game of resource allocation. They must decide which foreign countries to show a movie in, how many copies to print, and where to spend their limited advertising budget. To make these choices, they rely on forecasts—predictions of how much money a film will make in each territory. For decades, the standard way to judge if a forecasting model is good has been simple: check how close its numbers are to the actual box office results. If a model's predictions are, on average, very close to the real earnings, it is considered accurate and trusted. This approach assumes that being mathematically precise is the same thing as being useful for making business decisions. However, this assumption overlooks a crucial detail: a distributor does not care about the average error across every single country. They care about picking the right few countries that will generate the most revenue. A model can be very accurate on average but still fail to identify the specific top markets, leading a distributor to invest in the wrong places and lose money.
This is the central puzzle tackled by researchers Lu Chao, Xu AnQi, and XiaoXi Ma in their study on cross-regional box office forecasting. They investigated whether the models that are best at predicting exact numbers are also the ones that help distributors make the best choices. To find out, they built a new way of testing forecasts that focuses on the final outcome of the decision rather than just the precision of the prediction. They gathered data on fourteen Chinese films released between 2023 and 2025, tracking their weekly earnings across eighteen different overseas territories. This dataset included seventy-six specific combinations of films and countries, capturing a wide range of performance from massive hits to modest releases. The researchers then compared five different forecasting methods, ranging from simple statistical averages to complex machine learning algorithms, to see which one would actually help a distributor maximize their revenue.
The researchers set up a specific test scenario to mimic a real-world decision. They imagined a distributor with a limited budget who could only promote a film in a small number of countries, perhaps the top three or five. The goal was to see which model would correctly identify those specific high-earning markets. They ran their tests using different methods to ensure the results were robust, including testing the models on films released in future years to see if they could predict new products, and testing them on entirely different regions to see if they could generalize to new places. They compared the results of these decision-based tests against the traditional method of simply checking which model had the smallest average error.
The findings revealed a surprising disconnect. In the majority of the test cases, the model that was best at predicting the exact dollar amount was not the same model that was best at picking the right countries. In one critical test involving films released in 2025, the model that the researchers' new framework identified as the best for decision-making captured nearly 89.3 percent of the possible revenue share. In contrast, the model that was technically the most accurate at predicting numbers captured only 88.7 percent. While this difference might seem small, it translated to a tangible gap of over $152,000 in lost revenue for the distributor. The study showed that the model favored by traditional accuracy metrics often made mistakes in the specific markets that mattered most, while the model that prioritized the correct ranking of markets performed better, even if its raw numbers were slightly less precise.
The researchers traced this problem to the nature of the film market itself. Box office earnings are highly uneven; a few countries generate the vast majority of the money, while many others contribute very little. Traditional accuracy metrics treat every country equally, so a model can be "accurate" by getting the small markets right while completely missing the big ones. Because the decision to enter a market depends on getting the big ones right, a model that minimizes average error can still lead to poor business choices. The study demonstrated that this gap between accuracy and decision quality is not a fluke or a result of bad data, but a systematic issue that appears whenever market sizes vary significantly. The researchers used computer simulations to confirm that when two models have similar overall accuracy but different patterns of error, the one that looks better on paper often performs worse in practice.
This work suggests that the way we validate forecasting models needs to change. Relying solely on error statistics is like judging a navigator only by how close their map is to the terrain, without checking if they actually chose the right path to the destination. The researchers argue that for any forecasting system used to make decisions, the evaluation must include the outcome of those decisions. They propose that researchers and industry professionals should report not just how close a prediction was, but how much value the prediction actually generated when used to select markets. By adopting this decision-focused approach, distributors can avoid the trap of choosing models that look good on a spreadsheet but fail to deliver the revenue they need, ensuring that their limited resources are invested in the markets that truly matter.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.