Validating the Crystal Ball: Retrospective Evaluation and Uncertainty-Aware Forecasting of Severe Housing Cost Burden in New York City

In the complex landscape of American cities, where the price of a home often dictates the shape of a community, there is a single number that policymakers watch with intense scrutiny: the share of renter households spending more than half their income on housing. When a family reaches this threshold, they are considered to be under a "severe housing cost burden," a condition that leaves little room for food, healthcare, or savings. For decades, city officials have relied on government surveys to track this burden, but these surveys arrive with a significant delay, often telling leaders about the past rather than the present. To bridge this gap, researchers have turned to machine learning, training computers to predict future housing costs based on historical patterns. The hope has been that these digital forecasts could act as a crystal ball, guiding resources to the neighborhoods that need them most before a crisis fully takes hold.

A new study from researchers at institutions including Shahjalal University of Science and Technology and Indiana Wesleyan University puts this crystal ball to the test. The team revisited a specific machine learning model that had previously generated forecasts for New York City, projecting how severe housing cost burdens would evolve from 2023 through 2025. The original model was praised for its ability to fit past data, but it had never been checked against what actually happened. Now that the real-world data for those years has arrived, the researchers performed a retrospective evaluation, comparing the model's predictions directly with the actual outcomes recorded by the government. The result was a clear and systematic failure: the model had been consistently too optimistic. It predicted that housing would remain more affordable than it actually did, underestimating the severity of the burden in every single borough of the city.

The discrepancy was not a small fluctuation; it was a growing chasm. On average, the original forecasts missed the mark by 3.4 percentage points. This means that if the model predicted 30 percent of renters would be severely burdened, the reality was closer to 33.4 percent. The error did not stay constant; it widened over time. In 2023, the model was off by about 2 percentage points, but by 2025, that gap had nearly doubled to 4.7 percentage points. The situation was most acute in the Bronx, the borough with the highest concentration of housing stress, where the model underestimated the burden by 5.4 percentage points. The researchers traced this failure not to the machine learning algorithm itself, which performed well when looking at past data, but to the assumptions fed into it. The original forecast assumed that rents would rise by a steady 3 percent each year and incomes would grow by 2.5 percent. In the real world, however, rents accelerated faster than expected after the pandemic while incomes stagnated, causing the model's projections to drift further from reality with each passing year.

Recognizing that a single prediction is rarely enough to capture the volatility of housing markets, the team rebuilt the forecasting framework to include a measure of uncertainty. Instead of offering a single number, the new approach provides a range of possible outcomes, much like a weather forecast that gives a probability of rain rather than a simple yes or no. They used a statistical technique called conformal prediction to create a 90 percent confidence band around every forecast, ensuring that the true value would fall within that range most of the time. They also accounted for the natural margins of error in the government data itself. This recalibrated system was then used to generate new forecasts for the years 2025 through 2028 under three different scenarios: a baseline scenario, a "rent-income parity" scenario where incomes finally catch up to rent, and an adverse scenario where rents surge while incomes stall.

The findings from these new, uncertainty-aware forecasts paint a sobering picture for the future of New York City. Even in the most optimistic scenario, where income growth matches rent growth, the study suggests that severe housing cost burdens will remain higher in 2028 than they were in 2022 across all boroughs. The post-pandemic shift in the housing market appears to have created a new, higher baseline of cost that is not easily reversed. Under the adverse scenario, where rents continue to outpace incomes, the situation becomes critical, particularly in the Bronx, where nearly 45 percent of renter households could face severe cost burdens by 2028. The researchers also demonstrated that adding specific data about individual buildings, such as the number of rent-stabilized units or the volume of housing code complaints, significantly improved the accuracy of the predictions, validating the need for more granular data in future models.

Ultimately, this work serves as a vital correction for the field of urban forecasting. It proves that while machine learning models can be powerful tools, they are only as reliable as the assumptions they are built upon. A model that cannot see a sudden shift in the economic landscape will inevitably produce a forecast that looks good on paper but fails in practice. By exposing these blind spots and introducing methods to quantify uncertainty, the researchers have provided city planners with a more honest and useful tool. The path forward involves not just better algorithms, but a willingness to publish the assumptions behind every prediction and to regularly check those predictions against the reality they are meant to describe. For New York City, and potentially for other major cities facing similar pressures, this means preparing for a future where housing stress remains high, regardless of how quickly incomes might eventually rise.