Empirically Understanding the Value of Prediction in Allocation

1. Problem Statement

Public institutions increasingly rely on predictive systems to allocate scarce social resources (e.g., public housing, job training, cash transfers). Historically, research has focused on solving a pre-defined allocation problem (e.g., optimizing a decision rule given fixed constraints and data).

However, the authors argue that the more critical "meta-design" question is which allocation problem to solve. Planners face a vast design space where they must choose between competing policy levers:

• Prediction: Improving data quality, collecting new features, or refining models.
• Capacity: Expanding the number of resources available (e.g., more housing units, more caseworker hours).
• Treatment Quality: Improving the intervention itself (e.g., increasing cash transfer amounts or program efficacy).

Existing theoretical frameworks exist but rely on stylized assumptions and specific comparisons (e.g., aggregate prediction vs. aggregate capacity). There is a lack of empirical tools to help practitioners determine the relative value of these levers in their specific, real-world contexts with unique cost structures, distributions, and welfare objectives.

2. Methodology

The authors propose an empirical framework and a corresponding software toolkit called rvp (Relative Value of Prediction) to simulate and compare policy levers.

A. Formal Definitions

• Allocation Problem: Defined by a distribution $D$ over features $x$ and outcomes $w$, a utility function $u$, and a capacity constraint $\alpha$. The goal is to find a policy $\pi$ that maximizes expected utility subject to $\alpha$.
• Policy Levers: Interventions that modify the primitives of the problem:
  • Expanding Capacity: Relaxing $\alpha$.
  • Improving Treatment: Modifying the utility function $u$ (e.g., increasing transfer size).
  • Improving Prediction: Modifying the joint distribution $D$ to make features $x$ more informative about $w$ (e.g., reducing prediction error).

B. The rvp Toolkit Approach

The toolkit allows planners to simulate scenarios based on three levels of cost information availability:

1. Fixed Budget Allocation (Q1): When costs for all levers are known, the tool optimizes the allocation of a total budget $B$ across levers to maximize total welfare gain.
2. Break-Even Analysis (Q2): When costs are known for one lever but uncertain for another, the tool calculates the "break-even" point. It determines how much improvement the uncertain lever (e.g., prediction) must achieve to match the welfare gain of a known investment (e.g., capacity expansion).
3. Relative Value Ratio (Q3): When cost information is unavailable for both, the tool compares the ratio of welfare gains. This establishes a threshold: e.g., "Prediction is worth investing in if its cost is less than $X$ times the cost of capacity expansion."

C. Simulation Techniques

• Interpolation: The toolkit simulates prediction improvements by interpolating between current model performance and perfect prediction ($\tilde{p} = p + \eta(w - p)$), where $\eta$ controls the magnitude of improvement.
• Subgroup Targeting: It allows for targeted improvements (e.g., improving prediction only for a specific demographic or for individuals near the decision boundary).

3. Key Contributions

1. Conceptual Shift: Moves the focus from "how to solve an allocation problem" to "which allocation problem design yields the best welfare."
2. Empirical Toolkit (rvp): Provides the first open-source software framework enabling practitioners to derive context-specific conclusions about the value of prediction without relying on stylized theoretical assumptions.
3. Generalizability: The framework handles diverse utility functions (step functions, CRRA utility), continuous treatment intensities, and varying cost structures.
4. Data Availability: The authors release the rvp toolkit and parts of the data used in their case studies to facilitate future research.

4. Results from Case Studies

Case Study 1: German Employment Services (Long-Term Unemployment)

• Context: Allocating support to job seekers at risk of long-term unemployment (LTU).
• Finding 1 (Subgroup Value): For job seekers over 35 with missing employment history, collecting additional data (via interviews) is only worth the cost if it reduces prediction error (RMSE) by at least 20% (to match population averages). However, if data collection is targeted only at individuals near the decision threshold, the break-even RMSE reduction drops to <5%, making it a highly efficient investment.
• Finding 2 (Harm vs. Benefit): When allocation can cause harm (e.g., "lock-in" effects where training delays employment for those likely to find work anyway), the value of prediction increases dramatically. In low-capacity regimes where misallocation is harmless, capacity expansion usually dominates. However, if misallocation causes harm, better prediction becomes the priority even in low-capacity settings, as it prevents harmful interventions on low-risk individuals.

Case Study 2: Poverty Targeting in Ethiopia (Cash Transfers)

• Context: Using Proxy Means Tests (PMT) to target cash transfers.
• Finding 1 (Budget Allocation): Under a CRRA utility function (which values transfers to the poorest more highly), the optimal strategy for a new program with a limited budget is a mix of data collection and capacity expansion. As the budget grows, the optimal strategy shifts to 100% data collection to ensure no poor households are excluded due to lack of data.
• Finding 2 (Marginal Investments): The optimal use of additional budget depends heavily on the baseline:
  • If initial data coverage is low, new funds should go to data collection.
  • If initial capacity is low, funds should go to expanding capacity.
  • If initial transfer sizes are small, funds should go to increasing benefit intensity.
• Utility Sensitivity: The value of prediction is significantly higher under CRRA utility (prioritizing the poorest) than under a simple step-function utility (prioritizing any poor household), especially when capacity is constrained.

5. Significance and Implications

• Practical Decision-Making: The paper provides a concrete method for policymakers to move beyond intuition. Instead of assuming "better AI is always better," they can quantify whether investing in data collection, model refinement, or simply hiring more staff yields higher social welfare.
• Context-Dependency: The results demonstrate that the "value of prediction" is not a universal constant; it depends entirely on the specific cost structures, the potential for harm in misallocation, and the specific welfare objectives (e.g., equity vs. efficiency).
• Policy Design: The framework highlights that in many real-world scenarios (especially where misallocation causes harm or where data coverage is sparse), investing in data infrastructure and targeted prediction can be more valuable than simply expanding program capacity.
• Reproducibility: By releasing rvp, the authors lower the barrier for other researchers and institutions to apply these rigorous empirical comparisons to their own domains (e.g., healthcare, education, criminal justice).

In summary, this work bridges the gap between theoretical allocation literature and practical policy implementation, offering a robust empirical toolkit to optimize the design of resource allocation systems.