Optimal Regularization for Performative Learning

Technical Summary: Optimal Regularization for Performative Learning

Problem Statement
In performative learning, the data distribution reacts to the deployed model, creating a feedback loop where strategic users adapt their features to game the system or self-fulfilling prophecies alter outcomes (e.g., in loan approvals or predictive policing). This dynamic complicates standard supervised learning, as optimizing for the current data distribution may steer the distribution toward a new, potentially worse state. While existing methods attempt to estimate and optimize for these performative effects, they often rely on repeated deployments and precise characterization of the distribution shift, which is infeasible in high-dimensional settings or when deployment is limited to a single instance after training (Repeated Risk Minimization, or RRM). The paper addresses the challenge of mitigating these effects without explicitly estimating the distribution shift, focusing instead on the role of regularization.

Methodology
The authors investigate the impact of ridge regularization in linear regression under performative effects, analyzing two distinct regimes:

1. Population Regime: A setting with sufficient data to exactly recover the regression coefficients at each deployment. The authors model the performative effect as a linear shift in the label ($y = x^\top \theta^*_{pop} + x^\top D\theta + w$), where $D$ encodes the performativity. They derive the fixed point of the RRM sequence and analyze the excess risk as a function of the regularization parameter $\lambda$.
2. Over-Parameterized Regime: A high-dimensional setting where the number of features $p$ exceeds the number of samples $n$ ($p/n = \kappa > 1$). Here, the authors employ tools from high-dimensional statistics to establish a deterministic equivalent of the risk at the performative fixed point. This equivalent depends only on the population covariance, the ground-truth parameters, the performative matrix $D$, and the regularization strength, effectively averaging out the randomness of finite samples.

The analysis distinguishes between predictive features (which correlate with the true label) and spurious features (which do not), examining how performativity interacts with both, particularly when they are correlated via the covariance matrix.

Key Contributions and Results

• Population Regime Findings:

  • The optimal regularization parameter $\lambda^*_{pop}$ is shown to be proportional to the strength of the performative effect ($\bar{b}$).
  • Regularization mitigates the performance loss caused by performativity. Specifically, zero excess risk can be achieved if the covariance is isotropic and the performative effect is uniform across features.
  • In the presence of complex covariance structures or highly variable performative effects, the risk remains significant, but optimal regularization still provides the best possible mitigation.
  • The paper notes that in some cases, the optimal regularization can be negative (i.e., "anti-regularization"), a phenomenon previously observed in over-parameterized settings but here shown to occur in the population setting due to performativity.

• Over-Parameterized Regime Findings:

  • The authors derive a deterministic equivalent for the risk, revealing a nuanced phenomenology regarding the direction of optimal regularization:
    • Low Noise: When noise variance is small, the optimal regularization moves in the same direction as the performative effect on predictive features.
    • High Noise: When noise variance is large, the optimal regularization moves in the opposite direction to the performative effect. This is interpreted as the model reverting to its "prior" when noise dominates.
  • Risk Improvement: Contrary to the population setting where performativity worsens risk, the paper finds that in the over-parameterized regime, if performativity reinforces existing trends (positive $\bar{b}$), the optimally regularized risk can actually improve compared to the non-performative case.
  • Spurious Features: The effect of performativity on spurious features is generally attenuated by the factor $\rho^2$ (the correlation between predictive and spurious features), making its impact on optimal regularization and risk less pronounced than on predictive features.

• Empirical Validation:

  • Experiments on synthetic data and real-world datasets (Housing and LSAC) confirm the theoretical predictions.
  • In the population-like setting (large $n$), optimal regularization increases with performative strength, and risk increases with performativity.
  • In the over-parameterized setting (small $n$), the behavior aligns with the high-noise theoretical prediction: optimal regularization decreases as performative strength increases, and risk improves.
  • The findings extend beyond ridge regression to other regularizers (Dropout, Lasso, Elastic Net) and even to neural networks in a strategic classification setting, suggesting the robustness of the phenomenon.

Significance and Claims
The paper claims to be the first to utilize high-dimensional statistical tools to study performative learning. Its primary significance lies in demonstrating that regularization is a principled, tractable, and agnostic mechanism for coping with performative effects, even when the exact nature of the distribution shift is unknown.

The authors argue that while estimation-based methods are limited to low-dimensional or toy examples, regularization offers a practical solution for real-world scenarios where deployment is costly and repeated. The work suggests that practitioners can anticipate performative shifts and adjust regularization accordingly (scaling it with the expected strength of the effect) to improve model robustness and performance. The paper maintains a modest tone, acknowledging that while the theory focuses on linear models and random data, the empirical results on real-world data and neural networks suggest these insights generalize, though further work is needed to explore more complex data and models.