Algorithm Design and Physician Liability

Technical Summary: Algorithm Design and Physician Liability

Problem Statement
The paper investigates the equilibrium consequences of deployment-facing liability regulations on the design and adoption of clinical artificial intelligence (AI). While existing literature often treats algorithm design and human adoption as separate domains, this work addresses their interdependence. Specifically, it examines how liability rules—such as the U.S. Centers for Medicare & Medicaid Services (CMS) Section 1557 provision regarding clinical decision support—reshape incentives for both AI firms (upstream) and physicians (downstream). The core tension arises when a single algorithm delivers unequal accuracy across patient groups (e.g., advantaged vs. disadvantaged), creating a liability exposure for physicians who rely on such disparate tools for disadvantaged patients. The paper asks whether such liability rules successfully induce disparity-reducing investments or inadvertently distort adoption and design in ways that harm the very groups they aim to protect.

Methodology
The authors construct a two-stage game-theoretic model involving an AI firm and a physician.

1. Players and Structure:

   • AI Firm: A profit-maximizing entity that designs a clinical algorithm for two patient types: an advantaged group ($x$) and a disadvantaged group ($y$). Improving accuracy for group $y$ is more costly ($\kappa_y > \kappa_x$) due to data sparsity and measurement frictions. The firm chooses between a disparate design ($\rho_x > \rho_y$) and an equal-accuracy design ($\rho_x = \rho_y$).
   • Physician: An expert intermediary who decides whether to use the AI tool for a specific patient case and, conditional on use, whether to follow its recommendation. The physician balances patient health outcomes against non-clinical objectives (reimbursement $r$) and liability exposure ($\ell$).
   • Patient: Two types with observable characteristics. The physician holds a prior belief ($\alpha$) about the appropriate treatment, which is updated via Bayes' rule upon receiving an AI signal.

2. Liability Mechanism:
   The model incorporates a specific liability channel: if a physician follows an incorrect recommendation from a disparate algorithm for a disadvantaged patient ($y$), they face a liability cost $\ell$. This cost does not apply if the algorithm has equal accuracy across groups or if the patient is from the advantaged group.

3. Solution Concept:
   The game is solved via backward induction. First, the physician's optimal usage and treatment decisions are derived given fixed algorithmic accuracies. Second, the AI firm anticipates these downstream behaviors to choose the optimal accuracy levels and design regime (disparate vs. equal) to maximize profit.

Key Contributions

• Integration of Design and Adoption: The paper bridges the gap between literature on algorithmic design under constraints and literature on human-AI interaction, demonstrating how downstream regulatory constraints feed back into upstream product design.
• Liability as a Dual-Edged Sword: It formalizes how liability rules intended to protect disadvantaged groups can paradoxically reduce their access to AI by increasing the "shadow cost" of reliance for physicians.
• Non-Monotonicity: The study identifies a non-monotone relationship between liability levels and AI utilization for disadvantaged patients, driven by the interplay between direct deterrence and endogenous design responses.
• Welfare Implications of Mandates: It challenges the assumption that mandating equal algorithmic accuracy inherently improves welfare, showing that such mandates can distort investment incentives and utilization decisions, potentially harming both patient groups.

Key Results

1. Disparate Use of AI:
   Even when the algorithm's accuracy is fixed, the liability rule induces disparate use. Physicians are less likely to use AI for disadvantaged patients because the expected legal exposure acts as an additional cost. Consequently, the policy intended to shield disadvantaged patients from unequal performance may limit their access to AI altogether.

2. Non-Monotone Effect of Liability:
   The impact of increasing liability ($\ell$) on AI use for disadvantaged patients is non-monotone:

   • Low Liability: The direct deterrence effect dominates. Physicians reduce AI use for disadvantaged patients, widening the utilization gap.
   • Intermediate Liability: The firm responds endogenously by reallocating investment to improve accuracy for the disadvantaged group to preserve profitability. This reduces the physician's expected liability exposure, potentially restoring or increasing AI use.
   • High Liability: Beyond a threshold ($\bar{\ell}$), the firm switches to an equal-accuracy design to eliminate the disparity-triggered liability channel entirely. In this regime, utilization becomes independent of liability levels.

3. Appropriate vs. Inappropriate Use:
   The paper defines "appropriate use" relative to an altruistic physician ($\theta=0$).

   • Under equal-accuracy designs, reimbursement incentives ($r$) can lead to overuse of AI for both groups, as the liability constraint is removed.
   • Under disparate designs, the physician may underuse AI for disadvantaged patients at intermediate liability levels due to fear of legal exposure, even when clinical uncertainty is high.
   • A tri-phase pattern (overuse $\to$ underuse $\to$ overuse) can emerge for disadvantaged patients as liability increases.

4. Welfare Effects of Equal-Accuracy Mandates:
   Mandating equal accuracy ($\rho_x = \rho_y$) does not guarantee welfare improvement.

   • Accuracy Trade-off: Because improving accuracy for the disadvantaged group is costlier, parity is often achieved by significantly reducing the accuracy for the advantaged group while only modestly increasing it for the disadvantaged group.
   • Utilization Distortion: The mandate relaxes the liability constraint for disadvantaged patients, potentially inducing overuse where clinical gains are small relative to the cost of use ($c$). Simultaneously, it reduces use for advantaged patients who lose accuracy benefits.
   • Aggregate Harm: In intermediate parameter regions (specifically regarding reimbursement rates and liability levels), a uniform accuracy requirement can reduce aggregate welfare for both patient groups.

Significance and Claims
The paper claims that regulatory design is crucial because the same legal objective can generate vastly different equilibrium outcomes depending on incentive structures. The authors argue that:

• Liability standards aimed at reducing algorithmic disparity must be paired with complementary instruments (e.g., reimbursement design, utilization guidelines) to ensure appropriate clinical reliance.
• Equalizing measured algorithmic performance is distinct from ensuring appropriate clinical reliance; the former does not automatically achieve the latter.
• The mechanism described is generalizable beyond healthcare to any domain where expert intermediaries (e.g., judges, lenders) are accountable for decisions informed by algorithms that exhibit group-specific performance disparities.

The paper concludes that without addressing the joint incentives of design and deployment, policies focused solely on algorithmic performance metrics may fail to improve patient welfare and could inadvertently exacerbate disparities in access and outcomes.