Who Benefits from AI? Self-Selection, Skill Gap, and the Hidden Costs of AI Feedback

1. Problem Statement

The paper addresses a critical gap in the literature regarding Artificial Intelligence (AI) and organizational learning. While existing research (often based on controlled experiments with forced exposure) suggests AI improves performance and learning, it largely ignores endogenous selection: the reality that in real-world settings, individuals choose when, how, and if to seek AI feedback.

The authors investigate three core questions:

1. Endogeneity: Does the apparent learning from AI feedback reflect a causal effect of the technology, or is it driven by pre-existing individual traits (motivation, skill) that drive both the decision to use AI and performance improvement?
2. Skill Gap: How does endogenous selection into AI use affect the distribution of skills across a population? Does AI act as an equalizer or a multiplier of existing inequalities?
3. Intellectual Diversity: Does reliance on a centralized AI source for feedback lead to the homogenization of strategies and a decline in collective intellectual diversity?

2. Methodology and Data

The study utilizes a unique, large-scale field setting with high observability of individual choices.

• Data Source: Data from Lichess.org, a free, non-profit online chess platform.
  • Sample: 52,251 human players over a 5-year period.
  • Observations: 403,010 game-level observations (games played by humans against AI bots).
  • Context: Players can voluntarily use "Stockfish" (the world's strongest chess AI) to analyze their games move-by-move. This provides high-quality, expert-level feedback.
• Key Variables:
  • Dependent Variables: (1) Whether a player sought AI feedback for a specific game; (2) Player performance (move accuracy calculated via Stockfish); (3) Population-level intellectual diversity (measured via the Gini coefficient of opening move strategies).
  • Independent Variables: AI feedback usage, game outcome (win/loss), player skill (Elo rating), and motivation proxies.
• Empirical Strategy:
  • Control Function Approach (Instrumental Variables): To address endogeneity in feedback seeking, the authors use a control function approach with four instruments:
    1. Average losses in the previous five games (proxy for ambition).
    2. Time elapsed between games (proxy for engagement/motivation).
    3. Inexperienced user dummy (proxy for feature awareness).
    4. Opponent strength.
  • Generalized Random Forests (GRF): Used to estimate heterogeneous treatment effects of AI feedback across the skill distribution.
  • Regression Discontinuity in Time (RDiT) & Natural Experiments: To establish causality for population-level diversity, the authors analyzed 42 platform updates (24 related to AI features, 18 unrelated) as natural experiments. They compared diversity metrics before and after these updates using a stacked difference-in-differences framework.

3. Key Results

A. Endogeneity and the "Illusion" of Effectiveness

• Self-Selection: Motivated and higher-skilled individuals are significantly more likely to seek AI feedback, particularly after losses (failures).
• Null Causal Effect: Once endogenous motivation and effort are accounted for using the control function approach, AI feedback has no average causal effect on individual learning.
  • The observed performance gains are entirely driven by the unobserved traits (motivation, discipline) that cause individuals to seek feedback in the first place.
  • Nuance: Feedback on losses appears beneficial, while feedback on wins appears detrimental (potentially due to overconfidence or reactive usage), but these effects vanish when controlling for selection bias.

B. Widening of the Skill Gap

• Heterogeneous Effects: While the causal effect of AI is zero, the realized relationship between AI use and performance is positive for high-skilled users and negligible for low-skilled users.
• Mechanism: Higher-skilled individuals use AI more frequently and more productively (e.g., analyzing losses). This creates a "competence-motivation spiral" where existing advantages are compounded.
• Outcome: Access to AI feedback increases the skill gap rather than closing it. AI acts as a complement to high skill and motivation, not a substitute for them.

C. Decline in Intellectual Diversity

• Convergent Updating: The study finds a causal link between AI feedback and reduced intellectual diversity.
• Evidence: Using the 42 natural experiments, the authors show that platform updates related to AI features cause a significant decrease in the diversity of opening strategies played on the platform.
• Mechanism: As individuals rely on a single, centralized AI source (Stockfish), they converge on the same "optimal" strategies. This leads to "specialization" where players narrow their repertoires, reducing the collective pool of diverse problem-solving approaches.

4. Key Contributions

1. Decoupling Access and Effort: The paper introduces the concept of "access-effort decoupling." It demonstrates that while AI lowers the barrier to access feedback, it does not lower the barrier to productive engagement (effort/motivation). Consequently, observed benefits are often selection artifacts.
2. Micro-Macro Linkage: It bridges individual learning dynamics with collective outcomes. It shows how individual choices to optimize via a shared algorithmic signal aggregate into macro-level consequences: increased inequality (skill gap) and reduced diversity (homogenization).
3. Re-evaluating AI as an Equalizer: The findings challenge the prevailing narrative that AI will level the playing field. Instead, in voluntary adoption settings, AI amplifies existing inequalities and creates new collective risks (loss of cognitive diversity).

5. Significance and Implications

• For Organizational Learning: Simply deploying AI tools does not guarantee organizational learning. Without addressing the motivational and skill-based selection processes, organizations may see "illusory" gains while their workforce becomes more stratified and less diverse in thought.
• For Policy and Management:
  • Training: Interventions must focus on training employees how to use AI productively (e.g., seeking feedback on failures) rather than just providing access.
  • Inequality: Policymakers should be wary of assuming AI will reduce labor market inequality; without targeted interventions, it may exacerbate the divide between high-skill/high-motivation workers and others.
  • Diversity Risk: The concentration of cognitive influence in centralized AI models poses a risk of "epistemological homogenization," where diverse human perspectives are replaced by a single algorithmic standard.
• Theoretical Impact: The study extends the exploration-exploitation tradeoff literature, identifying a new mechanism where shared algorithmic signals drive individual specialization, leading to collective stagnation in exploration.

In summary, the paper argues that AI feedback is not a magic bullet for learning. Its benefits are contingent on the user's pre-existing motivation and skill, leading to a scenario where the "haves" (skilled/motivated) get richer in capability, while the collective intelligence of the population becomes more uniform and less diverse.