Technical Summary: Proximal State Nudging (PSN)

Problem Statement

The paper addresses the issue of skill atrophy in shared-autonomy systems, where human operators relying on AI assistance may experience a gradual decline in their own capabilities. This phenomenon poses safety risks in semi-autonomous domains (e.g., aviation, surgery, driving) because operators may fail to distinguish their own inputs from autonomous corrections, leading to an inability to intervene effectively when automation fails.

Current approaches to shared autonomy typically present a binary choice: rely heavily on AI for immediate task performance and safety, or remove AI entirely to encourage learning. The authors argue that existing methods often fail to balance these goals, either prioritizing safety at the expense of long-term skill retention or failing to provide sufficient scaffolding for learning. Furthermore, prior works attempting to balance these factors often rely on task-specific assumptions (e.g., access to near-optimal Q-values) or lack grounding in cognitive psychology theories regarding human learning.

Methodology: Proximal State Nudging (PSN)

The authors propose Proximal State Nudging (PSN), a learning-aware shared autonomy algorithm designed to minimize skill atrophy while maintaining task performance. The method is grounded in the Zone of Proximal Development (ZPD) theory from cognitive psychology, which posits that learning is most productive when users face challenges matched to the boundary of their current ability.

Core Components

1. State Learnability Estimator ($\phi_{zpd,t}(s)$):

   • PSN estimates the "learnability" of a state $s$, defined as the likelihood that a state supports the student's skill improvement.
   • The estimator is operationalized by calculating the difference in expected task reward between a trajectory generated with assistance ($\pi_{shared}$) and one without ($\pi_{student}$).
   • States that are too easy (high reward regardless of assistance) or too hard (low reward regardless of assistance) are assigned low learnability scores.
   • Crucially, this estimator does not require an explicit model of human adaptation or access to the student's internal Q-values, making it applicable to real human learners.

2. Beam Search Planner:

   • PSN modifies the shared control policy by performing a beam search over a set of length-$T$ sampled trajectories starting from the current state.
   • The planner selects the trajectory that maximizes a score function $J_t(\tau)$, which is a weighted sum of:
     • Average Predicted State Learnability: The mean $\phi_{zpd,t}$ across the trajectory.
     • Task Reward: The cumulative reward $R(\tau)$ of the trajectory.
   • The first action of the optimal trajectory is executed.

3. Adaptive Assistance Level ($\alpha$):

   • Unlike standard shared autonomy where the assistance level $\alpha$ is fixed, PSN dynamically tapers $\alpha$ based on the estimated learnability of the current state.
   • The effective assistance is calculated as $\alpha' = \alpha \times (1 - \phi_{zpd,t}(s))$.
   • This ensures that assistance is reduced in "high-learnability" states (encouraging the student to struggle productively) while maintaining higher assistance in states where safety or task completion is critical.

Key Contributions

1. Algorithm: The introduction of a learning-compatible shared autonomy algorithm grounded in cognitive scaffolding theory (ZPD) that does not require explicit student models or Q-value access.
2. Simulated Validation: Empirical validation in the LunarLander environment using simulated students (Double-DQN agents), demonstrating that PSN outperforms existing baselines (Stochastic Q-Bumpers, model-free RL assistance, and standard blending) in balancing unassisted skill improvement with assisted safety.
3. Human Subject Studies: The first human-subject studies of a planner incorporating learning-compatible shared autonomy. The study involved 60 participants across two realistic driving tasks in the CARLA simulator:
   • High Performance Racing (HPR): Minimizing lap time.
   • Parallel Parking: Precise spatial reasoning and planning.

Experimental Results

Simulated Learners (LunarLander)

• Learning: PSN achieved significantly higher unassisted task rewards and success rates compared to all baselines, including the "No AI Assistance" condition (which learned slowly).
• Safety: PSN maintained high assisted task rewards comparable to standard shared autonomy baselines and significantly reduced the number of crashes during the learning phase compared to unassisted practice.
• Efficiency: The method successfully navigated the Pareto frontier between safety and learning without assuming Q-learning updates.

Human Subject Studies (CARLA)

• High Performance Racing:
  • Learning: PSN resulted in up to 7x larger gains in unassisted skill performance (measured by lap time, similarity to expert, and control jerk) compared to standard Blended Shared Autonomy.
  • Safety: PSN incurred 50% fewer collisions during practice compared to unassisted self-practice.
  • Qualitative: Participants using PSN reported learning specific turning strategies, whereas self-practice participants focused narrowly on lap time and ignored smooth control.
• Parallel Parking:
  • Learning: PSN showed significant improvements in parking time and collision reduction, particularly in out-of-distribution scenarios (unfamiliar starting angles), where gains were over 7x compared to baselines.
  • Safety: PSN resulted in significantly fewer collisions during practice than unassisted self-practice (<60% reduction), though standard Blended Shared Autonomy was slightly safer (at the cost of learning).
  • Qualitative: PSN participants described learning new parking strategies, while unassisted participants focused on mechanical sensitivity.

Significance and Claims

The paper claims that Proximal State Nudging successfully resolves the traditional trade-off between immediate task performance (safety) and long-term human skill development. By guiding users toward states estimated to be most "learnable" (the ZPD), PSN prevents skill atrophy without compromising safety.

The authors emphasize that this is the first human-subject study of a shared autonomy planner that explicitly incorporates learning objectives. The results suggest that AI assistance can be structured not just to correct errors, but to actively scaffold human learning, ensuring that operators retain and improve their skills even while relying on automation. The approach is presented as domain-agnostic, with potential extensions to other cyber-physical and cognitive domains, though the current work is limited to rapid-control tasks.