A Contextual-Bandit Oversight Game with Two-Sided Informational Asymmetry

Technical Summary: A Contextual-Bandit Oversight Game with Two-Sided Informational Asymmetry

1. Problem Statement

The paper addresses the challenge of calibrating human intervention in autonomous systems (robotic or software) when two-sided informational asymmetry exists. Unlike standard models where only one party holds private information, this setting posits:

• Human Private Information ($\theta$): The human supervisor privately knows their reward function (preferences, risk tolerance, and valuation of outcomes).
• AI Private Information ($\omega$): The AI agent privately knows the "quality" or failure modes of its proposed action, derived from observations (e.g., sensor data, code inspection) inaccessible to the human.

The core problem is determining the optimal runtime interaction protocol (a "play/ask/trust/oversee" interface) where the AI chooses whether to defer (ask) and the human chooses whether to override (oversee). The authors argue that existing frameworks fail to capture this specific duality: Cooperative Inverse Reinforcement Learning (CIRL) models one-sided preference uncertainty but lacks the runtime deferral interface and AI-private proposal quality; the Oversight Game models the interface but assumes full information.

2. Methodology

To achieve exact characterizations that remain conjectural in full Partially Observable Markov Decision Process (POMDP) settings, the authors adopt a Contextual-Bandit framework. This removes physical state transitions ($S \to S'$), isolating the value of information and correction from state dynamics, while retaining the common belief as a dynamically controlled state across rounds.

The Model (CB-Oversight-CIRL Game):

• Structure: A finite-horizon (or discounted infinite-horizon) game with i.i.d. contexts $s_t$.
• Types: $\theta \in \Theta$ (human reward type) and $\omega \in \Omega$ (AI observation model type).
• Bilinear Payoff: The team payoff is defined as $f(\theta, \omega) = \langle O_\omega, R_\theta \rangle$, where $O_\omega$ is the distribution of outcomes given the AI's private type, and $R_\theta$ is the human's reward function. This factorization separates the AI's knowledge of what happens from the human's knowledge of what matters.
• Interface:
  1. AI proposes action $a_\sigma$.
  2. AI selects meta-action: play or ask.
  3. If ask, Human selects meta-action: trust or oversee.
  4. Oversee triggers a correction (e.g., shutdown) based on the human's belief.
• Costs: $c_{ask}$ (cost of deferring) and $c_{ov}$ (cost of overriding).

3. Key Contributions

The paper provides three primary contributions:

1. Formal Game Definition: A contextual-bandit team game (Definition 1) that integrates the Oversight Game interface with two-sided asymmetric information, recovering CIRL and Off-Switch Game variants as limit cases.
2. One-Shot Characterizations: Two distinct solutions for a single round ($T=1$):
   • Team Optimum: A global coordinator optimizes both agents' policies jointly. The optimal policy is a deterministic pair of sets $(B, C)$ where $B \subseteq \Omega$ are AI types that ask, and $C \subseteq \Theta$ are human types that oversee.
   • Myopic Rule: A behaviorally natural rule where the human treats the AI's "ask" as uninformative (non-signaling) and acts based on her prior belief.
3. Dynamic Resolution Analysis: A partial multi-round analysis demonstrating how the gap between the myopic and team-optimal policies narrows over time via passive learning (human updates belief from observed outcomes) and active signaling (AI uses "ask" as a credible signal).

4. Key Results

The "Slab" of Avoidable Harm (One-Shot Analysis)
The paper identifies a critical region of failure termed the "slab" in the space of priors $(b, q)$, where $b$ is the probability the human is a "high-stakes" type ($\theta_1$) and $q$ is the probability the action is harmful ($\omega_H$).

• Team Optimum: Asks if $b > b^*$, where $b^* = \frac{c_{ask}}{|f_{1H}| - c_{ov}}$. Crucially, this threshold is independent of $q$. If the AI asks, the human updates to the post-ask posterior (knowing $\omega = \omega_H$) and shuts down.
• Myopic Rule: Asks only if $b > b^*$ AND $q > q^*$, where $q^*$ is a threshold derived from the expected payoff under the prior.
• The Gap: The region $\{b > b^*, q \le q^*\}$ represents the slab. Here, the AI knows the action is harmful and shutdown would be beneficial, but the myopic human, trusting her prior that the action is likely safe ($q$ is low), declines to oversee. Consequently, the AI does not ask, and the harmful action is executed.
• Economic Interpretation: The gap is the "price of non-credible oversight communication." If the ask is a credible signal, the human's prior $q$ becomes irrelevant to the decision to shut down; if not, the prior $q$ acts as a barrier to intervention.

Multi-Round Dynamics
The paper analyzes how the system escapes the "slab" over repeated rounds:

1. Passive Learning: Even without signaling, if the AI plays the proposal, the human observes outcomes. Under Assumption 2 (informative proposals), the human's belief $q_n$ converges to 1 almost surely. The expected time to exit the failure regime is bounded by Wald-type inequalities involving the KL divergence ($\eta$) between outcome distributions.
2. Active Signaling: If the AI adopts a separating policy (ask iff $\omega = \omega_H$), the human's belief jumps to 1 immediately upon observing an "ask." However, under a one-period-lagged myopic response (where the human cannot react to the ask in the same round), this signaling is only beneficial if $b > b^{**} = b^*/\gamma$. The AI pays the cost $c_{ask}$ in the current round to secure effective oversight in all future rounds.

5. Significance and Scope

The paper claims that its primary significance lies in formalizing the two-sided informational asymmetry inherent in autonomous systems where agents possess private knowledge of failure modes.

• Theoretical Contribution: It provides exact, closed-form characterizations of optimal deferral policies in a setting that would otherwise require intractable POMDP analysis.
• Practical Insight: It demonstrates that "non-credible" communication channels (where an AI's request for help is not trusted as a signal of danger) create a specific region of avoidable harm. The solution is not necessarily better AI or better humans, but an interface design that makes the "ask" a credible signal, thereby collapsing the myopic failure region.
• Limitations: The authors explicitly state that the results rely on a contextual-bandit structure (no state transitions) and product-measure priors. The optimal multi-round policy, the effects of correlated beliefs, and the extension to full POMDPs with Markov dynamics remain open problems. The analysis of multi-round dynamics is partial, comparing specific mechanisms (passive vs. active) rather than characterizing the global optimum.