Technical Summary: Learning to Trust: Bayesian Adaptation to Varying Suggester Reliability in Sequential Decision Making

Problem Statement

Autonomous agents operating in sequential decision-making tasks under uncertainty often benefit from external action suggestions provided by humans or automated systems. However, the reliability of these suggesters is rarely static; it fluctuates due to factors such as human fatigue, sensor degradation, or changing environmental conditions. Existing methods for incorporating advice typically assume that suggester quality parameters are known and static, a limitation that hinders practical deployment in dynamic real-world scenarios. Furthermore, prior approaches often lack mechanisms for agents to proactively decide when to seek guidance, leaving the timing of information acquisition entirely to the suggester. This paper addresses the challenge of dynamically inferring suggester reliability and strategically managing suggestion requests within partially observable environments.

Methodology

The authors propose a unified framework based on Partially Observable Markov Decision Processes (POMDPs) and Mixed Observable Markov Decision Processes (MOMDPs) to handle suggestion uncertainty.

1. Integrating Suggester Reliability into Belief States

The core methodological contribution is the treatment of suggester reliability as a hidden state variable within the agent's belief representation.

• State Space Expansion: The state space is expanded to $S \times T$, where $S$ represents the environmental state and $T$ represents a discrete set of suggester types (e.g., varying rationality coefficients $\lambda$).
• Noisy Rational Model: The framework utilizes a noisy rational model where the probability of a suggestion $\sigma$ given state $s$ and type $\lambda$ is $p(\sigma | s, \lambda) \propto \exp(\lambda Q(s, \sigma))$. Here, $\lambda$ characterizes the suggester's quality, ranging from random ($\lambda=0$) to highly rational.
• Bayesian Inference: Agents update beliefs over both environmental states and suggester types simultaneously using Bayes' rule. The belief update incorporates both the agent's own observations and the received suggestions.
• Dynamic Transitions: To model changing reliability (e.g., operator fatigue), the authors introduce a dynamic suggester model where the suggester type can transition between states with a probability $t_p$. This allows the agent to continuously reassess reliability rather than assuming a fixed type.
• MOMDP Formulation: To mitigate the computational complexity of the expanded state space, the problem is formulated as a MOMDP. Suggester types are placed in the hidden component ($Y$), while directly observable states remain in the fully observable component ($X$).

2. Strategic "Ask" Action

The framework introduces an explicit "ask" action ($a_{ask}$) that allows the agent to proactively solicit suggestions.

• Mechanism: Executing $a_{ask}$ elicits a suggestion from the suggester model but incurs a cost (either a direct reward penalty or a limit on the total number of allowed asks).
• Implementation: The observation space is expanded to include suggestions corresponding to feasible actions. The agent uses pre-computed state-action values ($Q$) from the base POMDP to parameterize the suggestion observation model.
• Constraints: To prevent over-reliance or burdening the suggester, the framework supports two constraints: a direct cost in the reward function and a hard limit on the number of asks per trial (modeled as a counter in the observable state space).

Key Contributions

1. Dynamic Reliability Inference: The paper integrates suggester reliability directly into the agent's belief state, enabling continuous inference and adaptive trust calibration without assuming static suggester quality.
2. Proactive Information Gathering: The introduction of an explicit "ask" action allows agents to strategically balance the informational value of a suggestion against its acquisition cost, moving beyond passive reception of advice.
3. Robustness to Dynamic Conditions: By modeling transitions between suggester types, the framework enables agents to adapt to changing reliability, reducing overconfidence in deteriorating suggesters.
4. Unified Framework: The approach combines reliability inference and query management within a single sequential decision-making framework, addressing a gap where previous methods treated these as separate or static problems.

Experimental Results

The framework was evaluated in the Tag and RockSample domains using the SARSOP algorithm.

• Static Suggesters: Agents maintaining beliefs over multiple suggester types (Multiple-Type or MT agents) achieved performance comparable to agents with perfect knowledge of the true suggester rationality ($\lambda^*$). In contrast, naive agents (those not modeling reliability) suffered significant performance degradation when suggester quality was low.
• Dynamic Suggesters: In scenarios where $\lambda^*$ changed over time, MT agents demonstrated robust adaptation. Specifically, agents modeling dynamic transitions ($t_p = 0.05$) adjusted their beliefs and recovered near-optimal performance significantly faster than static MT agents ($t_p = 0.00$) after a suggester's quality degraded.
• Strategic Asking:
  • In the Tag domain with unlimited asks, MT agents effectively reduced request frequency when interacting with low-quality suggesters (e.g., random suggesters), querying only during initial uncertainty.
  • In RockSample with a cost per ask, agents refrained from requesting suggestions from low-quality sources ($\lambda < 2$) when the cost outweighed the informational gain.
  • Under strict constraints (one ask per trial), MT agents adapted better than fixed-type agents, though performance remained below that of an agent with perfect knowledge.
• Heuristic Suggesters: The framework successfully leveraged a heuristic-based suggester (simulating partial sensor data) that did not strictly adhere to the noisy rational model, demonstrating robustness to deviations from assumed generative mechanisms.

Significance and Claims

The authors claim that this work provides a foundation for adaptive human-agent collaboration by addressing suggestion uncertainty in uncertain environments. The significance lies in:

• Practical Applicability: Moving beyond the assumption of static, known suggester quality to handle the dynamic and unpredictable nature of real-world guidance.
• Strategic Autonomy: Enabling agents to make principled decisions about when to seek help, balancing information gain against costs, which is critical for human workload management and system efficiency.
• Robustness: Demonstrating that agents can maintain high performance even when suggester reliability fluctuates or when suggestions come from non-rational, heuristic sources.

The paper concludes that while the discretization of reliability and the use of a uniform transition model are simplifications, the framework offers a significant step toward more realistic and robust collaborative decision-making systems. Future work is suggested to explore event-driven transition models that correlate reliability changes with specific environmental or operational events.