Human AI Construction of Bayesian Networks for Operational Decision Support -- A Virtual Survey Approach

Technical Summary: Human–AI Construction of Bayesian Networks for Operational Decision Support

Problem Statement
Bayesian Belief Networks (BBNs) are established frameworks for decision-making under uncertainty, combining graph theory and probability theory. However, their adoption in operations research (OR) and management science has been hindered by a specific bottleneck: the elicitation of Conditional Probability Tables (CPTs). Building a BBN requires two distinct steps: structure learning (identifying the Directed Acyclic Graph) and parameter learning (estimating CPTs). While structure learning can be addressed through expert knowledge or data-driven algorithms, parameter learning is computationally and cognitively demanding.

• Expert Elicitation: Relying on human experts to estimate probabilities for nodes with multiple parents creates an exponential growth in required parameters (e.g., a node with four parents and three states each requires 27 independent estimates). This imposes a high cognitive load, often exceeding the bounds of human rationality.
• Data-Driven Learning: Machine learning approaches require large volumes of representative data, which are frequently unavailable in management research contexts (e.g., supply chain risk, project management, or niche healthcare markets).
  Consequently, there is a lack of a scalable "middle path" to parameterize BBNs when real-world data is scarce and expert time is expensive.

Methodology: LLM-Assisted Virtual Survey
The authors propose a six-step methodology that integrates Large Language Models (LLMs) to bridge the gap between expert opinion and data-driven learning. The approach does not use LLMs for structure learning (which remains theory or expert-driven) but utilizes them for parameter elicitation.

1. Structure Definition: The user defines the DAG based on established theory (e.g., Social Cognition Models) or domain expertise.
2. State Specification: Discrete states for each node are defined with semantic descriptions.
3. LLM-Based Virtual Survey (Core Step):
   • Persona Generation: An LLM generates a panel of $N$ synthetic agents (e.g., $N=100$) with diverse demographics and personas relevant to the decision context.
   • Prompt Engineering: For every node and every possible combination of parent states, the system queries the LLM. Each query includes the persona, network context, node states, and a specific description of the variable.
   • Aggregation: The LLM returns probability vectors for each cell of the CPT. To mitigate "hallucinations" and individual response bias, the authors apply a trimmed-mean rule, discarding the top and bottom 10% of estimates and averaging the remaining 80%. The resulting vector is normalized to sum to 1.
4. Visualization and Manual Override: The initial CPTs are visualized, allowing users to manually correct implausible values.
5. Simulation and Refinement: The system performs forward sampling to generate synthetic data from the network. Maximum Likelihood Estimation (MLE) is then applied to this synthetic data to refine the CPTs, ensuring self-consistency and smoothing sampling noise.
6. Inference: The final model supports probabilistic queries (predictive, diagnostic) and causal queries using do-calculus (graph mutilation) to distinguish between correlation and causation.

The methodology is implemented in a three-tier web architecture using a Flask backend, the pgmpy Python library for inference, and provider-agnostic LLM adapters.

Key Contributions

1. Methodological Innovation: The paper introduces a novel framework for eliciting CPTs using a panel of synthetic AI agents and a trimmed-mean aggregation rule. This addresses the cognitive burden of human elicitation and the data scarcity of machine learning.
2. System Integration: The authors developed a modular, three-tier web application that integrates theory-based structure specification, LLM-based parameter elicitation, synthetic data generation, and causal inference.
3. Empirical Application: A numerical study was conducted on a health services decision problem: modeling customer intention to consult an Ayurvedic doctor for diabetes management. The study utilized 8 variables and 99 independent parameters, testing 11 structural assumptions derived from social cognition theory.

Results and Findings
The application to the Ayurvedic consultation model yielded several counter-intuitive insights that highlight the value of causal inference over simple observational analysis:

• Self-Efficacy (SE): Observational data suggested SE was a strong predictor of intention ($P(I=High|SE=High) = 0.837$). However, the causal intervention $P(I=High | do(SE=High))$ revealed a negligible causal lift (+0.9%). The strong observational correlation was driven by confounding factors (specifically, favorable Attitude).
• Subjective Norms (SN): Observational data showed a weak or negative correlation with intention. However, the causal intervention $P(I=High | do(SN=Highly Influenced))$ revealed a significant true causal lift (+7.5%).
• Joint Interventions: The most effective strategy identified was a joint intervention on Self-Efficacy and Subjective Norms, yielding a causal lift of approximately +15%.
• Validation: Of the 11 structural assumptions tested, 7 were validated, 3 were rejected (providing interpretable insights into mediation effects), and 1 was inconclusive. This suggests the LLM-elicited CPTs encoded meaningful domain logic consistent with theory.

Significance and Claims
The paper positions this work as a "middle path" for Operations Research. It claims that:

• Scalability: The methodology allows for the rapid parameterization of BBNs in data-scarce environments at a low cost (the study cost approximately USD 6 for 100 agents).
• Causal Clarity: By integrating do-calculus, the approach enables decision-makers to distinguish between spurious correlations and true causal effects, offering guidance that regression or structural equation modeling cannot provide without strong assumptions.
• Human-AI Synergy: The authors argue that LLMs do not replace human expertise but augment it. Humans define the causal structure and review the outputs, while AI handles the scalable, repetitive task of probability estimation.
• Future Potential: The authors modestly claim that aggregated LLM responses can encode meaningful slices of human reasoning. They suggest this approach opens new avenues for customer-intention modeling and broader operational decision-making, provided that future research validates these synthetic priors against real-world empirical data.

The paper concludes that while full real-world validation is pending, the methodology successfully demonstrates a viable workflow for constructing causal decision models where traditional data collection is impractical.