Technical Summary: Expert-Guided g-computation with LLMs for Estimating Causal Effects on Timings

1. Problem Statement

Hospital Quality Improvement (QI) programs frequently face the challenge of prioritizing candidate interventions to optimize hospital flow, specifically targeting the Average Length of Stay (LOS). While numerous interventions are proposed (e.g., improved triaging, workload balancing, AI decision support), existing methods struggle to accurately estimate and rank their causal effects.

Current approaches fall into two categories with distinct limitations:

• Qualitative Approaches: Rely on expert judgment and tools like Gantt charts (or value stream maps) to map patient trajectories. While intuitive, these methods are susceptible to cognitive biases and often fail to accurately predict downstream effects of interventions due to complex interactions between events.
• Quantitative Approaches: Rely on data-driven statistical models (e.g., regression). These fail when interventions are entirely hypothetical (lacking historical data), when the "positivity assumption" is violated (as new policies have never been implemented), or when the causal mechanisms are too complex to be learned solely from tabular data without incorporating clinical reasoning.

The core problem is the lack of a framework that can estimate the "average time saved" ($\tau = E[Y(0) - Y(1)]$) for hypothetical interventions in settings where patient trajectories are heterogeneous, data is limited for new policies, and clinical reasoning is required to understand unstructured causal mechanisms.

2. Methodology: Expert-Guided g-computation (Egg-computation)

The authors propose egg-computation, a hybrid framework that integrates qualitative expert reasoning with quantitative data-driven modeling. The methodology bridges the gap between Gantt charts (used in operations management) and Directed Acyclic Graphs (DAGs) used in causal inference.

2.1 Causal Model over Gantt Charts

The authors formalize a Gantt chart as a causal DAG where:

• Nodes represent event times ($T_k$) or waiting times ($W_k$) between events.
• Edges represent dependencies (e.g., start-start, finish-start).
• Intervention ($A$): A binary node representing the current policy ($A=0$) versus a candidate QI policy ($A=1$).
• Potential Outcomes: $Y(a) = T_K(a)$ represents the LOS under policy $a$.

The model assumes a local Markov property where the waiting time of an event is conditionally independent of non-descendants given its parents and baseline covariates ($X$).

2.2 The Egg-Computation Pipeline

The estimation of the causal effect $\tau$ is achieved by simulating counterfactual patient trajectories ($Y(1)$) through a three-step process that classifies events into three types based on their relationship to the intervention:

1. Type I (Non-descendants of $A$): Events unaffected by the intervention. Their waiting times are copied directly from the observed factual timeline ($W_k(1) = W_k$).
2. Type II (Descendants of $A$, but not direct children): Events downstream of the intervention targets. Their waiting times are imputed using statistical models trained on observational data, conditioned on the (modified) parent timings.
3. Type III (Direct children of $A$ / Intervention Targets): Events directly modified by the policy. Since these are unobserved in the data, their waiting time distributions are supplied by expert judgment (or an aligned LLM).

The final estimator is the average difference between the observed LOS and the simulated counterfactual LOS across the population.

2.3 Scaling with Large Language Models (LLMs)

To make egg-computation feasible at scale, the authors developed an LLM-assisted pipeline to replace human experts in two critical steps:

• Step 1: DAG Construction: The LLM reads unstructured clinical notes and intervention descriptions to:
  • Determine patient eligibility.
  • Construct a patient-specific causal DAG.
  • Classify nodes as Type I, II, or III.
  • Implementation: A five-step prompting strategy (reasoning, node extraction, edge extraction, formatting, validation) ensures rigorous causal reasoning and prevents hallucination.
• Step 2: Timing Simulation:
  • For Type II nodes, the LLM can be used via In-Context Learning (ICL) to predict waiting times based on similar historical cases, or standard regression models can be used.
  • For Type III nodes, the LLM is prompted with detailed intervention specifications (including examples of expected timings) to sample from the expert-supplied distribution $p_{expert}$.

3. Key Contributions

1. Theoretical Framework: The paper establishes the first causal identification framework for Gantt charts, proving that under specific assumptions (Consistency, Exogenous Intervention, Exclusion Restriction, Local Markov, and Expert Accuracy), the average time saved is identifiable via a variant of g-computation.
2. Hybrid Methodology: It introduces "expert-guided g-computation," which strategically allocates tasks: experts (or LLMs) handle unidentifiable components (intervention targets), while data models handle identifiable components (downstream dependencies).
3. LLM Pipeline: It demonstrates a practical, prompt-based pipeline that scales expert reasoning, allowing for the generation of patient-specific DAGs and counterfactual simulations without model retraining.
4. Empirical Validation: The framework is validated through both simulations and a real-world study at an urban safety-net hospital.

4. Results

4.1 Simulation Studies

In simulations where patient causal structures were heterogeneous (varying dependencies between CT and MRI reports), egg-computation significantly outperformed conventional methods:

• Bias Reduction: Methods relying on a single shared DAG incurred large biases (up to 14 hours) when the assumed structure was wrong for a majority of patients.
• Accuracy: Egg-computation using patient-specific DAGs (extracted by the LLM) and exact timing propagation achieved zero bias. Even with imperfect timing models (OLS or kNN), the bias remained low (< 0.4 hours) compared to structural misspecification.

4.2 Real-World Evaluation (Zuckerberg San Francisco General Hospital)

The pipeline was applied to 11 candidate QI interventions across 2,193 inpatient timelines.

• Expert Concordance: LLM-generated DAGs showed high concordance with human QI experts.
  • Specificity for intervention targets: ~96.5%.
  • Precision for downstream edges: ~91.8–94.2%.
  • Impact on Estimates: Re-running egg-computation with LLM-generated DAGs versus expert-corrected DAGs resulted in a mean difference in estimated time saved of only -2.6 to 8.1 hours, with 19 of 24 pairs yielding identical estimates.
• Timing Model Performance: In-Context Learning (ICL) using GPT-5 achieved the lowest Mean Absolute Error (24.1 hours) for predicting downstream waiting times, outperforming traditional regression models (Random Forest, Ridge) and baselines.
• Prioritization Insights:
  • Causal rankings differed significantly from rankings based solely on eligibility rates. For example, "Consult response escalation" was ranked first by causal impact (18.9 hours saved) despite having lower eligibility than "Disposition-critical imaging priority."
  • Clinicians found the causal rankings more aligned with their intuition than eligibility-based rankings.
  • The analysis revealed design flaws in interventions (e.g., early discharge planning was often blocked by unresolvable medical needs), providing actionable insights for intervention redesign.

5. Significance and Claims

The paper claims that egg-computation provides a broadly applicable framework for estimating the average time saved for candidate interventions where causal mechanisms can be represented via Gantt charts.

• Bridging the Gap: It successfully combines the strengths of qualitative reasoning (handling hypothetical scenarios and complex mechanisms) with quantitative rigor (eliminating confounding and leveraging data).
• Scalability: By leveraging LLMs, the framework makes expert-guided causal inference feasible for large-scale hospital operations, overcoming the resource constraints of manual chart review.
• Actionable Intelligence: Beyond providing a single number, the method generates counterfactual Gantt charts that allow clinicians to audit the plausibility of causal reasoning and understand why an intervention might or might not work, facilitating better intervention design.

The authors conclude that while the focus is on hospital LOS, the methodology is applicable to any domain where task dependencies and process timings are critical and can be modeled via Gantt charts.