Predicting Infant Nonattendance at the Next Recommended Well-Child Visit: Model Development and Validation

This study demonstrates that a simple logistic regression model using routine electronic health record data can effectively predict infant nonattendance at well-child visits with performance comparable to complex algorithms, offering a practical tool for clinicians to proactively support at-risk families.

The Gist

Imagine a pediatrician's office as a train station. Every month, a specific train (a "Well-Child Visit") is scheduled to leave for a child's health journey. These trains are crucial because they carry vaccines, check growth charts, and spot developmental delays early.

However, many families miss their trains. When they do, they often end up taking emergency "taxis" (urgent care visits) later, or worse, they miss the chance to catch a problem before it becomes a crisis.

This paper is about building a smart weather forecast for that train station. Instead of just waiting for people to miss their trains and then chasing them down, the researchers wanted to know: "Who is likely to miss the next train while they are still standing here at the station?"

Here is the breakdown of their work using simple analogies:

1. The Goal: Predicting the "No-Show"

The researchers wanted to create a tool that could look at a family currently sitting in the waiting room and say, "Hey, based on what we know about you, there's a high chance you might miss your next appointment."

If the doctor knows this before the family leaves the office, they can offer help right then and there—like helping schedule a ride, setting up a reminder, or connecting them with a social worker. It's about fixing the problem before it happens, rather than reacting after the fact.

2. The Data: The "Digital Fingerprint"

To build this forecast, the team looked at the electronic health records (EHR) of thousands of babies from two different clinics in Chicago. Think of the data as a digital fingerprint of each family's habits.

They didn't just look at medical history; they looked at "process" clues, such as:

• The Schedule: How far in advance was the appointment made? (Last-minute bookings are riskier).
• The History: Has this family missed a train before?
• The Timing: Is the baby due for a visit at a tricky time (like 6 months vs. 2 months)?
• The Attitude: Did they refuse a vaccine at the last visit? (This signals hesitation).

3. The Experiment: The "Toolbox" Test

The researchers tried three different types of "computers" (algorithms) to see which one was best at predicting who would miss the next visit:

• The Simple Calculator (Logistic Regression): A straightforward math formula.
• The Decision Tree (Random Forest): A computer that asks a series of "Yes/No" questions to make a guess.
• The Super-Brain (XGBoost): A highly complex, powerful AI model.

The Surprise: They expected the "Super-Brain" to win. But, the Simple Calculator performed just as well as the complex ones!

Why does this matter? Imagine trying to fix a leaky roof. You could hire a team of rocket scientists with laser scanners (the complex AI), or you could use a reliable hammer and a level (the simple calculator). If the hammer works just as well, you should use the hammer because it's cheaper, easier to carry, and easier for the roofer to understand. In this case, the "Simple Calculator" is easier to plug into a doctor's computer system without breaking it.

4. The Results: The "Top 6 Clues"

The study found that the simple model only needed six clues to make a good prediction:

1. Timepoint: How old the baby is.
2. Delay: How late the current visit was compared to the schedule.
3. Past No-Shows: Have they missed appointments before?
4. Lead Time: How many days in advance was the appointment booked?
5. New Patient: Is this their first time at this specific clinic?
6. Vaccine Refusal: Did they say "no" to shots recently?

These aren't deep secrets about a family's finances or race; they are behavioral signals that a doctor can see right in front of them.

5. The Big Picture: From "Chasing" to "Catching"

Currently, most clinics operate like a postman who only delivers a letter after the package is lost. They call you only after you've missed the appointment.

This study suggests a shift to being a traffic controller. By spotting the red flags while the family is still in the office, the doctor can say, "I see you're new here, and you missed your last shot. Let's make sure we get your next appointment locked in and maybe send you a text reminder."

The Takeaway

This research proves that we don't need expensive, complicated AI to solve this problem. We just need to look at the simple, everyday patterns of how families interact with the clinic. By using a simple, easy-to-build tool, doctors can spot families who are struggling to stay on track and give them a helping hand before they fall behind, ensuring every child gets the care they need.

Technical Summary

1. Problem Statement

Well-child visits (WCVs) are critical for pediatric preventive care, including developmental monitoring, immunizations, and parental education. However, missed appointments are common and lead to preventable acute care visits, delayed developmental interventions, and health disparities.

• Current Limitations: Existing interventions are largely reactive (re-engaging families after a missed visit) or uniform (sending generic reminders to all patients), which are resource-intensive and often ineffective for high-risk families.
• The Gap: There is a lack of tools that use routine Electronic Health Record (EHR) data to proactively identify families at risk of missing their next scheduled visit while they are still present in the clinic. Furthermore, there is limited evidence on whether complex Machine Learning (ML) models offer significant advantages over simpler statistical models for this specific task in a real-world clinical setting.

2. Methodology

Data Sources and Cohorts

The study utilized visit-level data from two pediatric practices within the Advocate Health system in Chicago:

• Practice A (Training/Internal Validation): North side of Chicago; predominantly White/Asian population; 1,215 patients (3,654 visits).
• Practice B (External Validation): South side of Chicago; predominantly Black/Hispanic population; 1,271 patients (3,044 visits); high Medicaid enrollment.
• Outcome Definition: A "missed" visit was defined as the failure to complete the next recommended WCV before the subsequent recommended timepoint (e.g., missing the 9-month visit if no visit occurred between 6 and 9 months).

Feature Engineering

The models utilized data available at the time of the current visit to predict the next visit's attendance. Features included:

• Visit Context: Appointment timepoint (age in months), schedule lead time (days in advance), visit delay (time since due date), new patient status.
• Historical Utilization: Prior no-shows, prior missed WCVs, prior emergency department (ED) visits.
• Clinical Factors: Immunization refusal (current or prior), birth history (NICU admission, preterm, low birth weight).
• Demographics: Sex, insurance type (Medicaid vs. non-Medicaid). Note: Race/ethnicity and preferred language were assessed but excluded from final models due to lack of predictive utility or to avoid potential bias amplification.

Model Development

Three distinct algorithms were compared:

1. Regularized Logistic Regression (LASSO): Selected for interpretability and ease of clinical integration.
2. Random Forest: A tree-based ensemble method.
3. XGBoost: An optimized gradient boosting algorithm.

Technical Handling:

• Class Imbalance: Addressed using Synthetic Minority Over-sampling Technique (SMOTE) for tree-based models and balanced class weights for logistic regression.
• Validation Strategy: Nested group $k$-fold cross-validation (grouped by patient) to prevent data leakage. Practice A served as the training set; Practice B served as the external validation set.
• Feature Selection: Permutation importance was used to prune non-informative features (e.g., preferred language was removed).
• Evaluation Metrics: Average Precision (preferred over AUC for imbalanced data), ROC curves, Precision-Recall curves, Sensitivity (Recall), Specificity, and F1 score (optimized for threshold selection).
• Interpretability: SHAP (SHapley Additive exPlanations) values were calculated to determine feature importance and directionality.

3. Key Results

Performance Metrics

• Missed Visit Rates: 16.2% in Practice A (Training) and 20.7% in Practice B (External).
• Discrimination (AUC): All models performed similarly in external validation, with AUCs ranging from 0.66 to 0.68. Random Forest and XGBoost showed marginally higher AUC and Average Precision than Logistic Regression, but the differences were negligible.
• Classification Performance (F1 Threshold):
  • Logistic Regression achieved the highest F1 score in the internal validation set, indicating the best balance between precision and recall.
  • In external validation, all models showed similar F1 scores, with Random Forest performing slightly better.
  • Recall (Sensitivity): Ranged from 0.54 to 0.71 in external validation.
  • Precision: Ranged from 0.30 to 0.33 in external validation.
  • Negative Predictive Value (NPV): High, suggesting the models are reliable at identifying families who will attend.

Key Predictors (SHAP Analysis)

Across all models, the most influential predictors for non-attendance were:

1. Timepoint: Younger age at the current visit.
2. Visit Delay: Longer delay between the recommended date and the actual visit.
3. Prior No-Shows: History of missing appointments.
4. Schedule Lead Time: Shorter time between scheduling and the visit.
5. New Patient Status: Being new to the department.
6. Immunization Refusal: Documented refusal of vaccines.

4. Key Contributions

1. Proactive vs. Reactive Shift: The study demonstrates the feasibility of predicting future non-attendance during the current visit, enabling clinicians to intervene before a lapse occurs (e.g., scheduling follow-ups, addressing barriers) rather than reacting after a missed appointment.
2. Simplicity vs. Complexity: The findings challenge the assumption that complex "black box" models are necessary for this task. The LASSO Logistic Regression model performed comparably to Random Forest and XGBoost while offering superior interpretability and easier integration into standard EHR workflows (e.g., Epic).
3. Process-Oriented Predictors: The model relies heavily on process measures (scheduling lead time, visit delays) rather than static demographics, suggesting that operational interventions (e.g., better scheduling practices) could directly improve attendance.
4. External Validation: The model was validated across two distinct demographic and geographic populations within the same health system, demonstrating robustness across different patient profiles (North vs. South Chicago).

5. Significance and Implications

• Clinical Workflow Integration: Because the top predictors are easily accessible process measures (often auto-populated in EHRs), the Logistic Regression model can be embedded directly into clinical workflows without requiring complex computational licenses or manual data entry.
• Resource Optimization: The high Negative Predictive Value allows health systems to safely de-prioritize intensive outreach for low-risk families, focusing limited resources (e.g., care coordinators, social workers) on the 30–40% of families identified as high-risk.
• Equity Considerations: By identifying structural barriers (like visit delays and scheduling issues) rather than relying solely on demographic proxies, the tool offers a pathway to address disparities without amplifying bias.
• Future Directions: The authors note the need for ongoing monitoring to prevent model drift over time and suggest future integration with other reminder systems (texts, calls) to create a multi-modal support strategy.

Conclusion: The study establishes that predicting infant WCV nonattendance is feasible using routine EHR data. A simple, interpretable logistic regression model offers a practical, scalable solution for clinical decision support, potentially transforming preventive care from a reactive to a proactive model.