Optimising supervised machine learning algorithms predicting cigarette cravings and lapses for a smoking cessation just-in-time adaptive intervention (JITAI)

This study found that while machine learning models can detect smoking lapse risks, their modest overall performance and high inter-individual variability suggest that reducing assessment frequency or simplifying predictors does not consistently improve outcomes, indicating that such algorithms are best used in combination with rules-based approaches rather than as standalone solutions for just-in-time adaptive interventions.

The Gist

Imagine you are trying to help a friend quit smoking. You know that the hardest part isn't the physical addiction, but the sudden, overwhelming urge to smoke that hits at random moments—like when they are stressed, bored, or hanging out with friends.

This study is like a digital safety net experiment. The researchers wanted to build a smart phone app that could predict exactly when these "danger zones" (cravings or slips) would happen, so it could send a helpful message just in time to stop the person from lighting up.

However, there's a catch: If the app asks your friend too many questions (like "How are you feeling?" every hour), they will get annoyed and stop using it. But if it asks too few questions, the app might be too dumb to predict the danger.

The researchers tested different ways to build this "smart predictor" to find the perfect balance between being helpful and not being annoying. Here is how they did it, using some simple analogies:

1. The "Weather Forecast" Analogy (Predicting the Outcome)

The researchers tried to predict two things:

• The "Lapse": Did your friend actually smoke a cigarette? (This is like predicting if it will rain).
• The "Craving": Did your friend feel a strong urge to smoke, even if they didn't? (This is like predicting if the sky looks stormy).

The Finding: It was much easier to predict if it would rain (a lapse) than to predict if the sky looked stormy (a craving). Cravings are like mood swings—they change too fast and are too personal for a computer to guess perfectly.

2. The "Camera Frequency" Analogy (How often to ask questions)

The team tested asking questions at different speeds:

• High Speed: Asking 16 times a day (every waking hour).
• Low Speed: Asking only 3 times a day.

The Surprise: You might think asking more questions gives a smarter computer. But for predicting actual slips (lapses), asking fewer questions actually worked better!

• Why? Imagine trying to spot a thief in a crowded room. If you take a photo every second, you get a blur of motion. If you take a photo every few minutes, you get a clearer picture of who is actually moving.
• For cravings, however, asking fewer questions made the app dumber. You need frequent updates to catch a fleeting feeling like a craving.

3. The "Personal Trainer" Analogy (Using Personal Data)

The researchers tested two types of "trainers":

• The Group Trainer: A coach who knows what works for everyone based on data from 37 people.
• The Personal Trainer: A coach who spends the first few days watching just your friend to learn their specific habits before making predictions.

The Finding:

• For predicting lapses, the "Personal Trainer" (using your friend's own data) did a slightly better job at catching the specific moments they slipped.
• For predicting cravings, the "Personal Trainer" actually did worse. It seems that trying to learn a specific person's habits from just a few days of data confused the algorithm. The "Group Trainer" (general rules) was actually more reliable.

4. The "Backpack" Analogy (How many variables to check)

The app could check 34 different things (mood, location, time of day, pain levels, etc.). The researchers asked: "Do we need to carry all 34 items in our backpack, or can we just carry the top 15?"

The Finding: They could throw away most of the heavy items! The algorithm performed almost the same with just a few key questions (like "Are you stressed?" or "Do you have a cigarette nearby?") as it did with the full list. This is great news because it means the app can be much simpler and less annoying for the user.

The Bottom Line: What Does This Mean for You?

The study found that while computers are getting better at predicting smoking slips, they aren't perfect crystal balls yet.

• The Good News: We can build a simpler app that asks fewer questions (3 to 5 a day) and still catches most "slip" moments.
• The Bad News: The app isn't perfect. It will sometimes send a warning when you aren't in danger (a false alarm), or miss a danger when you are.
• The Solution: Don't rely on the computer alone. The best approach is a hybrid: Use the computer to give a "heads up" based on general rules, but combine it with human wisdom and simple rules (like "If you feel stressed, take a walk").

In short: The researchers found a way to make a "smart safety net" that isn't too heavy to carry, but they warn us not to trust it 100%. It's a helpful tool, but it needs a human hand to guide it.

Technical Summary

1. Problem Statement

Smoking cessation is a critical public health priority, yet traditional behavioral interventions often fail to prevent relapse at critical moments. Just-in-time adaptive interventions (JITAIs) aim to deliver tailored support during high-risk moments (e.g., cravings or lapses) by using real-time data. However, a fundamental trade-off exists in JITAI design:

• Data Density vs. User Burden: Effective machine learning (ML) models typically require high-frequency data (Ecological Momentary Assessment, or EMA) to capture rapid behavioral dynamics. However, frequent prompts (e.g., hourly) lead to participant fatigue, reduced compliance, and disengagement.
• Algorithmic Complexity vs. Feasibility: There is a lack of empirical guidance on the optimal balance between sampling frequency, the number of predictor variables, and the degree of personalization (group-level vs. hybrid-level algorithms) required to predict high-risk moments accurately without overburdening users.

This study seeks to optimize these parameters to determine if ML algorithms can reliably predict smoking lapses and high cravings with a feasible data collection burden.

2. Methodology

Study Design & Data Source

• Data: Secondary analysis of EMA data from 37 participants attempting to quit smoking in London.
• Protocol: Participants provided up to 16 EMAs per day (one every waking hour) for 10 days post-quit date.
• Variables:
  • Outcomes: (1) Lapses (smoking event between prompts); (2) High Cravings (dichotomized as $\ge$ 7 on an 11-point scale).
  • Predictors: 34 variables total, including 12 between-person (baseline) variables (e.g., demographics, motivation) and 22 within-person (momentary) variables (e.g., mood, context, pain, caffeine/alcohol use, self-efficacy).
• Preprocessing: Missing data were imputed using a univariate Kalman filter (suitable for time-series and future JITAI implementation). Participants with <60% EMA adherence were excluded.

Machine Learning Approach

• Algorithm: Random Forest (RF) classifiers were selected based on prior comparative studies showing superior balance of ROC-AUC, sensitivity, and specificity.
• Training Strategy: Two levels were tested:
  • Group-level: Trained on all other participants' data.
  • Hybrid-level (Warm-start): Trained on group data plus a proportion (10%, 20%, or 30%) of the test participant's own early data.
• Experimental Manipulations: The study systematically varied four factors to test their impact on performance:
  1. Prompt Frequency: 16, 6, 5, 4, and 3 prompts per day (quasi-randomly selected).
  2. Feature Selection: Full set of predictors vs. an optimal subset selected via Recursive Feature Elimination with Cross-Validation (RFE-CV).
  3. Training Data Composition: Proportion of the test participant's own data included in training (0% to 30%).
  4. Outcome Definition: Lapses vs. High Cravings.

Evaluation Metrics

• Primary: F1-score (harmonic mean of precision and recall) and ROC-AUC (Area Under the Receiver Operating Characteristic Curve).
• Secondary: Sensitivity and Specificity.
• Statistical Analysis: Generalized Linear Mixed Models (GLMMs) were used to account for clustering within individuals, analyzing the effects of the experimental manipulations on performance metrics.

3. Key Results

Overall Performance

• Performance was modest on average with high inter-individual variability.
• Lapse Prediction: Median F1-score = 0.436; Median ROC-AUC = 0.659.
• Craving Prediction: Median F1-score = 0.400; Median ROC-AUC = 0.628.
• Many configurations fell below the commonly accepted F1-score threshold of 0.5.

Impact of Prompt Frequency

• Lapses: Counter-intuitively, reducing prompt frequency improved F1-scores. The median F1-score increased from 0.254 (16 prompts) to 0.588 (3 prompts). However, ROC-AUC showed a slight, non-monotonic decline with fewer prompts.
• Cravings: Reducing prompt frequency degraded performance on both metrics. Both F1-score and ROC-AUC declined significantly as prompts decreased from 16 to 3 per day.

Impact of Feature Selection

• Lapses: Feature selection had a negligible impact. Performance with selected features was statistically similar to using the full set of predictors.
• Cravings: Feature selection resulted in a slight decrease in performance (both F1 and ROC-AUC), suggesting that a broader set of variables is necessary for predicting cravings.

Impact of Personalization (Hybrid Algorithms)

• Lapses: Including participant-specific data (20–30%) significantly improved F1-scores but did not significantly improve ROC-AUC.
• Cravings: Including participant-specific data impaired performance, particularly ROC-AUC, compared to group-level models.

Outcome Differences

• Lapse prediction consistently outperformed craving prediction, particularly regarding ROC-AUC. The authors suggest cravings are inherently harder to predict due to their fluctuating nature and the longer, more variable gaps between EMAs.

4. Key Contributions

1. Empirical Optimization of JITAI Parameters: The study provides the first systematic evidence on how EMA frequency, feature count, and training data composition affect ML performance in smoking cessation.
2. Counter-Intuitive Finding on Sampling: It challenges the assumption that "more data is always better." For lapse prediction, lower sampling frequencies (3–6 prompts/day) yielded better F1-scores, likely because higher frequencies introduce noise or because the "event" (lapse) is discrete and easier to capture with less granular, but strategically spaced, data.
3. Differentiation of Outcomes: It highlights that "lapses" and "cravings" require different modeling strategies. Lapses are robust to feature reduction and benefit from personalization, whereas cravings are sensitive to data sparsity and do not benefit from warm-start personalization in this context.
4. Pragmatic Implementation Guidance: The study suggests that a parsimonious, group-level model using 3–6 prompts per day and feature selection is a viable, low-burden starting point for real-world JITAIs.

5. Significance and Implications

• Feasibility of JITAIs: The results suggest that machine learning can detect high-risk moments for smoking lapses without requiring the burdensome 16-prompt-per-day protocol, making JITAI implementation more scalable and acceptable to users.
• Hybrid Decision Strategies: Given the modest average performance and high individual variability, the authors argue that ML should not be the sole decision-maker. Instead, ML predictions should be combined with rules-based approaches or clinician-informed heuristics to ensure robustness.
• Threshold Tuning: The study emphasizes that fixed classification thresholds (e.g., 0.5) may not be optimal. Real-world implementation should involve tuning thresholds to prioritize sensitivity (catching all risks) over specificity, accepting a higher rate of false positives to prevent relapse.
• Future Research: The authors call for experimental studies to validate these findings in diverse populations and to explore longer-term prediction windows, as predictors may change over the course of a quit attempt.

Conclusion: While EMA-based ML shows promise for predicting smoking lapses, it currently offers modest accuracy. The optimal strategy involves a trade-off: using fewer prompts (3–6/day) and a reduced feature set for lapse prediction, potentially combined with hybrid decision rules, rather than relying on high-frequency, fully personalized models.