Implementation and student outcomes of an implementation strategy for the sustainability of group evidence-based practices in schools

Technical Summary: Implementation and Student Outcomes of a Sustainment Strategy for Group Evidence-Based Practices in Schools

Problem Statement
Despite the clinical efficacy and acceptance of school-based mental health evidence-based practices (EBPs), schools frequently struggle to sustain these interventions over time. A primary barrier is the limited professional training available to implementers, such as school counselors. While initial training followed by ongoing supervision is known to maintain skills, sustaining full-intensity supervision is often unfeasible in under-resourced urban school settings. Consequently, there is a need for efficient training models that preserve implementation fidelity and student outcomes while minimizing the burden on school systems. This study addresses the gap in research regarding the effectiveness and cost-effectiveness of specific training strategies designed to sustain mental health EBPs in urban elementary schools.

Methodology
The study employed a hybrid Type 2 cluster-randomized trial design to compare two implementation strategies: Prepare for Sustainment (PFS) and Sustainment as Usual (SAU).

• Participants: The study involved 40 master's-level implementers (school counselors and SEL teachers), two district coaches, and 435 K–8 students in a large Northeastern urban school district. Students were identified as requiring Tier 2 support for externalizing or internalizing problems.
• Interventions: Two group EBPs were implemented: the Coping Power Program (CPP) for externalizing behaviors and CBT for Anxiety Treatment in Schools (CATS) for anxiety.
• Study Phases:
  • Phase 1 (Initial Implementation): All implementers received identical, full-intensity support (one year of direct training and consultation) from external research consultants. Schools were randomized 1:1 to PFS or SAU at the conclusion of this phase.
  • Phase 2 (Reduced Support): PFS implementers received consultation from district coaches (who were themselves supported by external consultants) at a rate of one consultation per two EBP sessions. SAU implementers received standard district supervision without external consultant support.
  • Phase 3 (No Support): Both conditions received supervision as usual from district coaches, with no external consultant support.
• Measures: Outcomes included content and process fidelity (using checklists), service penetration (percentage of eligible students served), student mental health symptoms (parent and student reports via BASC-3, BFS-YR, SCARED), and academic engagement (teacher reports via EvsD). Qualitative data were gathered through interviews with implementers and administrators.
• Analysis: Analyses followed an intention-to-treat (ITT) framework. Implementation outcomes were analyzed using change scores and t-tests, while student outcomes utilized mixed models for repeated measures (MMRM) to estimate effect sizes (Cohen's d) for condition × time interactions.

Key Contributions
This study contributes to the fields of school mental health and implementation science by:

1. Demonstrating that a sustainment strategy involving initial training followed by reduced-intensity consultation for district coaches (PFS) can maintain high process fidelity compared to standard supervision (SAU).
2. Providing evidence that students in the PFS condition showed greater engagement with academic learning and greater reductions in mental health symptoms compared to SAU students, particularly during the reduced support phase.
3. Highlighting the specific needs of implementers, revealing that those without prior support from trained coaches desired supervision on effective EBP delivery, and those experiencing reduced support requested more frequent guidance.
4. Suggesting that providing approximately half the original training intensity to district coaches is sufficient to maintain implementers' clinical skills during sustainment.

Results

• Implementation Fidelity: PFS implementers demonstrated superior process fidelity compared to SAU. In Phase 2, PFS implementers showed significantly higher scores in "Active Engagement" with students. In Phase 3, despite the absence of external support, PFS implementers delivered treatment with significantly higher "Organized Teaching" scores (warmer, clearer, and more organized delivery). Content fidelity and service penetration rates did not differ significantly between conditions in either phase.
• Student Outcomes:
  • Academic Engagement: In Phase 2, PFS students showed medium effect-size improvements in behavioral and emotional engagement with learning compared to SAU. In Phase 3, PFS students showed a large effect-size improvement in emotional engagement and a medium effect-size reduction in behavioral disaffection.
  • Mental Health Symptoms: Students receiving CPP in the PFS condition during Phase 2 demonstrated a large effect-size reduction in self-reported externalizing behaviors compared to SAU. Students receiving CATS in Phase 2 showed a medium effect-size reduction in internalizing symptoms. These effects were smaller or non-significant in Phase 3.
• Qualitative Findings: Implementers valued the external consultation received in Phase 1. Those in Phase 2 expressed a desire for more frequent supervision and specific feedback on group dynamics. Administrators noted that continued collaboration with district coaches would have enhanced their ability to support implementers.

Significance and Claims
The paper concludes that a sustainment strategy consisting of an initial comprehensive training for implementers followed by occasional consultation for school district coaches is effective. Specifically, this approach achieves high process fidelity and yields medium to large effects on student engagement with learning activities and mental health outcomes compared to a "Sustainment as Usual" strategy.

The authors modestly claim that their findings suggest under-resourced schools can feasibly sustain evidence-based practices when implementers receive strong initial training followed by supervision-as-usual, provided that the district coaches themselves receive a reduced but targeted level of external support. The study indicates that while reduced consultation did not increase the volume of services delivered (penetration) or the specific content covered, it significantly enhanced the quality of delivery (process fidelity) and subsequent student engagement and symptom reduction.