Implementation Outcomes of a Web-Based Platform for Reporting and Monitoring Continuous Quality Improvement (CQI) Activities: A Mixed-Methods Evaluation

A formative mixed-methods evaluation of a web-based Continuous Quality Improvement platform in Jamaica reveals that while the tool is generally accepted and effective for documentation and oversight, its sustained adoption depends on addressing critical barriers such as inconsistent internet connectivity, limited training, and workflow integration within supportive organizational contexts.

The Gist

Imagine a hospital in Jamaica as a busy, high-stakes kitchen. For years, the chefs (doctors and nurses) have been trying to make their meals (healthcare services) better using a method called Continuous Quality Improvement (CQI). It's like a team constantly tasting the soup, realizing it needs more salt, adding it, tasting again, and writing down what worked.

The Problem: The Sticky Note Chaos
For a long time, these chefs wrote their recipes and notes on scraps of paper, sticky notes, or basic computer spreadsheets.

• The Issue: If a note fell off the fridge, the recipe was lost. If the head chef wanted to see how the soup tasted across all the kitchens in the city, they had to collect hundreds of messy papers. It was hard to learn from each other, and important lessons often got forgotten when staff changed shifts.

The Solution: The "Smart Kitchen" App
To fix this, the Ministry of Health and their partners built a web-based app (a digital platform). Think of this app as a giant, magical digital whiteboard that connects every kitchen in the country.

• What it does: Instead of paper, chefs can log their experiments, see graphs of how the soup is improving over time, and share their best tricks with other kitchens instantly. It has two views: one for the chefs on the line (to log their work) and one for the head managers (to see the big picture).

The Experiment: Testing the New Tool
The researchers didn't just roll it out and hope for the best. They ran a "test drive" across 24 different hospitals (kitchens) to see how it actually worked in real life. They asked the staff three main questions:

1. Do you like it? (Acceptability)
2. Are you actually using it? (Adoption)
3. Does it fit your job? (Feasibility & Appropriateness)

What They Found: The Good, The Bad, and The "Maybe"

• The Good News (The Magic):
  When the internet was working and the staff knew how to use it, people loved it.

  • Analogy: It was like finally getting a GPS instead of a paper map. They could see their progress clearly. One chef said, "I can stop using all that paper!" Another said, "It helps me see our progress and stay motivated." It saved time and made it easy to spot trends (like realizing the soup tastes better on Tuesdays).

• The Bad News (The Glitches):
  Not everything was smooth sailing.

  • The Internet Woes: In some places, the internet was as unreliable as a Wi-Fi signal in a basement. Sometimes, the app would freeze or refuse to save, causing frustration.
  • The "Extra Work" Feeling: Some staff felt the app was just another chore added to their already overflowing to-do list. It was like being asked to cook the meal and write a blog post about it, with no extra time given.
  • Confusion: Nobody knew exactly who was supposed to type the notes. Was it the doctor? The nurse? The data clerk? Without clear roles, the digital whiteboard sometimes stayed blank.

• The "Maybe" (The Neutral Zone):
  About 19% of the staff said they had never used the app at all. They were like new hires who hadn't been shown the new kitchen equipment yet.

The Verdict: It's a Great Engine, But We Need Better Roads
The study concluded that the app is a powerful tool with huge potential. It's like a Ferrari engine installed in a car. The engine (the app) is amazing and can go fast, but if the roads (internet) are full of potholes, or if the driver (the staff) hasn't had enough training on how to steer, the car won't move very far.

What Needs to Happen Next?
To make this work for everyone, the researchers suggest:

1. Fix the Roads: Improve the internet connection so the app doesn't crash.
2. More Driving Lessons: Don't just train people once. Keep teaching them, especially when new staff join.
3. Clear Job Descriptions: Tell everyone exactly who is responsible for updating the app so it doesn't get ignored.
4. Boss Support: The managers need to champion the tool, making it clear that using the app is part of the job, not an extra burden.

In a Nutshell
Jamaica tried to upgrade its healthcare quality system from a "paper notebook" to a "smart digital dashboard." The dashboard is brilliant and helps everyone see the big picture, but to make it truly successful, they need to fix the internet, train the staff better, and make sure everyone knows their role in driving the car forward.

Technical Summary

1. Problem Statement

Despite the proven efficacy of Continuous Quality Improvement (CQI) in strengthening health systems, its implementation in Low- and Middle-Income Countries (LMICs), specifically Jamaica, is hindered by fragmented documentation and monitoring systems.

• Current State: CQI activities in Jamaica rely heavily on paper-based tools and basic software (e.g., Microsoft Excel/Word).
• Consequences: This reliance leads to incomplete, non-standardized data, hindering timely monitoring, cross-site learning, institutional memory retention, and the scaling of successful interventions.
• The Gap: While Jamaica has institutionalized CQI across priority programs (HIV, NCDs), the lack of a unified digital platform prevents real-time oversight and data-driven decision-making at the system level.

2. Methodology

The study employed a formative, convergent mixed-methods evaluation guided by the Implementation Outcomes Framework (Proctor et al.).

• Setting & Participants:
  • Location: 24 healthcare facilities across Jamaica's four Regional Health Authorities (WRHA, SRHA, NERHA, SERHA).
  • Quantitative: Structured survey of 43 healthcare workers (HCWs) with a 65% response rate.
  • Qualitative: Five Focus Group Discussions (FGDs) with 33 participants and three Key Informant Interviews (KIIs) with CQI leads/coaches.
• Intervention: A web-based CQI App (originally developed by UMB-CIHEB in Tanzania) adapted for the Jamaican context through a participatory, human-centered design process.
  • Features: Two modules—Facility Module (for logging projects, tracking indicators, PDSA cycles) and Manager Module (dashboards, trend visualization, remote oversight).
• Data Collection & Analysis:
  • Timing: Data collected between June and July 2024, following initial training and pilot deployment (Dec 2023).
  • Quantitative: Descriptive statistics (frequencies, percentages) of Likert-scale survey responses.
  • Qualitative: Rapid Qualitative Analysis (RQA) of transcripts to identify themes.
  • Integration: Findings were synthesized using joint displays to identify convergence, divergence, and complementarity between survey data and qualitative insights.
• Evaluation Focus: Four early-stage implementation outcomes: Acceptability, Adoption, Appropriateness, and Feasibility.

3. Key Contributions

• Contextual Adaptation: Demonstrates a successful participatory adaptation of a global digital health tool to a specific national context (Jamaica), aligning terminology and workflows with local health system realities.
• Formative Evaluation Model: Provides a blueprint for early-stage evaluation of digital health tools in resource-constrained settings, focusing on implementation outcomes rather than clinical impact.
• Mixed-Methods Insight: Offers a nuanced understanding of why adoption is uneven by triangulating self-reported survey data with rich qualitative narratives regarding workflow and infrastructure.
• Implementation Framework Application: Validates the utility of the Implementation Outcomes Framework for diagnosing barriers in the early rollout of health information systems.

4. Key Results

A. Acceptability

• Quantitative: Moderate to high perceived acceptability. 70% reported the app saved time; 67% felt it aligned with facility goals. However, interest in frequent use was mixed.
• Qualitative: Users appreciated the user-friendly dashboard, archiving capabilities, and visual trend tracking.
• Barriers: Internet instability, password reset issues, and difficulty editing entries reduced usability.

B. Adoption

• Quantitative: 19% of survey respondents reported never using the app. Among users, 67% used it only 1–2 times.
• Qualitative: Adoption was hindered by unclear team roles (e.g., data clerks vs. clinicians), lack of follow-up training, and the perception of the app as "additional work" rather than an integrated workflow.
• Enablers: Leadership engagement and the ability to share data across sites were cited as motivators.

C. Appropriateness

• Quantitative: High agreement that the app is suitable for tasks and aligns with goals.
• Qualitative: The app was seen as appropriate for documentation and institutional memory. However, many viewed it as an add-on rather than a core routine task.
• Gap: Unclear role definitions led to bottlenecks where only specific staff (data clerks) engaged with the tool.

D. Feasibility

• Quantitative: Mixed views on feasibility. While most felt it was feasible with current IT support, a significant portion was neutral regarding resource availability and maintenance consistency.
• Qualitative: Major barriers included unstable internet connectivity, limited device availability, and file upload errors.
• Support Needs: Participants expressed a need for embedded job aids, mobile versions, and more comprehensive onboarding/training.

5. Significance and Implications

• Strategic Insight: The study concludes that a digitally enabled CQI platform has high potential to strengthen health systems only if embedded within supportive organizational and infrastructural contexts. Technology alone is insufficient.
• Critical Enablers Identified:
  1. Leadership Engagement: Active championing by leaders is required to integrate the tool into strategic plans and workflows.
  2. Workflow Integration: Roles must be clearly defined, and the app must be viewed as a routine part of the job, not an extra burden.
  3. Capacity Building: Training must be continuous (refresher courses, onboarding for new staff) rather than a one-time event.
  4. Infrastructure: Reliable internet and device access are foundational prerequisites for feasibility in LMICs.
• Scalability: The findings offer transferable insights for other LMICs attempting to digitize quality improvement processes, emphasizing that "foundational system readiness" is a prerequisite for successful scale-up.
• Policy Recommendation: To realize the potential of the CQI app in Jamaica and similar settings, stakeholders must address the "last mile" challenges of connectivity, role clarity, and sustained technical support.