Measuring the impact of lived experience and caregiver engagement in research on the research conducted: development and pilot testing of an assessment tool

This study describes the development and pilot testing of the METRE, a new assessment tool designed to measure the impact of lived experience and caregiver engagement on research, which was found to be straightforward and effective while requiring further psychometric validation.

The Gist

Imagine you are baking a giant, complex cake for a community celebration. In the old days, professional bakers (the academic researchers) would design the recipe, buy the ingredients, bake the cake, and serve it to the community. Sometimes, they would ask the community, "Do you like this?" after the cake was already baked.

But recently, there's been a big shift. Now, the community members (the people with lived experience and caregivers) are invited into the kitchen while the cake is being made. They help pick the flavor, taste the batter, and decide how to decorate it. This is called engagement.

The problem? Everyone agrees this is a good idea, but nobody had a way to measure how much the cake actually changed because the community was in the kitchen. Did the community member just say "nice job" while the baker did everything? Or did they actually convince the baker to swap the chocolate for vanilla, change the frosting, and save the cake from being too sweet?

This paper is about building a new "Ruler" to measure that change.

The Problem: The Invisible Impact

The authors noticed that while everyone talks about "engaging" people in research, there was no good tool to answer the question: "How much did the community partner actually change the research project?"

Existing tools were like measuring how friendly the kitchen was, or how many people were wearing aprons. They didn't measure if the recipe itself was different because of the new voices in the room.

The Solution: The METRE Ruler

The team created a new tool called METRE (Measure of Engagement Tool for Research and lived Experience). Think of METRE as a 10-point checklist that acts like a "Change Detector."

Instead of just asking "Did you talk to them?", METRE asks:

1. Did the community help pick the flavor (Research Priority)?
2. Did they help write the recipe (Research Questions)?
3. Did they help get the money to buy ingredients (Grant Applications)?
4. Did they help design the oven (Study Planning)?
5. ...and so on, all the way to how the cake was sliced and served to the world (Reporting).

For each step, the tool asks: "Did the community partner have No Impact, a Small Impact, a Medium Impact, or a Huge Impact?"

How They Built It (The Recipe for the Ruler)

The authors didn't just guess what the ruler should look like. They built it in four stages, like a master chef refining a new utensil:

1. The Taste Test (Qualitative Study): They talked to 40 people (bakers and community members) to ask, "Where in the process do you actually make a difference?" They listened to stories about how a partner changed a study's direction or helped recruit people who were previously hard to reach.
2. Drafting the Ruler (Development): They turned those stories into 10 specific questions on a scale. They made sure the language was clear, like writing instructions that a 10-year-old could understand.
3. The Pilot Test (Trying it out): They gave the draft ruler to 23 people (13 community members and 10 researchers) and asked them to use it on a project they had worked on. They watched them use it (like a "think-aloud" session) to see if the ruler was confusing.
   • The Verdict: The ruler was easy to use and felt accurate! People said, "Yes, this captures exactly how much we changed the project."
   • The Tweaks: Some people said, "This question is a bit fuzzy," or "The instructions are too long." The team sharpened the ruler based on this feedback.
4. The Final Polish: They refined the tool one last time. Now, METRE is ready to be used by research teams to see if their "kitchen" is truly collaborative.

Why This Matters

Think of research as a map.

• Without engagement: The map is drawn by people who have never walked the terrain. It might look pretty, but it might lead hikers off a cliff.
• With engagement: The people who actually walked the terrain help draw the map.
• With METRE: We can finally measure how much better the map is because the walkers helped draw it.

If a research team uses METRE and finds they have "Low Impact" in the "Data Analysis" section, they know, "Hey, we need to invite our partners into the data room next time!" It helps teams move from just saying they are inclusive to actually being inclusive in a way that changes the outcome.

The Bottom Line

This paper introduces METRE, a new, co-created tool that helps researchers and community partners measure the real, tangible difference that community voices make in science. It turns the abstract idea of "partnership" into concrete data, ensuring that when people with lived experience step into the research kitchen, they aren't just tasting the batter—they are helping to bake the cake.

Note: The tool is currently in its "pilot" phase (like a prototype). The authors are now planning to test it on a much larger scale to make sure it works perfectly for everyone, everywhere.

Technical Summary

1. Problem Statement

Despite the increasing prevalence of "patient-oriented research" (involving people with lived/living experience and caregivers), there is a critical gap in measurement science regarding the impact of this engagement on the research itself.

• The Gap: Existing tools (e.g., PEIRS, PPEET) measure the quality of the engagement process or the experience of the partners, but none assess the tangible impact of engagement contributions on the research lifecycle (e.g., how much the research questions, methodology, or results changed due to partner input).
• The Need: Researchers and funding bodies lack a validated instrument to quantify the extent to which lived experience partners influence specific stages of a research project, from priority setting to knowledge translation.

2. Methodology

The study employed a sequential mixed-methods design consisting of four distinct stages to develop and pilot the Measure of Engagement Tool for Research and lived Experience (METRE).

• Stage 1: Qualitative Descriptive Study (Item Generation)

  • Participants: 40 individuals (28 lived/living experience/caregiver partners and 12 academic researchers) with experience in mental health and substance use research.
  • Procedure: Five virtual focus groups and 12 individual interviews were conducted using a co-designed semi-structured guide.
  • Analysis: Framework analysis was used to code transcripts. Initial codes were refined into 10 domains of impact (e.g., priority setting, grant applications, data analysis).
  • Engagement: A Lived/Living Experience and Caregiver Working Group was involved throughout, co-designing the interview guide and codebook.

• Stage 2: Tool Development

  • Based on Stage 1 findings, a draft 10-item scale was created.
  • Each item utilized a 4-point Likert scale: No impact, Small positive impact, Medium positive impact, Large positive impact.
  • Descriptive anchors were developed to define each level of impact.
  • The tool was iteratively refined with the Working Group and academic researchers.

• Stage 3: Pilot Testing (Cognitive Interviewing & Quantitative Analysis)

  • Participants: 23 new participants (13 lived/living experience/caregiver partners and 10 academic researchers) across mental health, substance use, and physical health research.
  • Procedure: Participants completed the METRE regarding a specific past project while engaging in a "think-aloud" cognitive interview to assess face validity and usability. They also answered an 8-question qualitative interview and a single quantitative scaling question (1–10 scale) for correlation analysis.
  • Analysis:
    • Qualitative: Conventional content analysis of interview transcripts.
    • Quantitative: Descriptive statistics (mean, SD, skewness, kurtosis) and Spearman correlation between the METRE total score and the global impact rating.

• Stage 4: Post-Pilot Refinement

  • Qualitative and quantitative findings were synthesized by the Working Group and scientists to finalize the tool.
  • Items were reworded for clarity, and instructions were simplified.

3. Key Contributions

• Development of METRE: The creation of the first assessment tool specifically designed to measure the perceived impact of lived experience and caregiver engagement on the research process itself, rather than just the engagement experience.
• Co-Production: The tool was rigorously co-developed with lived experience partners and caregivers at every stage, ensuring content validity and relevance.
• Comprehensive Scope: The tool covers the entire research lifecycle, including 10 specific domains:
  1. Research topic and priority setting
  2. Developing research questions
  3. Finding funding (Grant applications)
  4. Project planning
  5. Participant recruitment and data collection
  6. Data analysis and results generation
  7. Results interpretation
  8. Sharing results and reporting
  9. Research climate
  10. Research relevance

4. Results

• Qualitative Findings (Pilot):
  • Usability: Participants found the METRE straightforward, easy to understand, and well-supported by instructional videos.
  • Thoroughness: The tool was praised for covering the full research spectrum and prompting reflection on engagement practices.
  • Areas for Improvement: Participants noted difficulty distinguishing between "medium" and "large" impact levels for some items and suggested clearer wording and layout adjustments.
• Quantitative Findings (Pilot):
  • Item Performance: All 10 items showed acceptable variability (responses ranged from 1 to 4).
  • Distribution: Skewness and kurtosis values were not statistically significant at the conservative $\alpha < .01$ level, indicating acceptable distribution for a pilot sample.
  • Correlation: The METRE total score showed a moderate correlation ($r_s = .42, p = .053$) with the global 1–10 impact rating, suggesting the tool aligns with participants' overall perception of impact.
• Domain Insights:
  • Engagement was reported across all stages, but data analysis was identified as the stage with the least engagement, confirming a known gap in the field.
  • Academic researchers tended to report "high impact" more frequently than lived experience partners, suggesting a potential perceptual bias to be explored in future validation.

5. Significance and Future Directions

• Advancing Measurement Science: The METRE fills a critical void by providing a standardized metric to evaluate the outcome of engagement, moving beyond process evaluation.
• Practical Utility: Research teams can use the tool to identify strengths and gaps in their engagement strategies, while funders can use it to assess the depth of engagement in grant applications.
• Ethical Imperative: The authors emphasize that while measuring impact is valuable, engagement remains an ethical imperative and a matter of epistemic justice, regardless of the measured "impact" score.
• Next Steps: The study concludes that the METRE requires full psychometric validation (testing construct validity, reliability, and factor structure) in a larger, more diverse international sample before widespread clinical or research application. The authors are currently conducting this validation phase.

Limitations Noted: The pilot sample was small ($N=23$), predominantly female, and focused heavily on mental health/substance use. The "think-aloud" process may have influenced scoring, and selection bias may have favored participants with positive engagement experiences.