A tool to evaluate the impact of lived experience involvement in research: the Brain and Genomics Hub: Impact Log literature review and protocol.

This paper presents a protocol and literature review for the "Impact Log," a novel, lived-experience-led tool and framework designed to systematically record, evaluate, and enhance the impact of co-production across all stages of the Brain and Genomics Hub's multidisciplinary research on severe mental illness.

The Gist

Imagine you are trying to bake the perfect cake for a very specific group of people who have a unique allergy. For years, professional chefs (the researchers) have been baking this cake in their kitchens, tasting it, and adjusting the recipe based on their own expert knowledge. But they keep missing the mark because they don't fully understand the allergy or the specific texture the eaters need.

In the past, chefs might have asked a few people with the allergy to taste the cake at the very end and say, "Hmm, needs more sugar." But this paper argues that's not enough. We need the people with the allergy to be in the kitchen, wearing aprons, helping to mix the batter, choose the ingredients, and decide on the oven temperature from the very first step. This is called Co-production.

However, there's a problem: How do we know if having the allergy-sufferers in the kitchen actually made the cake better? Did their input change the recipe? Did it make the process happier? Or did they just stand there while the chefs did everything?

This paper introduces a new tool called the Impact Log to solve that mystery.

The Problem: The "Box-Ticking" Trap

Right now, many research projects try to include people with lived experience (people who have actually lived through the mental health conditions being studied, like bipolar disorder or schizophrenia). But often, it feels like "box-ticking." It's like inviting someone to a party just so you can say, "Look, we had a guest!" without actually letting them dance or talk to anyone.

The authors of this paper want to move past that. They want to know: Did the guest actually change the music? Did they help plan the menu?

The Solution: The "Impact Log"

Think of the Impact Log as a recipe journal or a flight recorder for the research project.

Every time the research team (the chefs) and the lived experience team (the guests) meet to make a decision, they fill out a simple form. This form asks:

1. What did we discuss? (e.g., "How do we ask people to join the study?")
2. What did the guest suggest? (e.g., "This question is too scary; let's make it friendlier.")
3. Did we listen? (e.g., "Yes, we changed the question.")
4. How did it change the cake? (e.g., "Because we changed the question, 20% more people signed up.")

This log isn't just for the chefs; the guests fill it out too. It's a two-way street to ensure everyone feels heard and valued.

The Three Areas of Impact

The paper explains that this "recipe journal" tracks changes in three specific areas, which the authors call "Domains":

1. The Recipe (Design & Delivery): Did the guest help change the actual steps of the research? Maybe they said, "Don't ask for blood samples on a Tuesday; people are too tired then."
2. The Kitchen Atmosphere (Interpersonal & Environmental): Did the presence of the guests make the chefs more kind? Did it reduce the fear or stigma the chefs had about mental illness? Did the kitchen feel more like a team effort and less like a hierarchy?
3. The Systems (Processes): Did the rules of the kitchen change? Maybe the chefs realized they needed to pay the guests for their time or give them a seat at the head table, not just a chair in the corner.

Who is doing this?

The most exciting part of this paper is who is leading it. The "Brain and Genomics Hub" (the big research project) is being run by a team that includes many people who have bipolar disorder or psychosis. They aren't just the "guests"; they are the Head Chefs.

They have created a special group called the LEAP (Lived Experience Advisory Panel). These are experts who know the condition from the inside and know how to do research. They are building this "Impact Log" tool to prove that when you mix "Expertise by Experience" with "Expertise by Training," you get a much better cake (research) for everyone.

Why does this matter?

Imagine if every time a new medical study was done, we had a clear, simple way to prove that the people living with the condition helped shape it.

• For Researchers: It stops them from guessing what patients need.
• For Patients: It ensures the research actually helps them, rather than just being done to them.
• For the Future: This "Impact Log" is designed to be a blueprint. Just like you can copy a great recipe, other research teams can copy this tool to make sure their projects are also inclusive and effective.

In a Nutshell

This paper is a manual for building a better kitchen. It says: "Let's stop pretending we know what the diners want. Let's invite them in, give them a tool to record their ideas, and track how their ideas make our work better, kinder, and more effective."

It's about moving from "Nothing about us without us" (a slogan) to "Nothing about us without us, and here is the proof that we made a difference."

Technical Summary

1. Problem Statement

Despite the widespread recognition of the value of Lived Experience (LE) involvement (also known as Patient and Public Involvement or Co-production) in healthcare research, significant gaps remain in how this involvement is implemented, recorded, and evaluated.

• Lack of Standardization: There are no widely used, standardized tools to systematically record and evaluate the impact of co-production across different research stages.
• Tokenism and Structural Barriers: LE involvement is often superficial ("box-ticking"), limited to early research stages, or restricted to psychosocial research rather than complex biomedical studies.
• Evaluation Deficits: Current methods for assessing impact are often inconsistent, lack clarity on who should provide reflections (LE vs. non-LE), and fail to capture the full breadth of impact, particularly regarding interpersonal, cultural, and systemic changes.
• Diversity Gaps: There is a lack of representation from individuals with severe mental illnesses (e.g., bipolar disorder, psychosis) and diverse demographic backgrounds in research leadership.

2. Methodology

The paper presents a two-part protocol: a narrative literature review (Part I) and the development/implementation protocol for a new evaluation tool (Part II).

Part I: Narrative Literature Review

• Objective: To synthesize existing literature on defining, evaluating, and enhancing the impact of co-production.
• Search Strategy: Conducted by a specialist Lived Experience Advisory Panel (LEAP). It included a non-systematic search of peer-reviewed journals (PubMed) and grey literature, supplemented by expert review and iterative screening.
• Data Extraction: 91 peer-reviewed articles and 5 grey literature items were analyzed using an iterative deductive methodology to identify themes regarding impact definition, evaluation methods, and strategies for increasing impact.

Part II: The Impact Log Protocol

• Setting: The Brain and Genomics Hub (BG Hub), a 5-year multidisciplinary, transdiagnostic research project (UKRI Mental Health Platform) focusing on bipolar disorder, schizophrenia, and schizoaffective disorder.
• Tool Development: The Impact Log is a novel, LE-led tool evolved from a pilot used in the "Mental Health and Justice" project.
  • Structure: Three distinct forms were co-designed for different user groups:
    1. Specialist LEAP Form: For LE researchers with dual expertise (experience + research skills) who are core team members.
    2. Lived Experience Advisory Work Form: For LE advisors participating in specific, non-core activities.
    3. Researcher Form: For non-LE academic staff to record how LE input influenced their specific work.
  • Data Collection: Uses a mixed-methods approach (quantitative Likert scales and qualitative open-text fields) via secure electronic forms (REDCap) or paper.
  • Anonymity: Forms are anonymous to encourage honest feedback, though unique identifiers can be used for longitudinal analysis of specific team members.
• Implementation Framework:
  • Dual Expertise: The project integrates "dual expertise" researchers (LE + research/clinical skills) into leadership to bridge gaps between academic and community perspectives.
  • Remuneration: All substantive LE advisory work is remunerated according to NIHR guidelines, with specific targets to involve at least 20% of study participants in paid advisory roles.
  • Analysis Plan: Data will be analyzed iteratively throughout the project and comprehensively in the final year. Quantitative data will be analyzed descriptively; qualitative data via thematic analysis. The analysis itself will be LE-led.

3. Key Contributions

• The Impact Log Tool: A replicable, adaptable framework for recording and evaluating co-production impact across three specific domains:
  1. Design and Delivery: Impact on research questions, methodology, and data collection.
  2. Systems and Processes: Impact on institutional structures, policies, and resource allocation.
  3. Interpersonal and Environmental: Impact on attitudes, stigma reduction, power dynamics, and team culture.
• Operationalizing Co-production: The paper moves beyond theory to provide a concrete, tested mechanism for integrating LE into complex biomedical research (genomics, neuroimaging), challenging the notion that LE is only suitable for qualitative studies.
• Dual Expertise Model: It formalizes the role of researchers with "dual expertise" (LE + professional skills) as essential leaders in co-production, rather than just advisors.
• Transparency and Accountability: By mandating anonymous, structured feedback, the tool aims to move co-production away from tokenism toward genuine accountability and continuous improvement.

4. Results (of the Protocol and Review)

• Literature Review Findings:
  • Impact is categorized into process/outcome changes (quality, relevance, systemic policy) and interpersonal/cultural changes (stigma reduction, attitude shifts).
  • Current evaluation relies heavily on qualitative reflections, often lacking clarity on the source of the reflection.
  • There is a consensus that structural changes (funding, leadership roles, payment) are necessary to move beyond tokenism.
• Pilot and Development Findings:
  • Initial piloting of the Impact Log forms showed high usability (completable in ~5 minutes) and positive engagement from LE participants.
  • A survey of the Specialist LEAP regarding the tool's design revealed four key themes: the value of collaboration, the necessity of integration within the research team, the tool's potential as a methodological resource, and its ability to increase transparency.
  • LEAP members emphasized that the tool helps operationalize subjective co-production processes and demonstrates value to funding bodies.

5. Significance and Future Directions

• Methodological Advancement: The Impact Log provides a standardized yet flexible framework that can be adapted for diverse mental health and general healthcare research contexts, addressing a critical gap in the literature.
• Systemic Change: By embedding LE leadership and remuneration into the core of a major biomedical hub, the project serves as a model for how to achieve genuine structural integration rather than superficial consultation.
• Scalability: The tool is designed for wide dissemination. The authors plan to host data on the DATAMIND platform and share forms publicly to encourage adoption by other researchers and funding bodies.
• Limitations and Mitigation: The authors acknowledge potential limitations, including resource intensity, the risk of bias in self-reporting (even with anonymity), and the health challenges of LE researchers. Contingency plans and a focus on psychological safety are built into the protocol to mitigate these risks.

Conclusion: This paper outlines a pioneering protocol for evaluating the impact of lived experience in research. By combining a rigorous literature review with a practical, LE-led evaluation tool (the Impact Log), it offers a pathway to transform co-production from a "box-ticking" exercise into a measurable, impactful, and integral component of high-quality mental health research.