Delphi panellists

Key factors that influence effective PI

At round 1, panellists were asked to consider a number of factors (as outlined in the online supplementary appendix 1) that likely impact either as a barrier or a driver to effective PI. The 21 factors were identified from data collected at our previously conducted workshops or from the extant PI literature; and related to the nature (12 items) and the interpersonal aspects (9 items) of the research process. On a seven-point scale from ‘major barrier’ through to ‘major driver’ panellists were asked to rate each item as either a barrier or a driver.

At round 1, there was critical consensus across all panellists for three, and clear consensus for one, major or moderate barriers to effective PI.

• Attitudes of researchers to relinquishing power and control (71% agreement);

• Scientific language used in research (70% agreement);

• Lack of support for PI from research funders (70% agreement);

• The perception that members of the public have biased views (63% agreement).

There was also clear consensus at round 1 around five major or moderate drivers to effective PI:

• The recognition that members of the public have a valuable contribution to make (69% agreement);

• Clear communication between research team members (67% agreement);

• Designated funding for PI (66% agreement);

• Time to build partnerships and trust (65% agreement);

• Training for researchers about PI (63% agreement).

At round 2, the 12 possible barriers or drivers for which there was no consensus at round 1 were presented back to panellists, who were asked to rank in order of importance which they regarded as the three greatest barriers and, similarly, the three greatest drivers. Three factors emerged as the most important barriers, the first two in the list being cited consistently and endorsed across all stakeholder groups:

• The attitudes of academic researchers/clinicians to involving the public in research;

• Perceived importance of PI;

• Lack of research experience of members of the public.

The three factors emerging as the most important drivers are identified below. Once again, the first two drivers in the list were cited consistently and endorsed across all stakeholder groups:

• Ability to be open and flexible to difference;

• Attitude of researchers;

• Perceived importance of PI in health and social care research.

Overall, at round 2 panellists recognised that the same factor when managed well could operate as a driver of PI while when managed poorly operated as a barrier. As one non-clinical academic explained: There are no major barriers if you want to do it… it is a lack of commitment and or interest in doing the necessary learning to do it well. When people do it badly it then reinforces their belief it is not of value. [NCA, round 2]

Open question responses highlighted that tensions across different stakeholder groups within health and social care research were seen as an inevitable consequence of collaborative working. Time to develop team cohesion as well as PI training for members of the public and researchers were seen as pivotal factors in effecting meaningful PI: There needs to be a recognition that all sides have valuable contributions to make to research and that peoples’ attitudes and beliefs, both researchers and the public, are valid and worthy of respect. Training is important and draws the public into the team [NCA, round 2]

Panellists at both rounds repeatedly acknowledged that stakeholder motivation and the positive attitude of all involved were essential prerequisites for good PI. As one clinical academic explained: I was involved in a collaborative group that met consistently since 2007. It has been a journey of experience. Over time that understanding has evolved and grown about good public involvement. This experiential learning took theoretical ideas and made them a reality. It gave the opportunity to challenge the internal subtle prejudice that most clinicians have to public involvement to create a real working relationship that can produce research. [CA, round 1]

Issues related to the potential for PI tokenism

Some panellists were of the opinion that tokenism in PI was value driven: The issue is a cultural one. In my experience, there are very, very few researchers, scientists, doctors who really value public input and involvement. It is done because it ticks the boxes for funding, but the attitude is of resigned tolerance rather than a view that the public add value. [MP, round 1]

On a more positive note it was argued by one research manager that: Changing cultures takes time and three years into my role, I am starting to see results. [RM, round 2]

It was felt that PI needed to be embedded into the culture of organisations; not least by challenging those whose PI endeavour was suggestive of tokenistic practice. Perspectives on potential barriers and drivers to PI were further explored at round 2 when panellists were asked to suggest what, in their opinion, needed to change in order to make PI more than just ‘tokenistic’. A number of key themes emerged from the data. These included:

• The need to provide clear guidance on the purposes of PI, together with models of good practice and measurable standards;

• The provision of and access to appropriate PI education and support for members of the public and clinical and non-clinical academic researchers;

• The need for hosting institutions, research ethics committees, journals and funders to be more proactive in facilitating and embedding PI within infrastructure systems and in promoting the reporting of PI;

• The need to redress the power imbalances in the research process which are felt to favour clinical and non-clinical academic researchers;

• The need for adequate resources, including the provision of funding early on (ie, preprotocol) to enable PI to be embedded early on in the research process.

Our data indicate that mediators to effective PI appeared to fit into two main categories: microlevel mediators including, for example, development of people skills, development and subsequent management of team dynamics; and macrolevel mediators including the quality of organisational infrastructures to support PI. Panellists suggested that training for members of the public should involve more than just an overview of research methods; it also needed to include education about political and policy context(s), as well as address any aspects of personal development training which people identified.

Our panellists also commented that effective PI is embedded in partnership and process values—doing good PI involves the development of relationships. This finding supports the position of INVOLVE43 who promote active ‘partnerships’ with members of the public in the research process, emphasising the need for engagement, support and training. Interestingly, many panellists expressed the view that the process of involvement, when carried out well, is often difficult to deconstruct in order to evaluate discrete elements of the PI contribution and/or impact.

Issues related to impacts of PI

At round 1, panellists were asked to consider 13 impact statements (see online supplementary appendix 1). There was consensus for 10 of the 13 statements, with critical consensus among panellists for three and clear consensus for seven of the statements (figure 1).

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

What are the impacts of public involvement (PI) in health and social care research?

However, many panellists also commented that assessing how PI influences a research project is methodologically challenging, as articulated by the following two panellists: At one level, it is about involving people in a positive way, ensuring their experience of research is constructive and meaningful. Effective implementation is also about the involvement meeting the goals or purpose intended, so that would need to be assessed against these, which are usually project-specific. Often, this will be looking at how the research is different as a result of public involvement, but sometimes that is difficult to discern and may not be very dramatic (if the research has been designed well in the first place). Also, public involvement may not result in changes to the research, but achieves greater acceptance of the research in the relevant communities and that may be difficult to assess. [RM, round 1] Each research project is different and has different objectives for public involvement so it is hard to evaluate scientifically what the effects are. [DR, round 2]

Non-clinical academics were the group that most strongly endorsed the position that assessing how PI influenced research was methodologically challenging. Seventy-one per cent strongly agreed/agreed, compared with 56% of members of the other stakeholder groups. A somewhat surprising finding was that despite high endorsement of the potential positive impacts of PI in research, there was no consensus that it necessarily improves the quality and relevance of research. Members of the public were most likely to think (55%) that PI leads to research of greater quality and relevance; while academic researchers were least likely to think this (32%). Likewise, there was no consensus across the stakeholder groups for the statement that PI makes it more likely that findings from research will be used. However, as one clinical academic pointed out: Absence of evidence isn’t evidence of absence and just ‘cos we can’t yet demonstrate the impact of PI on research quality and relevance it doesn’t mean we never will. As the body of evidence grows the likelihood of showing how and whether PI impacts on research quality and relevance grows and views on this may change [CA, round 2]

Given the level of agreement about methodological difficulties in assessing PI, we asked panellists at round 2 to consider how important they felt it was to do so. Overall, panellists expressed the view that assessment of PI was either very (58%) or fairly (31%) important, only a minority believing PI assessment to be unimportant. Across stakeholder groups, the proportion endorsing PI assessment as ‘very important’ ranged from 40% to 75%.

A number of panellists observed that to evaluate PI in isolation was ‘discriminatory’; rather, it was argued, all aspects of the research process required evaluation. A number of justifications for undertaking PI evaluation were cited and included the suggestion that evaluation provides a mechanism for examining policy and practice in relation to PI, and can be an advocate for change. In the comment below a clinical academic describes how evaluation of PI within her own research team had led to changes in PI practice: We now put more thought and preparation in to what we want the public members to contribute from the outset. If they are involved in developing research questions then it is more likely that their participation will be meaningful at subsequent stages. For each study we now develop a job specification of what is expected, as the basis for discussion and when multiple public members want to participate, to guide selection. It has made the process more formal but it has forced us to think through how and when involvement would be meaningful study by study. [CA, round 2]

At round 1 there was no consensus among panellists about the contribution of PI to improving the quality and relevance of research, or the ways in which research is used. In response to these round 1 findings, panellists were asked, at round 2, to consider whether lack of agreement about the contribution of PI to improving these elements undermined its intrinsic value. Over half the panellists (58%, ranging from 42% to 67% across stakeholder groups) said they did not believe this to be the case, but that a number of issues likely contributed to this lack of agreement—a key challenge being the lack of a common understanding as to the what, when and how of PI. Panellists articulated that questions about the value of PI were answerable only by good evidence. However, lack of sophistication in identifying the unique contribution of PI to the research process, together with lack of clarity around its implementation and practice made meaningful evaluation problematic.

The fact that only 33% and 35% of clinical and non-clinical academic researchers, respectively, said PI added value to research was felt by some panellists to be ‘damaging to the PI cause’ and was perceived as ‘a lever for providing academics with the excuse not to participate in future PI’ Conversely, others argued that the no value perception put forward by the academic community should not be interpreted as PI not having value but rather as a reflection of the way in which academics themselves practiced PI—that is tokenistically: If it is not seen to have value it is less likely to be embedded and will thus remain tokenistic without reaching its full potential value. [NCA, round 2]