biggest danger of relying on case study evidence is that it

Warning: The NCBI web site requires JavaScript to function. more...

An official website of the United States government

The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• Browse Titles

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Kyratsis Y, Ahmad R, Hatzaras K, et al. Making sense of evidence in management decisions: the role of research-based knowledge on innovation adoption and implementation in health care. Southampton (UK): NIHR Journals Library; 2014 Mar. (Health Services and Delivery Research, No. 2.6.)

Making sense of evidence in management decisions: the role of research-based knowledge on innovation adoption and implementation in health care.

Chapter 4 challenges in making sense of evidence.

In this chapter we summarise some key themes from the qualitative study, which drew on primary data collected through interviews in phase 1. These themes were deemed helpful in providing a conceptual understanding of the main ‘challenges to making sense of evidence’ reported by the informants. The emergent issues discussed in this chapter help place individual sensemaking of evidence by health-care managers in the context of the hospital and wider NHS environment.

• Ongoing sensemaking: keeping up with the evolving evidence

The very nature of evidence as emergent, iterative and changing featured in the majority of interviews with respondents, particularly as the context of this research was innovation. The accuracy of evidence therefore had a temporal dimension, irrespective of the source of the evidence, or the audience making sense of the evidence. The need for ‘up-to-date’ evidence, which sometimes needed to be generated locally, was an important theme in respondents’ accounts:

Well-written trials that have been peer reviewed and written up in trusted journals. But I am also now old enough to see that some of the things that we took as facts, 10 years ago, have already been proved incorrect. T5M4 – doctor

This evolving landscape of evidence posed a challenge, therefore, for individuals and groups when making decisions about adoption and methods of implementation. Further challenges to making sense of evidence reported by the respondents ranged from the individual’s internal capacity to process the scientific data presented to external factors, such as the lack of evidence in the specific areas of innovation and IPC. The lack of ‘high-quality’ evidence was reported by the majority of respondents, although this definition of quality varied across the professional groups and is discussed later when we look at the importance attributed to various sources and types of evidence (see Chapter 5 ):

As a doctor I go to the medical literature, didn’t find a lot. So my nurses came back with a lot of nursing literature evidence. Which I felt was of poorer quality evidence, but there was a large volume of it, so it was put into the mix somewhere. T3M3 – doctor

In terms of the individual’s internal capacity, respondents across the professional groups cited difficulties in understanding the evidence presented in published papers and reports. Specifically, 75% of medical hybrid managers and 77% of nursing hybrid managers said that they sometimes found ‘the content of presented evidence difficult to understand’. Similarly, the majority in each of these groups found it ‘difficult to relate evidence to practice’: 63% of doctors and 72% of nurses. The non-clinical managers reported a different experience – 60% stated that they sometimes found the content of presented evidence difficult to understand, but only 40% had difficulty in relating this to practice.

There was consistency among the groups in agreeing that different professional groups have access to different sources of evidence because of different needs for evidence. This access and need for different types of evidence was deemed to have direct implications for practice:

If everybody isn’t looking at the same piece of information it can affect how you make the decision because we can all be coming at it from different points of view. I can say generally speaking within this organisation when we are looking to do anything we get the relevant people round the table. It’s not that IPC would make a decision that would impact on the provider without involving, they would involve our actual service provider and we will be involved as well. And I do think we do that well really. T2M2 – non-clinical manager

• Missing research evidence

Looking in greater detail at the gap in evidence or ‘missing evidence’ as identified by the respondents, there is more consensus than variation on this issue across professional groups and across the trusts. Challenges arising from the lack of relevant evidence as well as incomplete evidence were mentioned by the majority of respondents. For example, a laboratory-based/microbiology study may be available for a given product but no studies relating to cost. Implementation studies may be available, but may only report a ward-based small study, which may not be relevant to the hospital-wide context. A lack of product trials was described, the products being either untested in the ‘real-world’ setting or untested in the locally relevant setting. Particularly among doctors, basing decisions within the context of incomplete evidence was reported as not just a challenge but undesirable:

There’s damn all evidence most of the time. So we’re very used to doing what seems sensible from first principles, which may not actually be, so often we do things without a formal level of evidence basically. T9M2 – doctor

It’s difficult because some of the things we do I must admit they are based on very little evidence. T1M17 – doctor

It [personalised care plans for renal patients] was a good idea but it wasn’t trialled anywhere, there was no sort of pilot study to demonstrate how much time it was going to take to fill these things out, whether they would actually be useful, did the patients think they were useful, did the doctors think it was useful. T5M5 – doctor

These findings have an important implication as to what managers currently perceive as ‘incomplete evidence’ in research when making decisions on innovation adoption, and what future research should focus on to meet such local needs.

Types or topics of research studies perceived as missing by the respondents did vary across the respondents and across the trusts, but views converged for three types of research study, which were identified as missing in the following order: behavioural studies, implementation research and organisational studies or management research.

Approximately one-third of respondents in trusts T1, T3, T6 and T9 felt a need for behavioural studies to assist decision-making, implementation and evaluation. Specifically, interest in behavioural studies was driven by the need to overcome ‘non-compliant’ behaviour; insights into bringing about change in the way people work; learning from training and development mechanisms; and better communication. More importantly, the respondents identified a need for better understanding of decision-making across the different levels of hospital staff, from senior management to front-line staff:

I think that there is quite a lot of management research that is missing. Partly because managers don’t tend to do a great deal of research in this organisation. Then again all the behavioural work that is done is linked to nursing or medical, I think this is the first research that I have seen that is linked to managers as well. I would like to see a lot more research based around behaviours and how managers and clinical staff could work much better together, to deliver a health service, because I see models out there where they work so well and yet somehow the NHS cannot get it right across the whole trust. T1M1 – non-clinical manager

Yeah not enough is done around the whole decision architecture and influencing behaviours in the clinical areas. How do we improve behaviours? Not just in the clinical areas, managerial areas, but how do we improve the way we work which is not around new technologies but how can we stop wasting huge amounts of time repeating the same things over and over again. T1M10 – doctor

The ‘missing evidence’ highlighted by respondents is interlinked and demonstrated a need by the hospital managers not only for applied, meaningful evidence use in adoption decision-making itself, but also more operational and managerial research. This ranged from effective management to psycho-social research about behaviour change and receptive organisational culture. The following respondent highlighted a shift in evidence needs – highlighting what is most useful to managers:

My research brain has gone exponential in the last year or two. I think there needs to be far more focus on the behavioural and cultural aspects of innovation spread as well as just the subject matter. Because the understanding ‘how’ to challenge the behaviours and ‘how’ to develop the people who are involved in the organisations is far more important than the actual evidence that drives it. Increasingly I’m convinced more and more. T3M4 – doctor

[. . .] where we’ve got the catheter project, the CAUTI project we’re doing, where we’ve got John who’s our clinical academic, we’re looking at doing some sort of research around people’s decision making as to why they’re putting catheters in. [. . .] Why do people make those decisions to do that or why do people make decisions to move away from the guideline that’s there [. . .] There’s a lot, from an infection prevention point of view there’s a lot of scientific type stuff we could do but that is quite difficult already because we don’t want to inject people with, but [. . .] I find that behaviour really interesting as to why people do make the decisions they do. T9M1 – nurse

I work with a public health doctor and he was really interested in implementing change, change methodology. And I think as much importance of thinking about that as thinking about the evidence. If the evidence stacks up, or evidence doesn’t stack up particularly well. You could have good evidence and poor implementation and no effect. Poor evidence not even particularly good but with really good implementation will make it improve but almost, I think there is something there. What I would say [is] that even if I’d like to assimilate stuff, actually it’s not what other people want to do, you don’t have the time to do it. Lots of people who are over-committed and busy and sometime go, I’m sure there is something better unless you tell me what you want to do. T9M8 – doctor

This type of management literature was not accessible to the respondents largely because of the sources used by these professionals, and also because of the time constraints faced by these professionals, who were not able to branch out to wider literature streams.

Of the professionals groups, pharmacists appeared to be more aware of the discrepancy between recommended practice (through national or local guidelines and protocols) and ‘non-adherence’ or ‘deviation’ of behaviour than the other professionals in our study sample. Approximately two-thirds of respondents, despite the small sample size, commented on the importance of behavioural studies and the lack of such studies. One-quarter of nurses and non-clinical managers identified the importance of studies to address ‘non-adherence’ to guidelines, whereas this view was less prevalent in accounts from medical managers and missing in accounts from managers with an allied health professional background. Medical managers were the ‘outliers’ in terms of being less concerned about understanding behavioural change in greater depth.

• Making sense of evidence for self and others

Sourcing practice-based evidence was mentioned as being important across professional groups. The practice of learning from other trusts and peers featured across respondents’ accounts. This was because of the locally relevant information in practice-based evidence but also because of the exchange of information which is possible through such means:

A microbiologist in another hospital or someone who has used something in practice and any research or studies they have done, that is usually the most useful. I guess because you are talking to them you can ask questions and get feedback straight away, so you know where you are with it. So that’s a really good source. T7M3 – doctor

Upon direct questioning, respondents reported a hierarchy of evidence, but this was articulated more as processual rather than as an objective vertical hierarchy, or means to exclude certain forms of evidence. Although the first port of call may be scientific randomised controlled trials (when available), this was assessed in tandem with experiential evidence:

We used that, literature searches for that [Gentamicin (antibiotic) as first line for the treatment of urinary tract infections]. But we also used experience of other hospitals, our own experiences, we drew on that. So actually it was probably a decision which was much more of a pragmatic decision rather than a pure academic-based decision. T3M17 – pharmacist

The approach described by the majority of respondents was an iterative process of ‘triangulating’ different types of evidence. There were few reports of an evidence dichotomy within professional groups, but rather a more complex picture of synthesis across the professional groups. Paradoxically, many respondents did view other professional groups as having a more dichotomous approach to evidence, as illustrated by this view of non-clinical managers:

[. . .] if they’re an accountant it will be purely based on cost effectiveness without looking at the wider picture of your added value this technique may bring. T1M19 – doctor

This view was reciprocated by non-clinical managers:

Partly because people spend more time critiquing the research paper than looking at how we can implement it, or not implement it or how we can try it ourselves. That’s how we get stuck sometimes, people spending too much time focusing on their research, was it true was it evidence-based, did it have flaws? T1M1 – non-clinical manager

A contested ground emerged, with each professional group claiming a more rounded view of evidence and perceiving other groups as taking a one-dimensional approach.

The quest for evidence of doctors was driven primarily by plausibility and accuracy to self. The evidence sought was largely of a biomedical nature. Doctors appreciated that the cost-effectiveness of interventions was important but, as shown above, described non-clinical managers’ approach as too focused on the business case.

Although both the nursing group and the non-clinical managers group reported a relatively balanced multidimensional view to evidence, the motivation for sourcing a diverse evidence base was different for these two groups. Non-clinical managers took a multidimensional view to satisfy the major objective in their organisational role, that is, to improve performance and outcomes. Nurses were driven primarily by the need to ‘make the case’ for others and appreciated that different professionals had different evidence needs:

Most of things I do are evidence-based. I would be looking for things such as standard of construction, standard of validation with processing that sort of thing. I can’t honestly say that I can think of an instance that I did something where I didn’t actually have the evidence. T2M12 – non-clinical manager

Non-clinical managers were similar to doctors in that the way they made sense of evidence was driven primarily by ‘plausibility and accuracy to self’, although their sensemaking was based on different views of evidence. That is, what came to count as evidence for doctors and non-clinical managers was different, but what counted most was that they themselves were satisfied with the evidence. The nursing group differed markedly in this respect from the doctors and non-clinical managers. The nursing hybrid managers focused on the pursuit of evidence for ‘plausibility and accuracy for self and others’. For the nurses, what counted as evidence to others mattered equally and sometimes more than their own satisfaction with evidence. This shaped the types and sources of evidence used by nurses.

In the nursing group, we found there was a high awareness of different types of evidence being relevant to different organisational members. They appreciated the evidence needs of those working both at the front-line and at more strategic levels, and the needs across professional groups. Nurses were also the only group to make explicit reference to the perceptions of patients. Plausibility to others thus featured highly in accounts by nurses. Nurses made purposeful attempts to frame evidence using language which was meaningful and tailored to the audience. Nurses also were aware of their own professional role and identity and how they were perceived by others – that is, being reflective on their own ‘credibility’ as sources of evidence. This non-clinical manager articulated this varying credibility of the presenter of evidence:

Although it galls me to say it but I think the medical colleagues within the team are better at accessing [evidence] and they may come to a meeting and say I have had a look at the evidence. I don’t think it could necessarily have been a systematic review of the evidence. Stating quite confidently a particular position and that could be quite influential so that is something they are more likely to do than nursing members of the team. T7M13 – non-clinical manager

Nurses therefore approached sourcing evidence in a systematic and comprehensive way in order to find evidence that was meaningful and accurate for themselves as well as for significant others. There was a convergence towards synthesising diverse forms of evidence, but, ultimately, evidence synthesis was grounded in the biomedical paradigm. This was partly a result of their own training but also reflected a need to resonate with doctors, who were consistently identified as influential stakeholders in organisational decision-making:

You will see it in very specialist nurses that they will do scoping exercises around what the evidence is, systematic review around evidence of implementing a certain thing and clinical evidence to support it. I think the reason why nurses do that is because they know that the doctors, that are going to try and influence [the decision], will ask them for that evidence, so they already do it. T1M2 – nurse

I think it is the availability of good quality evidence and research something that will convince the senior members and the medical staff that this is a good quality piece of research, peer reviewed etc. T6M5 – senior nurse

I can remember quite clearly presenting to our anaesthetist body, some 200 odd anaesthetists on one of the clinical audit days, on a topic, [. . .] around line care, and the changes we had made in the organisation. And there was one consultant, a specific consultant who’d been a problem all the way through, he’d not engaged well. We gave the presentation, we demonstrated what we’d achieved in the organisation since we’d introduced our changes in practice, and he actually turned round in front of the other 200, and he said ‘I change my opinion’, he says, ‘I accept what you’ve been championing’. And to be honest that was one of the most powerful moments in my career, to get that individual to, in front of 200 of his colleagues, to turn round and say ‘I’ve seen the light.’ [. . .] And sort of do, do the St Paul’s Damascus moment, it was just, it was tremendous, [. . .] It was strongly presented with good, we used, took an epidemiological approach to demonstrate that the changes we had made had had a significant impact. T8M1 – nurse

Nurses were aware of the use of evidence for different agendas, but overall perceived that evidence was used primarily for the benefit of patients in the context of financial constraints. This, in turn, led to the need for combining different types of evidence (i.e. clinical effectiveness, cost, usability) to satisfy the perceptions and priorities of key organisational stakeholders – from doctors to managers.

Doctors and non-clinical managers were both mindful of issues of cost-effectiveness, particularly given that our sample in phase 1 comprises senior managers.

The findings from the qualitative interviews are validated by quantitative analysis. The quantitative analysis shows that nurses were aware of, and utilise more widely, the full range of centrally available evidence sources when compared with the other professional groups (see Chapter 5 ). In addition, nurses were more formally engaged across the phases of the innovation process, whereas doctors were more formally involved in the later phases of technology adoption decisions and post-implementation evaluation. The nursing group was, across the trusts, more formally tasked with ‘making the case’ to diverse groups.

Across respondent groups, plausibility to self was closely linked with perceived ‘accuracy’ of the evidence. This was influenced by social and personal identities situated within a wider organisational context. For example, financial considerations were evident in the sensemaking of the majority of respondents. The influences of the local and macro context of financial parsimony added to the challenges of making sense of evidence:

Financial viability [. . .] that has rapidly changed, we have to justify everything that is new in terms of spending. T1M2 – nurse

• Reflection on this chapter

In summary, all respondents reported that they experienced challenges in making sense of evidence. Key issues that contributed to this were reported as a lack of capacity or skills to process presented evidence, a lack of time to thoroughly search for and review the evidence base, unawareness of appropriate literature on management and implementation research and poor perceived quality of available evidence. Professional background and training coupled with differential access to different evidence reinforced some of the divergence in the type of evidence accessed. Pursuit of evidence to satisfy oneself or others was found to guide action and explained some of the complexity in the process of decision-making. Looking across the professional groups, what counted as evidence for doctors and non-clinical managers was different, but what counted most was that they themselves (doctors and non-clinical managers) were satisfied with the evidence. For the nurses, what counted as evidence to others mattered equally and sometimes more than their own satisfaction with evidence. This shaped the types and sources of evidence used by nurses.

As regards perceived missing evidence, three research study types were identified by respondents: behavioural studies, implementation research, and organisational and management research. Pharmacists were particularly mindful of the need to understand behavioural change within organisations, particularly in relation to non-compliance with guidelines.

Included under terms of UK Non-commercial Government License .

• Cite this Page Kyratsis Y, Ahmad R, Hatzaras K, et al. Making sense of evidence in management decisions: the role of research-based knowledge on innovation adoption and implementation in health care. Southampton (UK): NIHR Journals Library; 2014 Mar. (Health Services and Delivery Research, No. 2.6.) Chapter 4, Challenges in making sense of evidence.
• PDF version of this title (12M)

In this Page

Other titles in this collection.

• Health Services and Delivery Research

Recent Activity

• Challenges in making sense of evidence - Making sense of evidence in management ... Challenges in making sense of evidence - Making sense of evidence in management decisions: the role of research-based knowledge on innovation adoption and implementation in health care

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers

Snapsolve any problem by taking a picture. Try it in the Numerade app?