importance of case study evaluation

• Open access
• Published: 10 November 2020

Case study research for better evaluations of complex interventions: rationale and challenges

• Sara Paparini   ORCID: orcid.org/0000-0002-1909-2481 1 ,
• Judith Green 2 ,
• Chrysanthi Papoutsi 1 ,
• Jamie Murdoch 3 ,
• Mark Petticrew 4 ,
• Trish Greenhalgh 1 ,
• Benjamin Hanckel 5 &
• Sara Shaw 1  

BMC Medicine volume  18 , Article number:  301 ( 2020 ) Cite this article

17k Accesses

42 Citations

35 Altmetric

Metrics details

The need for better methods for evaluation in health research has been widely recognised. The ‘complexity turn’ has drawn attention to the limitations of relying on causal inference from randomised controlled trials alone for understanding whether, and under which conditions, interventions in complex systems improve health services or the public health, and what mechanisms might link interventions and outcomes. We argue that case study research—currently denigrated as poor evidence—is an under-utilised resource for not only providing evidence about context and transferability, but also for helping strengthen causal inferences when pathways between intervention and effects are likely to be non-linear.

Case study research, as an overall approach, is based on in-depth explorations of complex phenomena in their natural, or real-life, settings. Empirical case studies typically enable dynamic understanding of complex challenges and provide evidence about causal mechanisms and the necessary and sufficient conditions (contexts) for intervention implementation and effects. This is essential evidence not just for researchers concerned about internal and external validity, but also research users in policy and practice who need to know what the likely effects of complex programmes or interventions will be in their settings. The health sciences have much to learn from scholarship on case study methodology in the social sciences. However, there are multiple challenges in fully exploiting the potential learning from case study research. First are misconceptions that case study research can only provide exploratory or descriptive evidence. Second, there is little consensus about what a case study is, and considerable diversity in how empirical case studies are conducted and reported. Finally, as case study researchers typically (and appropriately) focus on thick description (that captures contextual detail), it can be challenging to identify the key messages related to intervention evaluation from case study reports.

Whilst the diversity of published case studies in health services and public health research is rich and productive, we recommend further clarity and specific methodological guidance for those reporting case study research for evaluation audiences.

Peer Review reports

The need for methodological development to address the most urgent challenges in health research has been well-documented. Many of the most pressing questions for public health research, where the focus is on system-level determinants [ 1 , 2 ], and for health services research, where provisions typically vary across sites and are provided through interlocking networks of services [ 3 ], require methodological approaches that can attend to complexity. The need for methodological advance has arisen, in part, as a result of the diminishing returns from randomised controlled trials (RCTs) where they have been used to answer questions about the effects of interventions in complex systems [ 4 , 5 , 6 ]. In conditions of complexity, there is limited value in maintaining the current orientation to experimental trial designs in the health sciences as providing ‘gold standard’ evidence of effect.

There are increasing calls for methodological pluralism [ 7 , 8 ], with the recognition that complex intervention and context are not easily or usefully separated (as is often the situation when using trial design), and that system interruptions may have effects that are not reducible to linear causal pathways between intervention and outcome. These calls are reflected in a shifting and contested discourse of trial design, seen with the emergence of realist [ 9 ], adaptive and hybrid (types 1, 2 and 3) [ 10 , 11 ] trials that blend studies of effectiveness with a close consideration of the contexts of implementation. Similarly, process evaluation has now become a core component of complex healthcare intervention trials, reflected in MRC guidance on how to explore implementation, causal mechanisms and context [ 12 ].

Evidence about the context of an intervention is crucial for questions of external validity. As Woolcock [ 4 ] notes, even if RCT designs are accepted as robust for maximising internal validity, questions of transferability (how well the intervention works in different contexts) and generalisability (how well the intervention can be scaled up) remain unanswered [ 5 , 13 ]. For research evidence to have impact on policy and systems organisation, and thus to improve population and patient health, there is an urgent need for better methods for strengthening external validity, including a better understanding of the relationship between intervention and context [ 14 ].

Policymakers, healthcare commissioners and other research users require credible evidence of relevance to their settings and populations [ 15 ], to perform what Rosengarten and Savransky [ 16 ] call ‘careful abstraction’ to the locales that matter for them. They also require robust evidence for understanding complex causal pathways. Case study research, currently under-utilised in public health and health services evaluation, can offer considerable potential for strengthening faith in both external and internal validity. For example, in an empirical case study of how the policy of free bus travel had specific health effects in London, UK, a quasi-experimental evaluation (led by JG) identified how important aspects of context (a good public transport system) and intervention (that it was universal) were necessary conditions for the observed effects, thus providing useful, actionable evidence for decision-makers in other contexts [ 17 ].

The overall approach of case study research is based on the in-depth exploration of complex phenomena in their natural, or ‘real-life’, settings. Empirical case studies typically enable dynamic understanding of complex challenges rather than restricting the focus on narrow problem delineations and simple fixes. Case study research is a diverse and somewhat contested field, with multiple definitions and perspectives grounded in different ways of viewing the world, and involving different combinations of methods. In this paper, we raise awareness of such plurality and highlight the contribution that case study research can make to the evaluation of complex system-level interventions. We review some of the challenges in exploiting the current evidence base from empirical case studies and conclude by recommending that further guidance and minimum reporting criteria for evaluation using case studies, appropriate for audiences in the health sciences, can enhance the take-up of evidence from case study research.

Case study research offers evidence about context, causal inference in complex systems and implementation

Well-conducted and described empirical case studies provide evidence on context, complexity and mechanisms for understanding how, where and why interventions have their observed effects. Recognition of the importance of context for understanding the relationships between interventions and outcomes is hardly new. In 1943, Canguilhem berated an over-reliance on experimental designs for determining universal physiological laws: ‘As if one could determine a phenomenon’s essence apart from its conditions! As if conditions were a mask or frame which changed neither the face nor the picture!’ ([ 18 ] p126). More recently, a concern with context has been expressed in health systems and public health research as part of what has been called the ‘complexity turn’ [ 1 ]: a recognition that many of the most enduring challenges for developing an evidence base require a consideration of system-level effects [ 1 ] and the conceptualisation of interventions as interruptions in systems [ 19 ].

The case study approach is widely recognised as offering an invaluable resource for understanding the dynamic and evolving influence of context on complex, system-level interventions [ 20 , 21 , 22 , 23 ]. Empirically, case studies can directly inform assessments of where, when, how and for whom interventions might be successfully implemented, by helping to specify the necessary and sufficient conditions under which interventions might have effects and to consolidate learning on how interdependencies, emergence and unpredictability can be managed to achieve and sustain desired effects. Case study research has the potential to address four objectives for improving research and reporting of context recently set out by guidance on taking account of context in population health research [ 24 ], that is to (1) improve the appropriateness of intervention development for specific contexts, (2) improve understanding of ‘how’ interventions work, (3) better understand how and why impacts vary across contexts and (4) ensure reports of intervention studies are most useful for decision-makers and researchers.

However, evaluations of complex healthcare interventions have arguably not exploited the full potential of case study research and can learn much from other disciplines. For evaluative research, exploratory case studies have had a traditional role of providing data on ‘process’, or initial ‘hypothesis-generating’ scoping, but might also have an increasing salience for explanatory aims. Across the social and political sciences, different kinds of case studies are undertaken to meet diverse aims (description, exploration or explanation) and across different scales (from small N qualitative studies that aim to elucidate processes, or provide thick description, to more systematic techniques designed for medium-to-large N cases).

Case studies with explanatory aims vary in terms of their positioning within mixed-methods projects, with designs including (but not restricted to) (1) single N of 1 studies of interventions in specific contexts, where the overall design is a case study that may incorporate one or more (randomised or not) comparisons over time and between variables within the case; (2) a series of cases conducted or synthesised to provide explanation from variations between cases; and (3) case studies of particular settings within RCT or quasi-experimental designs to explore variation in effects or implementation.

Detailed qualitative research (typically done as ‘case studies’ within process evaluations) provides evidence for the plausibility of mechanisms [ 25 ], offering theoretical generalisations for how interventions may function under different conditions. Although RCT designs reduce many threats to internal validity, the mechanisms of effect remain opaque, particularly when the causal pathways between ‘intervention’ and ‘effect’ are long and potentially non-linear: case study research has a more fundamental role here, in providing detailed observational evidence for causal claims [ 26 ] as well as producing a rich, nuanced picture of tensions and multiple perspectives [ 8 ].

Longitudinal or cross-case analysis may be best suited for evidence generation in system-level evaluative research. Turner [ 27 ], for instance, reflecting on the complex processes in major system change, has argued for the need for methods that integrate learning across cases, to develop theoretical knowledge that would enable inferences beyond the single case, and to develop generalisable theory about organisational and structural change in health systems. Qualitative Comparative Analysis (QCA) [ 28 ] is one such formal method for deriving causal claims, using set theory mathematics to integrate data from empirical case studies to answer questions about the configurations of causal pathways linking conditions to outcomes [ 29 , 30 ].

Nonetheless, the single N case study, too, provides opportunities for theoretical development [ 31 ], and theoretical generalisation or analytical refinement [ 32 ]. How ‘the case’ and ‘context’ are conceptualised is crucial here. Findings from the single case may seem to be confined to its intrinsic particularities in a specific and distinct context [ 33 ]. However, if such context is viewed as exemplifying wider social and political forces, the single case can be ‘telling’, rather than ‘typical’, and offer insight into a wider issue [ 34 ]. Internal comparisons within the case can offer rich possibilities for logical inferences about causation [ 17 ]. Further, case studies of any size can be used for theory testing through refutation [ 22 ]. The potential lies, then, in utilising the strengths and plurality of case study to support theory-driven research within different methodological paradigms.

Evaluation research in health has much to learn from a range of social sciences where case study methodology has been used to develop various kinds of causal inference. For instance, Gerring [ 35 ] expands on the within-case variations utilised to make causal claims. For Gerring [ 35 ], case studies come into their own with regard to invariant or strong causal claims (such as X is a necessary and/or sufficient condition for Y) rather than for probabilistic causal claims. For the latter (where experimental methods might have an advantage in estimating effect sizes), case studies offer evidence on mechanisms: from observations of X affecting Y, from process tracing or from pattern matching. Case studies also support the study of emergent causation, that is, the multiple interacting properties that account for particular and unexpected outcomes in complex systems, such as in healthcare [ 8 ].

Finally, efficacy (or beliefs about efficacy) is not the only contributor to intervention uptake, with a range of organisational and policy contingencies affecting whether an intervention is likely to be rolled out in practice. Case study research is, therefore, invaluable for learning about contextual contingencies and identifying the conditions necessary for interventions to become normalised (i.e. implemented routinely) in practice [ 36 ].

The challenges in exploiting evidence from case study research

At present, there are significant challenges in exploiting the benefits of case study research in evaluative health research, which relate to status, definition and reporting. Case study research has been marginalised at the bottom of an evidence hierarchy, seen to offer little by way of explanatory power, if nonetheless useful for adding descriptive data on process or providing useful illustrations for policymakers [ 37 ]. This is an opportune moment to revisit this low status. As health researchers are increasingly charged with evaluating ‘natural experiments’—the use of face masks in the response to the COVID-19 pandemic being a recent example [ 38 ]—rather than interventions that take place in settings that can be controlled, research approaches using methods to strengthen causal inference that does not require randomisation become more relevant.

A second challenge for improving the use of case study evidence in evaluative health research is that, as we have seen, what is meant by ‘case study’ varies widely, not only across but also within disciplines. There is indeed little consensus amongst methodologists as to how to define ‘a case study’. Definitions focus, variously, on small sample size or lack of control over the intervention (e.g. [ 39 ] p194), on in-depth study and context [ 40 , 41 ], on the logic of inference used [ 35 ] or on distinct research strategies which incorporate a number of methods to address questions of ‘how’ and ‘why’ [ 42 ]. Moreover, definitions developed for specific disciplines do not capture the range of ways in which case study research is carried out across disciplines. Multiple definitions of case study reflect the richness and diversity of the approach. However, evidence suggests that a lack of consensus across methodologists results in some of the limitations of published reports of empirical case studies [ 43 , 44 ]. Hyett and colleagues [ 43 ], for instance, reviewing reports in qualitative journals, found little match between methodological definitions of case study research and how authors used the term.

This raises the third challenge we identify that case study reports are typically not written in ways that are accessible or useful for the evaluation research community and policymakers. Case studies may not appear in journals widely read by those in the health sciences, either because space constraints preclude the reporting of rich, thick descriptions, or because of the reported lack of willingness of some biomedical journals to publish research that uses qualitative methods [ 45 ], signalling the persistence of the aforementioned evidence hierarchy. Where they do, however, the term ‘case study’ is used to indicate, interchangeably, a qualitative study, an N of 1 sample, or a multi-method, in-depth analysis of one example from a population of phenomena. Definitions of what constitutes the ‘case’ are frequently lacking and appear to be used as a synonym for the settings in which the research is conducted. Despite offering insights for evaluation, the primary aims may not have been evaluative, so the implications may not be explicitly drawn out. Indeed, some case study reports might properly be aiming for thick description without necessarily seeking to inform about context or causality.

Acknowledging plurality and developing guidance

We recognise that definitional and methodological plurality is not only inevitable, but also a necessary and creative reflection of the very different epistemological and disciplinary origins of health researchers, and the aims they have in doing and reporting case study research. Indeed, to provide some clarity, Thomas [ 46 ] has suggested a typology of subject/purpose/approach/process for classifying aims (e.g. evaluative or exploratory), sample rationale and selection and methods for data generation of case studies. We also recognise that the diversity of methods used in case study research, and the necessary focus on narrative reporting, does not lend itself to straightforward development of formal quality or reporting criteria.

Existing checklists for reporting case study research from the social sciences—for example Lincoln and Guba’s [ 47 ] and Stake’s [ 33 ]—are primarily orientated to the quality of narrative produced, and the extent to which they encapsulate thick description, rather than the more pragmatic issues of implications for intervention effects. Those designed for clinical settings, such as the CARE (CAse REports) guidelines, provide specific reporting guidelines for medical case reports about single, or small groups of patients [ 48 ], not for case study research.

The Design of Case Study Research in Health Care (DESCARTE) model [ 44 ] suggests a series of questions to be asked of a case study researcher (including clarity about the philosophy underpinning their research), study design (with a focus on case definition) and analysis (to improve process). The model resembles toolkits for enhancing the quality and robustness of qualitative and mixed-methods research reporting, and it is usefully open-ended and non-prescriptive. However, even if it does include some reflections on context, the model does not fully address aspects of context, logic and causal inference that are perhaps most relevant for evaluative research in health.

Hence, for evaluative research where the aim is to report empirical findings in ways that are intended to be pragmatically useful for health policy and practice, this may be an opportune time to consider how to best navigate plurality around what is (minimally) important to report when publishing empirical case studies, especially with regards to the complex relationships between context and interventions, information that case study research is well placed to provide.

The conventional scientific quest for certainty, predictability and linear causality (maximised in RCT designs) has to be augmented by the study of uncertainty, unpredictability and emergent causality [ 8 ] in complex systems. This will require methodological pluralism, and openness to broadening the evidence base to better understand both causality in and the transferability of system change intervention [ 14 , 20 , 23 , 25 ]. Case study research evidence is essential, yet is currently under exploited in the health sciences. If evaluative health research is to move beyond the current impasse on methods for understanding interventions as interruptions in complex systems, we need to consider in more detail how researchers can conduct and report empirical case studies which do aim to elucidate the contextual factors which interact with interventions to produce particular effects. To this end, supported by the UK’s Medical Research Council, we are embracing the challenge to develop guidance for case study researchers studying complex interventions. Following a meta-narrative review of the literature, we are planning a Delphi study to inform guidance that will, at minimum, cover the value of case study research for evaluating the interrelationship between context and complex system-level interventions; for situating and defining ‘the case’, and generalising from case studies; as well as provide specific guidance on conducting, analysing and reporting case study research. Our hope is that such guidance can support researchers evaluating interventions in complex systems to better exploit the diversity and richness of case study research.

Availability of data and materials

Not applicable (article based on existing available academic publications)

Abbreviations

Qualitative comparative analysis

Quasi-experimental design

Randomised controlled trial

Diez Roux AV. Complex systems thinking and current impasses in health disparities research. Am J Public Health. 2011;101(9):1627–34.

Article   Google Scholar  

Ogilvie D, Mitchell R, Mutrie N, M P, Platt S. Evaluating health effects of transport interventions: methodologic case study. Am J Prev Med 2006;31:118–126.

Walshe C. The evaluation of complex interventions in palliative care: an exploration of the potential of case study research strategies. Palliat Med. 2011;25(8):774–81.

Woolcock M. Using case studies to explore the external validity of ‘complex’ development interventions. Evaluation. 2013;19:229–48.

Cartwright N. Are RCTs the gold standard? BioSocieties. 2007;2(1):11–20.

Deaton A, Cartwright N. Understanding and misunderstanding randomized controlled trials. Soc Sci Med. 2018;210:2–21.

Salway S, Green J. Towards a critical complex systems approach to public health. Crit Public Health. 2017;27(5):523–4.

Greenhalgh T, Papoutsi C. Studying complexity in health services research: desperately seeking an overdue paradigm shift. BMC Med. 2018;16(1):95.

Bonell C, Warren E, Fletcher A. Realist trials and the testing of context-mechanism-outcome configurations: a response to Van Belle et al. Trials. 2016;17:478.

Pallmann P, Bedding AW, Choodari-Oskooei B. Adaptive designs in clinical trials: why use them, and how to run and report them. BMC Med. 2018;16:29.

Curran G, Bauer M, Mittman B, Pyne J, Stetler C. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care. 2012;50(3):217–26. https://doi.org/10.1097/MLR.0b013e3182408812 .

Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015 [cited 2020 Jun 27];350. Available from: https://www.bmj.com/content/350/bmj.h1258 .

Evans RE, Craig P, Hoddinott P, Littlecott H, Moore L, Murphy S, et al. When and how do ‘effective’ interventions need to be adapted and/or re-evaluated in new contexts? The need for guidance. J Epidemiol Community Health. 2019;73(6):481–2.

Shoveller J. A critical examination of representations of context within research on population health interventions. Crit Public Health. 2016;26(5):487–500.

Treweek S, Zwarenstein M. Making trials matter: pragmatic and explanatory trials and the problem of applicability. Trials. 2009;10(1):37.

Rosengarten M, Savransky M. A careful biomedicine? Generalization and abstraction in RCTs. Crit Public Health. 2019;29(2):181–91.

Green J, Roberts H, Petticrew M, Steinbach R, Goodman A, Jones A, et al. Integrating quasi-experimental and inductive designs in evaluation: a case study of the impact of free bus travel on public health. Evaluation. 2015;21(4):391–406.

Canguilhem G. The normal and the pathological. New York: Zone Books; 1991. (1949).

Google Scholar  

Hawe P, Shiell A, Riley T. Theorising interventions as events in systems. Am J Community Psychol. 2009;43:267–76.

King G, Keohane RO, Verba S. Designing social inquiry: scientific inference in qualitative research: Princeton University Press; 1994.

Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004;82(4):581–629.

Yin R. Enhancing the quality of case studies in health services research. Health Serv Res. 1999;34(5 Pt 2):1209.

CAS   PubMed   PubMed Central   Google Scholar  

Raine R, Fitzpatrick R, Barratt H, Bevan G, Black N, Boaden R, et al. Challenges, solutions and future directions in the evaluation of service innovations in health care and public health. Health Serv Deliv Res. 2016 [cited 2020 Jun 30];4(16). Available from: https://www.journalslibrary.nihr.ac.uk/hsdr/hsdr04160#/abstract .

Craig P, Di Ruggiero E, Frohlich KL, E M, White M, Group CCGA. Taking account of context in population health intervention research: guidance for producers, users and funders of research. NIHR Evaluation, Trials and Studies Coordinating Centre; 2018.

Grant RL, Hood R. Complex systems, explanation and policy: implications of the crisis of replication for public health research. Crit Public Health. 2017;27(5):525–32.

Mahoney J. Strategies of causal inference in small-N analysis. Sociol Methods Res. 2000;4:387–424.

Turner S. Major system change: a management and organisational research perspective. In: Rosalind Raine, Ray Fitzpatrick, Helen Barratt, Gywn Bevan, Nick Black, Ruth Boaden, et al. Challenges, solutions and future directions in the evaluation of service innovations in health care and public health. Health Serv Deliv Res. 2016;4(16) 2016. https://doi.org/10.3310/hsdr04160.

Ragin CC. Using qualitative comparative analysis to study causal complexity. Health Serv Res. 1999;34(5 Pt 2):1225.

Hanckel B, Petticrew M, Thomas J, Green J. Protocol for a systematic review of the use of qualitative comparative analysis for evaluative questions in public health research. Syst Rev. 2019;8(1):252.

Schneider CQ, Wagemann C. Set-theoretic methods for the social sciences: a guide to qualitative comparative analysis: Cambridge University Press; 2012. 369 p.

Flyvbjerg B. Five misunderstandings about case-study research. Qual Inq. 2006;12:219–45.

Tsoukas H. Craving for generality and small-N studies: a Wittgensteinian approach towards the epistemology of the particular in organization and management studies. Sage Handb Organ Res Methods. 2009:285–301.

Stake RE. The art of case study research. London: Sage Publications Ltd; 1995.

Mitchell JC. Typicality and the case study. Ethnographic research: A guide to general conduct. Vol. 238241. 1984.

Gerring J. What is a case study and what is it good for? Am Polit Sci Rev. 2004;98(2):341–54.

May C, Mort M, Williams T, F M, Gask L. Health technology assessment in its local contexts: studies of telehealthcare. Soc Sci Med 2003;57:697–710.

McGill E. Trading quality for relevance: non-health decision-makers’ use of evidence on the social determinants of health. BMJ Open. 2015;5(4):007053.

Greenhalgh T. We can’t be 100% sure face masks work – but that shouldn’t stop us wearing them | Trish Greenhalgh. The Guardian. 2020 [cited 2020 Jun 27]; Available from: https://www.theguardian.com/commentisfree/2020/jun/05/face-masks-coronavirus .

Hammersley M. So, what are case studies? In: What’s wrong with ethnography? New York: Routledge; 1992.

Crowe S, Cresswell K, Robertson A, Huby G, Avery A, Sheikh A. The case study approach. BMC Med Res Methodol. 2011;11(1):100.

Luck L, Jackson D, Usher K. Case study: a bridge across the paradigms. Nurs Inq. 2006;13(2):103–9.

Yin RK. Case study research and applications: design and methods: Sage; 2017.

Hyett N, A K, Dickson-Swift V. Methodology or method? A critical review of qualitative case study reports. Int J Qual Stud Health Well-Being. 2014;9:23606.

Carolan CM, Forbat L, Smith A. Developing the DESCARTE model: the design of case study research in health care. Qual Health Res. 2016;26(5):626–39.

Greenhalgh T, Annandale E, Ashcroft R, Barlow J, Black N, Bleakley A, et al. An open letter to the BMJ editors on qualitative research. Bmj. 2016;352.

Thomas G. A typology for the case study in social science following a review of definition, discourse, and structure. Qual Inq. 2011;17(6):511–21.

Lincoln YS, Guba EG. Judging the quality of case study reports. Int J Qual Stud Educ. 1990;3(1):53–9.

Riley DS, Barber MS, Kienle GS, Aronson JK, Schoen-Angerer T, Tugwell P, et al. CARE guidelines for case reports: explanation and elaboration document. J Clin Epidemiol. 2017;89:218–35.

Download references

Acknowledgements

Not applicable

This work was funded by the Medical Research Council - MRC Award MR/S014632/1 HCS: Case study, Context and Complex interventions (TRIPLE C). SP was additionally funded by the University of Oxford's Higher Education Innovation Fund (HEIF).

Author information

Authors and affiliations.

Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK

Sara Paparini, Chrysanthi Papoutsi, Trish Greenhalgh & Sara Shaw

Wellcome Centre for Cultures & Environments of Health, University of Exeter, Exeter, UK

Judith Green

School of Health Sciences, University of East Anglia, Norwich, UK

Jamie Murdoch

Public Health, Environments and Society, London School of Hygiene & Tropical Medicin, London, UK

Mark Petticrew

Institute for Culture and Society, Western Sydney University, Penrith, Australia

Benjamin Hanckel

You can also search for this author in PubMed   Google Scholar

Contributions

JG, MP, SP, JM, TG, CP and SS drafted the initial paper; all authors contributed to the drafting of the final version, and read and approved the final manuscript.

Corresponding author

Correspondence to Sara Paparini .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Paparini, S., Green, J., Papoutsi, C. et al. Case study research for better evaluations of complex interventions: rationale and challenges. BMC Med 18 , 301 (2020). https://doi.org/10.1186/s12916-020-01777-6

Download citation

Received : 03 July 2020

Accepted : 07 September 2020

Published : 10 November 2020

DOI : https://doi.org/10.1186/s12916-020-01777-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Qualitative
• Case studies
• Mixed-method
• Public health
• Health services research
• Interventions

BMC Medicine

ISSN: 1741-7015

• Submission enquiries: [email protected]
• General enquiries: [email protected]

• SUGGESTED TOPICS
• The Magazine
• Newsletters
• Managing Yourself
• Managing Teams
• Work-life Balance
• The Big Idea
• Data & Visuals
• Reading Lists
• Case Selections
• HBR Learning
• Topic Feeds
• Account Settings
• Email Preferences

What the Case Study Method Really Teaches

• Nitin Nohria

Seven meta-skills that stick even if the cases fade from memory.

It’s been 100 years since Harvard Business School began using the case study method. Beyond teaching specific subject matter, the case study method excels in instilling meta-skills in students. This article explains the importance of seven such skills: preparation, discernment, bias recognition, judgement, collaboration, curiosity, and self-confidence.

During my decade as dean of Harvard Business School, I spent hundreds of hours talking with our alumni. To enliven these conversations, I relied on a favorite question: “What was the most important thing you learned from your time in our MBA program?”

• Nitin Nohria is the George F. Baker Professor of Business Administration, Distinguished University Service Professor, and former dean of Harvard Business School.

Partner Center

• Search Menu
• Browse content in Arts and Humanities
• Browse content in Archaeology
• Anglo-Saxon and Medieval Archaeology
• Archaeological Methodology and Techniques
• Archaeology by Region
• Archaeology of Religion
• Archaeology of Trade and Exchange
• Biblical Archaeology
• Contemporary and Public Archaeology
• Environmental Archaeology
• Historical Archaeology
• History and Theory of Archaeology
• Industrial Archaeology
• Landscape Archaeology
• Mortuary Archaeology
• Prehistoric Archaeology
• Underwater Archaeology
• Urban Archaeology
• Zooarchaeology
• Browse content in Architecture
• Architectural Structure and Design
• History of Architecture
• Residential and Domestic Buildings
• Theory of Architecture
• Browse content in Art
• Art Subjects and Themes
• History of Art
• Industrial and Commercial Art
• Theory of Art
• Biographical Studies
• Byzantine Studies
• Browse content in Classical Studies
• Classical History
• Classical Philosophy
• Classical Mythology
• Classical Literature
• Classical Reception
• Classical Art and Architecture
• Classical Oratory and Rhetoric
• Greek and Roman Epigraphy
• Greek and Roman Law
• Greek and Roman Papyrology
• Greek and Roman Archaeology
• Late Antiquity
• Religion in the Ancient World
• Digital Humanities
• Browse content in History
• Colonialism and Imperialism
• Diplomatic History
• Environmental History
• Genealogy, Heraldry, Names, and Honours
• Genocide and Ethnic Cleansing
• Historical Geography
• History by Period
• History of Emotions
• History of Agriculture
• History of Education
• History of Gender and Sexuality
• Industrial History
• Intellectual History
• International History
• Labour History
• Legal and Constitutional History
• Local and Family History
• Maritime History
• Military History
• National Liberation and Post-Colonialism
• Oral History
• Political History
• Public History
• Regional and National History
• Revolutions and Rebellions
• Slavery and Abolition of Slavery
• Social and Cultural History
• Theory, Methods, and Historiography
• Urban History
• World History
• Browse content in Language Teaching and Learning
• Language Learning (Specific Skills)
• Language Teaching Theory and Methods
• Browse content in Linguistics
• Applied Linguistics
• Cognitive Linguistics
• Computational Linguistics
• Forensic Linguistics
• Grammar, Syntax and Morphology
• Historical and Diachronic Linguistics
• History of English
• Language Acquisition
• Language Evolution
• Language Reference
• Language Variation
• Language Families
• Lexicography
• Linguistic Anthropology
• Linguistic Theories
• Linguistic Typology
• Phonetics and Phonology
• Psycholinguistics
• Sociolinguistics
• Translation and Interpretation
• Writing Systems
• Browse content in Literature
• Bibliography
• Children's Literature Studies
• Literary Studies (Asian)
• Literary Studies (European)
• Literary Studies (Eco-criticism)
• Literary Studies (Romanticism)
• Literary Studies (American)
• Literary Studies (Modernism)
• Literary Studies - World
• Literary Studies (1500 to 1800)
• Literary Studies (19th Century)
• Literary Studies (20th Century onwards)
• Literary Studies (African American Literature)
• Literary Studies (British and Irish)
• Literary Studies (Early and Medieval)
• Literary Studies (Fiction, Novelists, and Prose Writers)
• Literary Studies (Gender Studies)
• Literary Studies (Graphic Novels)
• Literary Studies (History of the Book)
• Literary Studies (Plays and Playwrights)
• Literary Studies (Poetry and Poets)
• Literary Studies (Postcolonial Literature)
• Literary Studies (Queer Studies)
• Literary Studies (Science Fiction)
• Literary Studies (Travel Literature)
• Literary Studies (War Literature)
• Literary Studies (Women's Writing)
• Literary Theory and Cultural Studies
• Mythology and Folklore
• Shakespeare Studies and Criticism
• Browse content in Media Studies
• Browse content in Music
• Applied Music
• Dance and Music
• Ethics in Music
• Ethnomusicology
• Gender and Sexuality in Music
• Medicine and Music
• Music Cultures
• Music and Religion
• Music and Media
• Music and Culture
• Music Education and Pedagogy
• Music Theory and Analysis
• Musical Scores, Lyrics, and Libretti
• Musical Structures, Styles, and Techniques
• Musicology and Music History
• Performance Practice and Studies
• Race and Ethnicity in Music
• Sound Studies
• Browse content in Performing Arts
• Browse content in Philosophy
• Aesthetics and Philosophy of Art
• Epistemology
• Feminist Philosophy
• History of Western Philosophy
• Metaphysics
• Moral Philosophy
• Non-Western Philosophy
• Philosophy of Science
• Philosophy of Language
• Philosophy of Mind
• Philosophy of Perception
• Philosophy of Action
• Philosophy of Law
• Philosophy of Religion
• Philosophy of Mathematics and Logic
• Practical Ethics
• Social and Political Philosophy
• Browse content in Religion
• Biblical Studies
• Christianity
• East Asian Religions
• History of Religion
• Judaism and Jewish Studies
• Qumran Studies
• Religion and Education
• Religion and Health
• Religion and Politics
• Religion and Science
• Religion and Law
• Religion and Art, Literature, and Music
• Religious Studies
• Browse content in Society and Culture
• Cookery, Food, and Drink
• Cultural Studies
• Customs and Traditions
• Ethical Issues and Debates
• Hobbies, Games, Arts and Crafts
• Lifestyle, Home, and Garden
• Natural world, Country Life, and Pets
• Popular Beliefs and Controversial Knowledge
• Sports and Outdoor Recreation
• Technology and Society
• Travel and Holiday
• Visual Culture
• Browse content in Law
• Arbitration
• Browse content in Company and Commercial Law
• Commercial Law
• Company Law
• Browse content in Comparative Law
• Systems of Law
• Competition Law
• Browse content in Constitutional and Administrative Law
• Government Powers
• Judicial Review
• Local Government Law
• Military and Defence Law
• Parliamentary and Legislative Practice
• Construction Law
• Contract Law
• Browse content in Criminal Law
• Criminal Procedure
• Criminal Evidence Law
• Sentencing and Punishment
• Employment and Labour Law
• Environment and Energy Law
• Browse content in Financial Law
• Banking Law
• Insolvency Law
• History of Law
• Human Rights and Immigration
• Intellectual Property Law
• Browse content in International Law
• Private International Law and Conflict of Laws
• Public International Law
• IT and Communications Law
• Jurisprudence and Philosophy of Law
• Law and Politics
• Law and Society
• Browse content in Legal System and Practice
• Courts and Procedure
• Legal Skills and Practice
• Primary Sources of Law
• Regulation of Legal Profession
• Medical and Healthcare Law
• Browse content in Policing
• Criminal Investigation and Detection
• Police and Security Services
• Police Procedure and Law
• Police Regional Planning
• Browse content in Property Law
• Personal Property Law
• Study and Revision
• Terrorism and National Security Law
• Browse content in Trusts Law
• Wills and Probate or Succession
• Browse content in Medicine and Health
• Browse content in Allied Health Professions
• Arts Therapies
• Clinical Science
• Dietetics and Nutrition
• Occupational Therapy
• Operating Department Practice
• Physiotherapy
• Radiography
• Speech and Language Therapy
• Browse content in Anaesthetics
• General Anaesthesia
• Neuroanaesthesia
• Browse content in Clinical Medicine
• Acute Medicine
• Cardiovascular Medicine
• Clinical Genetics
• Clinical Pharmacology and Therapeutics
• Dermatology
• Endocrinology and Diabetes
• Gastroenterology
• Genito-urinary Medicine
• Geriatric Medicine
• Infectious Diseases
• Medical Toxicology
• Medical Oncology
• Pain Medicine
• Palliative Medicine
• Rehabilitation Medicine
• Respiratory Medicine and Pulmonology
• Rheumatology
• Sleep Medicine
• Sports and Exercise Medicine
• Clinical Neuroscience
• Community Medical Services
• Critical Care
• Emergency Medicine
• Forensic Medicine
• Haematology
• History of Medicine
• Browse content in Medical Dentistry
• Oral and Maxillofacial Surgery
• Paediatric Dentistry
• Restorative Dentistry and Orthodontics
• Surgical Dentistry
• Browse content in Medical Skills
• Clinical Skills
• Communication Skills
• Nursing Skills
• Surgical Skills
• Medical Ethics
• Medical Statistics and Methodology
• Browse content in Neurology
• Clinical Neurophysiology
• Neuropathology
• Nursing Studies
• Browse content in Obstetrics and Gynaecology
• Gynaecology
• Occupational Medicine
• Ophthalmology
• Otolaryngology (ENT)
• Browse content in Paediatrics
• Neonatology
• Browse content in Pathology
• Chemical Pathology
• Clinical Cytogenetics and Molecular Genetics
• Histopathology
• Medical Microbiology and Virology
• Patient Education and Information
• Browse content in Pharmacology
• Psychopharmacology
• Browse content in Popular Health
• Caring for Others
• Complementary and Alternative Medicine
• Self-help and Personal Development
• Browse content in Preclinical Medicine
• Cell Biology
• Molecular Biology and Genetics
• Reproduction, Growth and Development
• Primary Care
• Professional Development in Medicine
• Browse content in Psychiatry
• Addiction Medicine
• Child and Adolescent Psychiatry
• Forensic Psychiatry
• Learning Disabilities
• Old Age Psychiatry
• Psychotherapy
• Browse content in Public Health and Epidemiology
• Epidemiology
• Public Health
• Browse content in Radiology
• Clinical Radiology
• Interventional Radiology
• Nuclear Medicine
• Radiation Oncology
• Reproductive Medicine
• Browse content in Surgery
• Cardiothoracic Surgery
• Gastro-intestinal and Colorectal Surgery
• General Surgery
• Neurosurgery
• Paediatric Surgery
• Peri-operative Care
• Plastic and Reconstructive Surgery
• Surgical Oncology
• Transplant Surgery
• Trauma and Orthopaedic Surgery
• Vascular Surgery
• Browse content in Science and Mathematics
• Browse content in Biological Sciences
• Aquatic Biology
• Biochemistry
• Bioinformatics and Computational Biology
• Developmental Biology
• Ecology and Conservation
• Evolutionary Biology
• Genetics and Genomics
• Microbiology
• Molecular and Cell Biology
• Natural History
• Plant Sciences and Forestry
• Research Methods in Life Sciences
• Structural Biology
• Systems Biology
• Zoology and Animal Sciences
• Browse content in Chemistry
• Analytical Chemistry
• Computational Chemistry
• Crystallography
• Environmental Chemistry
• Industrial Chemistry
• Inorganic Chemistry
• Materials Chemistry
• Medicinal Chemistry
• Mineralogy and Gems
• Organic Chemistry
• Physical Chemistry
• Polymer Chemistry
• Study and Communication Skills in Chemistry
• Theoretical Chemistry
• Browse content in Computer Science
• Artificial Intelligence
• Computer Architecture and Logic Design
• Game Studies
• Human-Computer Interaction
• Mathematical Theory of Computation
• Programming Languages
• Software Engineering
• Systems Analysis and Design
• Virtual Reality
• Browse content in Computing
• Business Applications
• Computer Security
• Computer Games
• Computer Networking and Communications
• Digital Lifestyle
• Graphical and Digital Media Applications
• Operating Systems
• Browse content in Earth Sciences and Geography
• Atmospheric Sciences
• Environmental Geography
• Geology and the Lithosphere
• Maps and Map-making
• Meteorology and Climatology
• Oceanography and Hydrology
• Palaeontology
• Physical Geography and Topography
• Regional Geography
• Soil Science
• Urban Geography
• Browse content in Engineering and Technology
• Agriculture and Farming
• Biological Engineering
• Civil Engineering, Surveying, and Building
• Electronics and Communications Engineering
• Energy Technology
• Engineering (General)
• Environmental Science, Engineering, and Technology
• History of Engineering and Technology
• Mechanical Engineering and Materials
• Technology of Industrial Chemistry
• Transport Technology and Trades
• Browse content in Environmental Science
• Applied Ecology (Environmental Science)
• Conservation of the Environment (Environmental Science)
• Environmental Sustainability
• Environmentalist Thought and Ideology (Environmental Science)
• Management of Land and Natural Resources (Environmental Science)
• Natural Disasters (Environmental Science)
• Nuclear Issues (Environmental Science)
• Pollution and Threats to the Environment (Environmental Science)
• Social Impact of Environmental Issues (Environmental Science)
• History of Science and Technology
• Browse content in Materials Science
• Ceramics and Glasses
• Composite Materials
• Metals, Alloying, and Corrosion
• Nanotechnology
• Browse content in Mathematics
• Applied Mathematics
• Biomathematics and Statistics
• History of Mathematics
• Mathematical Education
• Mathematical Finance
• Mathematical Analysis
• Numerical and Computational Mathematics
• Probability and Statistics
• Pure Mathematics
• Browse content in Neuroscience
• Cognition and Behavioural Neuroscience
• Development of the Nervous System
• Disorders of the Nervous System
• History of Neuroscience
• Invertebrate Neurobiology
• Molecular and Cellular Systems
• Neuroendocrinology and Autonomic Nervous System
• Neuroscientific Techniques
• Sensory and Motor Systems
• Browse content in Physics
• Astronomy and Astrophysics
• Atomic, Molecular, and Optical Physics
• Biological and Medical Physics
• Classical Mechanics
• Computational Physics
• Condensed Matter Physics
• Electromagnetism, Optics, and Acoustics
• History of Physics
• Mathematical and Statistical Physics
• Measurement Science
• Nuclear Physics
• Particles and Fields
• Plasma Physics
• Quantum Physics
• Relativity and Gravitation
• Semiconductor and Mesoscopic Physics
• Browse content in Psychology
• Affective Sciences
• Clinical Psychology
• Cognitive Psychology
• Cognitive Neuroscience
• Criminal and Forensic Psychology
• Developmental Psychology
• Educational Psychology
• Evolutionary Psychology
• Health Psychology
• History and Systems in Psychology
• Music Psychology
• Neuropsychology
• Organizational Psychology
• Psychological Assessment and Testing
• Psychology of Human-Technology Interaction
• Psychology Professional Development and Training
• Research Methods in Psychology
• Social Psychology
• Browse content in Social Sciences
• Browse content in Anthropology
• Anthropology of Religion
• Human Evolution
• Medical Anthropology
• Physical Anthropology
• Regional Anthropology
• Social and Cultural Anthropology
• Theory and Practice of Anthropology
• Browse content in Business and Management
• Business Strategy
• Business Ethics
• Business History
• Business and Government
• Business and Technology
• Business and the Environment
• Comparative Management
• Corporate Governance
• Corporate Social Responsibility
• Entrepreneurship
• Health Management
• Human Resource Management
• Industrial and Employment Relations
• Industry Studies
• Information and Communication Technologies
• International Business
• Knowledge Management
• Management and Management Techniques
• Operations Management
• Organizational Theory and Behaviour
• Pensions and Pension Management
• Public and Nonprofit Management
• Strategic Management
• Supply Chain Management
• Browse content in Criminology and Criminal Justice
• Criminal Justice
• Criminology
• Forms of Crime
• International and Comparative Criminology
• Youth Violence and Juvenile Justice
• Development Studies
• Browse content in Economics
• Agricultural, Environmental, and Natural Resource Economics
• Asian Economics
• Behavioural Finance
• Behavioural Economics and Neuroeconomics
• Econometrics and Mathematical Economics
• Economic Systems
• Economic History
• Economic Methodology
• Economic Development and Growth
• Financial Markets
• Financial Institutions and Services
• General Economics and Teaching
• Health, Education, and Welfare
• History of Economic Thought
• International Economics
• Labour and Demographic Economics
• Law and Economics
• Macroeconomics and Monetary Economics
• Microeconomics
• Public Economics
• Urban, Rural, and Regional Economics
• Welfare Economics
• Browse content in Education
• Adult Education and Continuous Learning
• Care and Counselling of Students
• Early Childhood and Elementary Education
• Educational Equipment and Technology
• Educational Strategies and Policy
• Higher and Further Education
• Organization and Management of Education
• Philosophy and Theory of Education
• Schools Studies
• Secondary Education
• Teaching of a Specific Subject
• Teaching of Specific Groups and Special Educational Needs
• Teaching Skills and Techniques
• Browse content in Environment
• Applied Ecology (Social Science)
• Climate Change
• Conservation of the Environment (Social Science)
• Environmentalist Thought and Ideology (Social Science)
• Natural Disasters (Environment)
• Social Impact of Environmental Issues (Social Science)
• Browse content in Human Geography
• Cultural Geography
• Economic Geography
• Political Geography
• Browse content in Interdisciplinary Studies
• Communication Studies
• Museums, Libraries, and Information Sciences
• Browse content in Politics
• African Politics
• Asian Politics
• Chinese Politics
• Comparative Politics
• Conflict Politics
• Elections and Electoral Studies
• Environmental Politics
• European Union
• Foreign Policy
• Gender and Politics
• Human Rights and Politics
• Indian Politics
• International Relations
• International Organization (Politics)
• International Political Economy
• Irish Politics
• Latin American Politics
• Middle Eastern Politics
• Political Methodology
• Political Communication
• Political Philosophy
• Political Sociology
• Political Behaviour
• Political Economy
• Political Institutions
• Political Theory
• Politics and Law
• Public Administration
• Public Policy
• Quantitative Political Methodology
• Regional Political Studies
• Russian Politics
• Security Studies
• State and Local Government
• UK Politics
• US Politics
• Browse content in Regional and Area Studies
• African Studies
• Asian Studies
• East Asian Studies
• Japanese Studies
• Latin American Studies
• Middle Eastern Studies
• Native American Studies
• Scottish Studies
• Browse content in Research and Information
• Research Methods
• Browse content in Social Work
• Addictions and Substance Misuse
• Adoption and Fostering
• Care of the Elderly
• Child and Adolescent Social Work
• Couple and Family Social Work
• Developmental and Physical Disabilities Social Work
• Direct Practice and Clinical Social Work
• Emergency Services
• Human Behaviour and the Social Environment
• International and Global Issues in Social Work
• Mental and Behavioural Health
• Social Justice and Human Rights
• Social Policy and Advocacy
• Social Work and Crime and Justice
• Social Work Macro Practice
• Social Work Practice Settings
• Social Work Research and Evidence-based Practice
• Welfare and Benefit Systems
• Browse content in Sociology
• Childhood Studies
• Community Development
• Comparative and Historical Sociology
• Economic Sociology
• Gender and Sexuality
• Gerontology and Ageing
• Health, Illness, and Medicine
• Marriage and the Family
• Migration Studies
• Occupations, Professions, and Work
• Organizations
• Population and Demography
• Race and Ethnicity
• Social Theory
• Social Movements and Social Change
• Social Research and Statistics
• Social Stratification, Inequality, and Mobility
• Sociology of Religion
• Sociology of Education
• Sport and Leisure
• Urban and Rural Studies
• Browse content in Warfare and Defence
• Defence Strategy, Planning, and Research
• Land Forces and Warfare
• Military Administration
• Military Life and Institutions
• Naval Forces and Warfare
• Other Warfare and Defence Issues
• Peace Studies and Conflict Resolution
• Weapons and Equipment

A newer edition of this book is available.

• < Previous chapter
• Next chapter >

22 Case Study Research: In-Depth Understanding in Context

Helen Simons, School of Education, University of Southampton

• Published: 01 July 2014
• Cite Icon Cite
• Permissions Icon Permissions

This chapter explores case study as a major approach to research and evaluation. After first noting various contexts in which case studies are commonly used, the chapter focuses on case study research directly Strengths and potential problematic issues are outlined and then key phases of the process. The chapter emphasizes how important it is to design the case, to collect and interpret data in ways that highlight the qualitative, to have an ethical practice that values multiple perspectives and political interests, and to report creatively to facilitate use in policy making and practice. Finally, it explores how to generalize from the single case. Concluding questions center on the need to think more imaginatively about design and the range of methods and forms of reporting requiredto persuade audiences to value qualitative ways of knowing in case study research.

Introduction

This chapter explores case study as a major approach to research and evaluation using primarily qualitative methods, as well as documentary sources, contemporaneous or historical. However, this is not the only way in which case study can be conceived. No one has a monopoly on the term. While sharing a focus on the singular in a particular context, case study has a wide variety of uses, not all associated with research. A case study, in common parlance, documents a particular situation or event in detail in a specific sociopolitical context. The particular can be a person, a classroom, an institution, a program, or a policy. Below I identify different ways in which case study is used before focusing on qualitative case study research in particular. However, first I wish to indicate how I came to advocate and practice this form of research. Origins, context, and opportunity often shape the research processes we endorse. It is helpful for the reader, I think, to know how I came to the perspective I hold.

The Beginnings

I first came to appreciate and enjoy the virtues of case study research when I entered the field of curriculum evaluation and research in the 1970s. The dominant research paradigm for educational research at that time was experimental or quasi- experimental, cost-benefit, or systems analysis, and the dominant curriculum model was aims and objectives ( House, 1993 ). The field was dominated, in effect, by a psychometric view of research in which quantitative methods were preeminent. But the innovative projects we were asked to evaluate (predominantly, but not exclusively, in the humanities) were not amenable to such methodologies. The projects were challenging to the status quo of institutions, involved people interpreting the policy and programs, were implemented differently in different contexts and regions, and had many unexpected effects.

We had no choice but to seek other ways to evaluate these complex programs, and case study was the methodology we found ourselves exploring, in order to understand how the projects were being implemented, why they had positive effects in some regions of the country and not others, and what the outcomes meant in different sociopolitical and cultural contexts. What better way to do this than to talk with people to see how they interpreted the “new” curriculum; to watch how teachers and students put it into practice; to document transactions, outcomes, and unexpected consequences; and to interpret all in the specific context of the case ( Simons, 1971 , 1987 , pp. 55–89). From this point on and in further studies, case study in educational research and evaluation came to be a major methodology for understanding complex educational and social programs. It also extended to other practice professions, such as nursing, health, and social care ( Zucker, 2001 ; Greenhalgh & Worrall, 1997 ; Shaw & Gould, 2001 ). For further details of the evolution of the case study approach and qualitative methodologies in evaluation, see House, 1993 , pp. 2–3; Greene, 2000 ; Simons, 2009 , pp. 14–18; Simons & McCormack, 2007 , pp. 292–311).

This was not exactly the beginning of case study, of course. It has a long history in many disciplines ( Simons, 1980; Ragin, 1992; Gomm, Hammersley, & Foster, 2004 ; Platt, 2007 ), many aspects of which form part of case study practice to this day. But its evolution in the context just described was a major move in the contemporary evolution of the logic of evaluative inquiry ( House, 1980 ). It also coincided with movement toward the qualitative in other disciplines, such as sociology and psychology. This was all part of what Denzin & Lincoln (1994) termed “a quiet methodological revolution” (p. ix) in qualitative inquiry that had been evolving over the course of forty years.

There is a further reason why I continue to advocate and practice case study research and evaluation to this day and that is my personal predilection for trying to understand and represent complexity, for puzzling through the ambiguities that exist in many contexts and programs and for presenting and negotiating different values and interests in fair and just ways.

Put more simply, I like interacting with people, listening to their stories, trials and tribulations—giving them a voice in understanding the contexts and projects with which they are involved, and finding ways to share these with a range of audiences. In other words, the move toward case study methodology described here suited my preference for how I learn.

Concepts and Purposes of Case Study

Before exploring case study as it has come to be established in educational research and evaluation over the past forty years, I wish to acknowledge other uses of case study. More often than not, these relate to purpose, and appropriately so in their different contexts, but many do not have a research intention. For a study to count as research, it would need to be a systematic investigation generating evidence that leads to “new” knowledge that is made public and open to scrutiny. There are many ways to conduct research stemming from different traditions and disciplines, but they all, in different ways, involve these characteristics.

Everyday Usage: Stories We Tell

The most common of these uses of case study is the everyday reference to a person, an anecdote or story illustrative of a particular incident, event, or experience of that person. It is often a short, reported account commonly seen in journalism but also in books exploring a phenomenon, such as recovery from serious accidents or tragedies, where the author chooses to illustrate the story or argument with a “lived” example. This is sometimes written by the author and sometimes by the person whose tale it is. “Let me share with you a story,” is a phrase frequently heard

The spirit behind this common usage and its power to connect can be seen in a report by Tim Adams of the London Olympics opening ceremony’s dramatization by Danny Boyle.

It was the point when we suddenly collectively wised up to the idea that what we are about to receive over the next two weeks was not only about “legacy collateral” and “targeted deliverables,” not about G4S failings and traffic lanes and branding opportunities, but about the second-by-second possibilities of human endeavour and spirit and communality, enacted in multiple places and all at the same time. Stories in other words. ( Adams, 2012 )

This was a collective story, of course, not an individual one, but it does convey some of the major characteristics of case study—that richness of detail, time, place, multiple happenings and experiences—that are also manifest in case study research, although carefully evidenced in the latter instance. We can see from this common usage how people have come to associate case study with story. I return to this thread in the reporting section.

Professions Individual Cases

In professional settings, in health and social care, case studies, often called case histories , are used to accurately record a person’s health or social care history and his or her current symptoms, experience, and treatment. These case histories include facts but also judgments and observations about the person’s reaction to situations or medication. Usually these are confidential. Not dissimilar is the detailed documentation of a case in law, often termed a case precedent when referred to in a court case to support an argument being made. However in law there is a difference in that such case precedents are publicly documented.

Case Studies in Teaching

Exemplars of practice.

In education, but also in health and social care training contexts, case studies have long been used as exemplars of practice. These are brief descriptions with some detail of a person or project’s experience in an area of practice. Though frequently reported accounts, they are based on a person’s experience and sometimes on previous research.

Case scenarios

Management studies are a further context in which case studies are often used. Here, the case is more like a scenario outlining a particular problem situation for the management student to resolve. These scenarios may be based on research but frequently are hypothetical situations used to raise issues for discussion and resolution. What distinguishes these case scenarios and the case exemplars in education from case study research is the intention to use them for teaching purposes.

Country Case Studies

Then there are case studies of programs, projects, and even countries, as in international development, where a whole-country study might be termed a case study or, in the context of the Organization for Economic Co-operation and Development (OECD), where an exploration is conducted of the state of the art of a subject, such as education or environmental science in one or several countries. This may be a contemporaneous study and/or what transpired in a program over a period of time. Such studies often do have a research base but frequently are reported accounts that do not detail the design, methodology, and analysis of the case, as a research case study would do, or report in ways that give readers a vicarious experience of what it was like to be there. Such case studies tend to be more knowledge and information-focused than experiential.

Case Study as History

Closer to a research context is case study as history—what transpired at a certain time in a certain place. This is likely to be supported by documentary evidence but not primary data gathering unless it is an oral history. In education, in the late 1970s, Stenhouse (1978) experimented with a case study archive. Using contemporaneous data gathering, primarily through interviewing, he envisaged this database, which he termed a “case record,” forming an archive from which different individuals,, at some later date, could write a “case study.” This approach uses case study as a documentary source to begin to generate a history of education, as the subtitle of Stenhouse’s 1978 paper indicates “Towards a contemporary history of education.”

Case Study Research

From here on, my focus is on case study research per se, adopting for this purpose the following definition:

Case study is an in-depth exploration from multiple perspectives of the complexity and uniqueness of a particular project, policy, institution or system in a “real-life” context. It is research based, inclusive of different methods and is evidence-led. ( Simons, 2009 , p. 21).

For further related definitions of case study, see Stake (1995) , Merriam (1998), and Chadderton & Torrance (2011) . And for definitions from a slightly different perspective, see Yin (2004) and Thomas (2011a) .

Not Defined by Method or Perspective

The inclusion of different methods in the definition quoted above definition signals that case study research is not defined by methodology or method. What defines case study is its singularity and the concept and boundary of the case. It is theoretically possible to conduct a case study using primarily quantitative data if this is the best way of providing evidence to inform the issues the case is exploring. It is equally possible to conduct case study that is mainly qualitative, to engage people with the experience of the case or to provide a rich portrayal of an event, project, or program.

Or one can design the case using mixed methods. This increases the options for learning from different ways of knowing and is sometimes preferred by stakeholders who believe it provides a firmer basis for informing policy. This is not necessarily the case but is beyond the scope of this chapter to explore. For further discussion of the complexities of mixing methods and the virtue of using qualitative methods and case study in a mixed method design, see Greene (2007) .

Case study research may also be conducted from different standpoints—realist, interpretivist, or constructivist, for example. My perspective falls within a constructivist, interpretivist framework. What interests me is how I and those in the case perceive and interpret what we find and how we construct or co-construct understandings of the case. This not only suits my predilection for how I see the world, but also my preferred phenomenological approach to interviewing and curiosity about people and how they act in social and professional life.

Qualitative Case Study Research

Qualitative case study research shares many characteristics with other forms of qualitative research, such as narrative, oral history, life history, ethnography, in-depth interview, and observational studies that utilize qualitative methods. However, its focus, purpose, and origins, in educational research at least, are a little different.

The focus is clearly the study of the singular. The purpose is to portray an in-depth view of the quality and complexity of social/educational programs or policies as they are implemented in specific sociopolitical contexts. What makes it qualitative is its emphasis on subjective ways of knowing, particularly the experiential, practical, and presentational rather than the propositional ( Heron, 1992 , 1999 ) to comprehend and communicate what transpired in the case.

Characteristic Features and Advantages

Case study research is not method dependent, as noted earlier, nor is it constrained by resources or time. Although it can be conducted over several years, which provides an opportunity to explore the process of change and explain how and why things happened, it can equally be carried out contemporaneously in a few days, weeks, or months. This flexibility is extremely useful in many contexts, particularly when a change in policy or unforeseen issues in the field require modifying the design.

Flexibility extends to reporting. The case can be written up in different lengths and forms to meet different audience needs and to maximize use (see the section on Reporting). Using the natural language of participants and familiar methods (like interview, observation, oral history) also enables participants to engage in the research process, thereby contributing significantly to the generation of knowledge of the case. As I have indicated elsewhere ( Simons, 2009 ), “This is both a political and epistemological point. It signals a potential shift in the power base of who controls knowledge and recognizes the importance of co-constructing perceived reality through the relationships and joint understandings we create in the field” (p. 23).

Possible Disadvantages

If one is an advocate, identifying advantages of a research approach is easier than pointing out its disadvantages, something detractors are quite keen to do anyway! But no approach is perfect, and here are some of the issues that often trouble people about case study research. The “sample of one” is an obvious issue that worries those convinced that only large samples can constitute valid research and especially if this is to inform policy. Understanding complexity in depth may not be a sufficient counterargument, and I suspect there is little point in trying to persuade otherwise For frequently, this perception is one of epistemological and methodological, if not ideological, preference.

However, there are some genuine concerns that many case researchers face: the difficulty of processing a mass of data; of “telling the truth” in contexts where people may be identifiable; personal involvement, when the researcher is the main instrument of data gathering; and writing reports that are data-based, yet readable in style and length. But one issue that concerns advocates and nonadvocates alike is how inferences are drawn from the single case.

Answers to some of these issues are covered in the sections that follow. Whether they convince may again be a question of preference. However, it is worth noting here that I do not think we should seek to justify these concerns in terms identified by other methodologies. Many of them are intrinsic to the nature and strength of qualitative case study research.

Subjectivity, for instance, both of participants and researcher is inevitable, as it is in many other qualitative methodologies. This is often the basis on which we act. Rather than see this as bias or something to counter, it is an intelligence that is essential to understanding and interpreting the experience of participants and stakeholders. Such subjectivity needs to be disciplined, of course, through procedures that examine both the validity of individuals’ representations of “their truth”, and demonstrate how the researcher took a reflexive approach to monitoring how his or her own values and predilections may have unduly influenced the data.

Types of Case Study

There are numerous types of case study, too many to categorize, I think, as there are overlaps between them. However, attempts have been made to do this and, for those who value typologies, I refer them to Bassey (1999) and, for a more extended typology, to Thomas (2011b) . A slightly different approach is taken by Gomm, Hammersley, and Foster (2004) in annotating the different emphases in major texts on case study. What I prefer to do here is to highlight a few familiar types to focus the discussion that follows on the practice of case study research.

Stake (1995) offers a threefold distinction that is helpful when it comes to practice, he says, because it influences the methods we choose to gather data (p. 4). He distinguishes between an intrinsic case study , one that is studied to learn about the particular case itself and an instrumental case study , in which we choose a case to gain insight into a particular issue (i.e., the case is instrumental to understanding something else; p. 3). The collective case study is what its name suggests: an extension of the instrumental to several cases.

Theory-led or theory-generated case study is similarly self-explanatory, the first starting from a specific theory that is tested through the case; the second constructing a theory through interpretation of data generated in the case. In other words, one ends rather than begins with a theory. In qualitative case study research, this is the more familiar route. The theory of the case becomes the argument or story you will tell.

Evaluation case study requires a slightly longer description as this is my context of practice, one which has influenced the way I conduct case study and what I choose to emphasize in this chapter. An evaluation case study has three essential features: to determine the value of the case, to include and balance different interests and values, and to report findings to a range of stakeholders in ways that they can use. The reasons for this may be found in the interlude that follows, which offers a brief characterization of the social and ethical practice of evaluation and why qualitative methods are so important in this practice.

Interlude: Social and Ethical Practice of Evaluation

Evaluation is a social practice that documents, portrays, and seeks to understand the value of a particular project, program, or policy. This can be determined by different evaluation methodologies, of course. But the value of qualitative case study is that it is possible to discern this value without decontextualizing the data. While the focus of the case is usually a project, program, policy, or some unit within, studies of key individuals, what I term case profiles , may be embedded within the overall case. In some instances, these profiles, or even shorter cameos of individuals, may be quite prominent. For it is through the perceptions, interpretations, and interactions of people that we learn how policies and programs are enacted ( Kushner, 2000 , p. 12). The program is still the main focus of analysis, but, in exploring how individuals play out their different roles in the program, we get closer to the actual experience and meaning of the program in practice.

Case study evaluation is often commissioned from an external source (government department or other agency) keen to know the worth of publicly funded programs and policies to inform future decision making. It needs to be responsive to issues or questions identified by stakeholders, who often have different values and interests in the expected outcomes and appreciate different perspectives of the program in action. The context also is often highly politicized, and interests can conflict. The task of the evaluator in such situations becomes one of including and balancing all interests and values in the program fairly and justly.

This is an inherently political process and requires an ethical practice that offers participants some protection over the personal data they give as part of the research and agreed audiences access to the findings, presented in ways they can understand. Negotiating what information becomes public can be quite difficult in singular settings where people are identifiable and intricate or problematic transactions have been documented. The consequences that ensue from making knowledge public that hitherto was private may be considerable for those in the case. It may also be difficult to portray some of the contextual detail that would enhance understanding for readers.

The ethical stance that underpins the case study research and evaluation I conduct stems from a theory of ethics that emphasizes the centrality of relationships in the specific context and the consequences for individuals, while remaining aware of the research imperative to publicly report. It is essentially an independent democratic process based on the concepts of fairness and justice, in which confidentiality, negotiation, and accessibility are key principles ( MacDonald, 1976 ; Simons, 2009 , pp. 96–111; and Simons 2010 ). The principles are translated into specific procedures to guide the collection, validation, and dissemination of data in the field. These include:

engaging participants and stakeholders in identifying issues to explore and sometimes also in interpreting the data;

documenting how different people interpret and value the program;

negotiating what data becomes public respecting both the individual’s “right to privacy” and the public’s “right to know”;

offering participants opportunities to check how their data are used in the context of reporting;

reporting in language and forms accessible to a wide range of audiences;

disseminating to audiences within and beyond the case.

For further discussion of the ethics of democratic case study evaluation and examples of their use in practice, see Simons (2000 , 2006 , 2009 , chapter 6, 2010 ).

Designing Case Study Research

Design issues in case study sometimes take second place to those of data gathering, the more exciting task perhaps in starting research. However, it is critical to consider the design at the outset, even if changes are required in practice due to the reality of what is encountered in the field. In this sense, the design of case study is emergent, rather than preordinate, shaped and reshaped as understanding of the significance of foreshadowed issues emerges and more are discovered.

Before entering the field, there are a myriad of planning issues to think about related to stakeholders, participants, and audiences. These include whose values matter, whether to engage them in data gathering and interpretation, the style of reporting appropriate for each, and the ethical guidelines that will underpin data collection and reporting. However, here I emphasize only three: the broad focus of the study, what the case is a case of, and framing questions/issues. These are steps often ignored in an enthusiasm to gather data, resulting in a case study that claims to be research but lacks the basic principles required for generation of valid, public knowledge.

Conceptualize the Topic

First, it is important that the topic of the research is conceptualized in a way that it can be researched (i.e., it is not too wide). This seems an obvious point to make, but failure to think through precisely what it is about your research topic you wish to investigate will have a knock-on effect on the framing of the case, data gathering, and interpretation and may lead, in some instances, to not gathering or analyzing data that actually informs the topic. Further conceptualization or reconceptualization may be necessary as the study proceeds, but it is critical to have a clear focus at the outset.

What Constitutes the Case

Second, I think it is important to decide what would constitute the case (i.e., what it is a case of) and where the boundaries of this lie. This often proves more difficult than first appears. And sometimes, partly because of the semifluid nature of the way the case evolves, it is only possible to finally establish what the case is a case of at the end. Nevertheless, it is useful to identify what the case and its boundaries are at the outset to help focus data collection while maintaining an awareness that these may shift. This is emergent design in action.

In deciding the boundary of the case, there are several factors to bear in mind. Is it bounded by an institution or a unit within an institution, by people within an institution, by region, or by project, program or policy,? If we take a school as an example, the case could be comprised of the principal, teachers, and students, or the boundary could be extended to the cleaners, the caretaker, the receptionist, people who often know a great deal about the subnorms and culture of the institution.

If the case is a policy or particular parameter of a policy, the considerations may be slightly different. People will still be paramount—those who generated the policy and those who implemented it—but there is likely also to be a political culture surrounding the policy that had an influence on the way the policy evolved. Would this be part of the case?

Whatever boundary is chosen, this may change in the course of conducting the study when issues arise that can only be understood by going to another level. What transpires in a classroom, for example, if this is the case, is often partly dependent on the support of the school leadership and culture of the institution and this, in turn, to some extent is dependent on what resources are allocated from the local education administration. Much like a series of Russian dolls, one context inside the other.

Unit of analysis

Thinking about what would constitute the unit of analysis— a classroom, an institution, a program, a region—may help in setting the boundaries of the case, and it will certainly help when it comes to analysis. But this is a slightly different issue from deciding what the case is a case of. Taking a health example, the case may be palliative care support, but the unit of analysis the palliative care ward or wards. If you took the palliative care ward as the unit of analysis this would be as much about how palliative care was exercised in this or that ward than issues about palliative care support in general. In other words, you would need to have specific information and context about how this ward was structured and managed to understand how palliative care was conducted in this particular ward. Here, as in the school example above, you would need to consider which of the many people who populate the ward form part of the case—nurses, interns, or doctors only, or does it extend to patients, cleaners, nurse aides, and medical students?

Framing Questions and Issues

The third most important consideration is how to frame the study, and you are likely to do this once you have selected the site or sites for study. There are at least four approaches. You could start with precise questions, foreshadowed issues ( Smith & Pohland, 1974 ), theories, or a program logic. To some extent, your choice will be dictated by the type of case you have chosen, but also by your personal preference for how to conduct it—in either a structured or open way.

Initial questions give structure; foreshadowed issues more freedom to explore. In qualitative case study, foreshadowed issues are more common, allowing scope for issues to change as the study evolves, guided by participants’ perspectives and events in the field. With this perspective, it is more likely that you will generate a theory of the case toward the end, through your interpretation and analysis.

If you are conducting an instrumental case study, staying close to the questions or foreshadowed issues is necessary to be sure you gain data that will illuminate the central focus of the study. This is critical if you are exploring issues across several cases, although it is possible to do a cross-case analysis from cases that have each followed a different route to discovering significant issues.

Opting to start with a theoretical framework provides a basis for formulating questions and issues, but it can also constrain the study to only those questions/issues that fit the framework. The same is true with using program logic to frame the case. This is an approach frequently adopted in evaluation case study where the evaluator, individually or with stakeholders, examines how the aims and objectives of the program relate to the activities designed to promote it and the outcomes and impacts expected. It provides direction, although it can lead to simply confirming what was anticipated, rather than documenting what transpired in the case.

Whichever approach you choose to frame the case, it is useful to think about the rationale or theory for each question and what methods would best enable you to gain an understanding of them. This will not only start a reflexive process of examining your choices—an important aspect of the process of data gathering and interpretation—it will also aid analysis and interpretation further down the track.

Methodology and Methods

Qualitative case study research, as already noted, appeals to subjective ways of knowing and to a primarily qualitative methodology, that captures experiential understanding ( Stake, 2010 , pp. 56–70). It follows that the main methods of data gathering to access this way of knowing will be qualitative. Interviewing, observation, and document analysis are the primary three, often supported by critical incidents, focus groups, cameos, vignettes, diaries/journals, and photographs. Before gathering any primary data, however, it is useful to search relevant existing sources (written or visual) to learn about the antecedents and context of a project, program, or policy as a backdrop to the case. This can sharpen framing questions, avoid unnecessary data gathering, and shorten the time needed in the field.

Given that there are excellent texts on qualitative methods (see, for example, Denzin & Lincoln, 1994 ; Seale, 1999 ; Silverman, 2000 , 2004 ), I will not discuss all potential relevant methods here, but simply focus on the qualities of the primary methods that are particularly appropriate for case study research.

Primary Qualitative Data Gathering Methods

Interviewing.

The most effective style of interviewing in qualitative case study research to gain in-depth data, document multiple perspectives and experiences and explore contested issues is the unstructured interview, active listening and open questioning are paramount, whatever prequestions or foreshadowed issues have been identified. This can include photographs—a useful starting point with certain cultural groups and the less articulate, to encourage them to tell their story through connecting or identifying with something in the image.

The flexibility of unstructured interviewing has three further advantages for understanding participants’ experiences. First, through questioning, probing, listening, and, above all, paying attention to the silences and what they mean, you can get closer to the meaning of participants’ experiences. It is not always what they say.

Second, unstructured interviewing is useful for engaging participants in the process of research. Instead of starting with questions and issues, invite participants to tell their stories or reflect on specific issues, to conduct their own self-evaluative interview, in fact. Not only will they contribute their particular perspective to the case, they will also learn about themselves, thereby making the process of research educative for them as well as for the audiences of the research.

Third, the open-endedness of this style of interviewing has the potential for creating a dialogue between participants and the researcher and between the researcher and the public, if enough of the dialogue is retained in the publication ( Bellah, Madsen, Sullivan, Swidler, & Tipton, 1985 ).

Observations

Observations in case study research are likely to be close-up descriptions of events, activities, and incidents that detail what happens in a particular context. They will record time, place, specific incidents, transactions, and dialogue, and note characteristics of the setting and of people in it without preconceived categories or judgment. No description is devoid of some judgment in selection, of course, but, on the whole, the intent is to describe the scene or event “as it is,” providing a rich, textured description to give readers a sense of what it was like to be there or provide a basis for later interpretation.

Take the following excerpt from a study of the West Bromwich Operatic Society. It is the first night of a new production, The Producers , by this amateur operatic society. This brief excerpt is from a much longer observation of the overture to the first evening’s performance, detailing exactly what the production is, where it is, and why there is such a tremendous sense of atmosphere and expectation surrounding the event. Space prevents including the whole observation, but I hope you can get a glimmer of the passion and excitement that precedes the performance:

Birmingham, late November, 2011, early evening.... Bars and restaurants spruce up for the evening’s trade. There is a chill in the air but the party season is just starting....

A few hundred yards away, past streaming traffic on Suffolk Street, Queensway, an audience is gathering at the New Alexandra Theatre. The foyer windows shine in the orange sodium night. Above each one is the rubric: WORLD CLASS THEATRE.

Inside the preparatory rituals are being observed; sweets chosen, interval drinks ordered and programmes bought. People swap news and titbits about the production.... The bubble of anticipation grows as the 5-minute warning sounds. People make their way to the auditorium. There have been so many nights like this in the past 110 years since a man named William Coutts invested £10,000 to build this palace of dreams.... So many fantasies have been played under this arch: melodramas and pantomimes, musicals and variety.... So many audiences, settling down in their tip-up seats, wanting to be transported away from work, from ordinariness and private troubles.... The dimming lights act like a mother’s hush. You could touch the silence. Boinnng! A spongy thump on a bass drum, and the horns pipe up that catchy, irrepressible, tasteless tune and already you’re singing under your breath, ‘Springtime for Hitler and Germany....’ The orchestra is out of sight in the pit. There’s just the velvet curtain to watch as your fingers tap along. What’s waiting behind? Then it starts it to move. Opening night.... It’s opening night! ( Matarasso, 2012 , pp. 1–2)

For another and different example—a narrative observation of an everyday but unique incident that details date, time, place, and experience—see Simons (2009 , p. 60).

Such naturalistic observations are also useful in contexts where we cannot understand what is going on through interviewing alone—in cultures with which we are less familiar or where key actors may not share our language or have difficulty expressing it. Careful description in these situations can help identify key issues, discover the norms and values that exist in the culture, and, if sufficiently detailed, allow others to cross corroborate what significance we draw from these observations. This last point is very important to avoid the danger in observation of ascribing motivations to people and meanings to transactions.

Finally, naturalistic observations are very important in highly politicized environments, often the case in commissioned evaluation case study, where individuals in interview may try to elude the “truth” or press on you that their view is the “right” view of the situation. In these contexts, naturalistic observations not only enable you to document interactions as you perceive them, but they also provide a cross-check on the veracity of information obtained in interviews.

Document analysis

Analysis of documents, as already intimated, is useful for establishing what historical antecedents might exist to provide a springboard for contemporaneous data gathering. In most cases, existing documents are also extremely pertinent for understanding the policy context.

In a national policy case study I conducted on a major curriculum change, the importance of preexisting documentation was brought home to me sharply when certain documentation initially proved elusive to obtain. It was difficult to believe that it did not exist, as the evolution of the innovation involved several parties who had not worked together before. There was bound, I thought, to be minuted meetings sharing progress and documentation of the “new” curriculum. In the absence of some crucial documents, I began to piece together the story through interviewing. Only there were gaps, and certain issues did not make sense.

It was only when I presented two versions of what I discerned had transpired in the development of this initiative in an interim report eighteen months into the study that things started to change. Subsequent to the meeting at which the report was presented, the “missing” documents started to appear. Suddenly found. What lay behind the “missing documents,” something I suspected from what certain individuals did and did not say in interview, was a major difference of view about how the innovation evolved, who was key in the process, and whose voice was more important in the context. Political differences, in other words, that some stakeholders were trying to keep from me. The emergence of the documents enabled me to finally produce an accurate and fair account.

This is an example of the importance of having access to all relevant documents relating to a program or policy in order to study it fairly. The other major way in which document analysis is useful in case study is for understanding the values, explicit and hidden, in policy and program documents and in the organization where the program or policy is implemented. Not to be ignored as documents are photographs, and these, too, can form the basis of a cultural and value analysis of an organization ( Prosser, 2000 ).

Creative artistic approaches

Increasingly, some case study researchers are employing creative approaches associated with the arts as a means of data gathering and analysis. Artistic approaches have often been used in representing findings, but less frequently in data gathering and interpretation ( Simons & McCormack, 2007 ). A major exception is the work of Richardson (1994) , who sees the very process of writing as an interpretative act, and of Cancienne and Snowber (2003) , who argue for movement as method.

The most familiar of these creative and artistic forms are written—narratives and short stories ( Clandinin & Connelly, 2000 ; Richardson, 1994 ; Sparkes, 2002 ), poems or poetic form ( Butler-Kisber, 2010 ; Duke, 2007 ; Richardson, 1997 ; Sparkes & Douglas, 2007 ), cameos of people, or vignettes of situations. These can be written by participants or by the researcher or developed in partnership. They can also be shared with participants to further interpret the data. But photographs also have a long history in qualitative research for presenting and constructing understanding ( Butler-Kisber, 2010 ; Collier, 1967 ; Prosser, 2000 ; Rugang, 2006 ; Walker, 1993 ).

Less common are other visual forms of gathering data, such as “draw and write” ( Sewell, 2011 ), artefacts, drawings, sketches, paintings, and collages, although all forms are now on the increase. For examples of the use of collage in data gathering, see Duke (2007) and Butler-Kisber (2010) , and for charcoal drawing, Elliott (2008) .

In qualitative inquiry broadly, these creative approaches are now quite common. And in the context of arts and health in particular (see, for example, Frank, 1997 ; Liamputtong & Rumbold, 2008 ; Spouse, 2000 ), we can see how artistic approaches illuminate in-depth understanding. However, in case study research to date, I think narrative forms have tended to be most prominent.

Finally, for capturing the quality and essence of peoples’ experience, nothing could be more revealing than a recording of their voices. Video diaries—self-evaluative portrayals by individuals of their perspectives, feelings, or experience of an event or situation—are a most potent way both of gaining understanding and communicating that to others. It is rather more difficult to gain access for observational videos, but they are useful for documentation and have the potential to engage participants and stakeholders in the interpretation.

Getting It All Together

Case study is so often associated with story or with a report of some event or program that it is easy to forget that much analysis and interpretation has gone on before we reach this point. In many case study reports, this process is hidden, leaving the reader with little evidence on which to assess the validity of the findings and having to trust the one who wrote the tale.

This section briefly outlines possibilities, first, for analyzing and interpreting data, and second, for how to communicate the findings to others. However it is useful to think of these together and indeed, at the start, because decisions about how you report may influence how you choose to make sense of the data. Your choice may also vary according to the context of the study—what is expected or acceptable—and your personal predilections, whether you prefer a more rational than intuitive mode of analysis, for example, or a formal or informal style of writing up that includes images, metaphor, narratives, or poetic forms.

Analyzing and Interpreting Data

When it comes to making sense of data, I make a distinction between analysis—a formal inductive process that seeks to explain—and interpretation, a more intuitive process that gains understanding and insight from a holistic grasp of data, although these may interact and overlap at different stages.

The process, whichever emphasis you choose, is one of reducing or transforming a large amount of data to themes that can encapsulate the overarching meaning in the data. This involves sorting, refining, and refocusing data until they make sense. It starts at the beginning with preliminary hunches, sometimes called “interpretative asides” or “working hypotheses,” later moving to themes, analytic propositions, or a theory of the case.

There are many ways to conduct this process. Two strategies often employed are concept mapping —a means of representing data visually to explore links between related concepts—and progressive focusing ( Parlett & Hamilton, 1976 ), the gradual reframing of initially identified issues into themes that are then further interpreted to generate findings. Each of these strategies tends to have three stages: initial sense making, identification of themes, and examination of patterns and relationships between them.

If taking a formal analytic approach to the task, the data would likely be broken down into segments or datasets (coded and categorized) and then reordered and explored for themes, patterns, and possible propositions. If adopting a more intuitive process, you might focus on identifying insights through metaphors and images, lateral thinking, or puzzling over paradoxes and ambiguities in the data, after first immersing yourself in the total dataset, reading and re-reading interview scripts, observations and field notes to get a sense of the whole. Trying out different forms of making sense through poetry, vignettes, cameos, narratives, collages, and drawing are further creative ways to interpret data, as are photographs taken in the case arranged to explain or tell the story of the case.

Reporting Case Study Research

Narrative structure and story.

As indicated in the introduction, telling a story is often associated with case study and some think this is what a case study is. In one sense, it is and, given that story is the natural way in which we learn ( Okri, 1997 ), it is a useful framework both for gathering data and for communicating case study findings. Not any story will do however. To count as research, it must be authentic, grounded in data, interpreted and analyzed to convey the meaning of the case.

There are several senses in which story is appropriate in qualitative case study: in capturing stories participants tell, in generating a narrative structure that makes sense of the case (i.e., the story you will tell), and in deciding how you communicate this narrative (i.e., in story form). If you choose a written story form (and advice here can be sought from Harrington (2003) and Caulley (2008) ), it needs to be clearly structured, well written, and contain only the detail that is necessary to give readers the vicarious experience of what it was like in the case. If the story is to be communicated in other ways, through, for example, audio or videotape, or computer or personal interaction, the same applies, substituting visual and interpersonal skill for written.

Matching forms of reporting to audience

The art of reporting is strongly connected to usability, so forms of reporting need to connect to the audiences we hope to inform: how they learn, what kind of evidence they value, and what kind of reporting maximizes the chances they will use the findings to promote policies and programs in the interests of beneficiaries. As Okri (1997) further reminds us, the writer only does half the work; the reader does the other (p. 41).

There may be other considerations as well: how open are commissioners to receiving stories of difficulties, as well as success stories? What might they need to hear beyond what is sought in the technical brief? And through what style of reporting would you try and persuade them? If conducting noncommissioned case study research, the scope for different forms of reporting is wider. In academia, for instance, many institutions these days accept creative and artistic forms of reporting when supported by supervisors and appreciated by examiners.

Styles of Reporting

The most obvious form of reporting is linear, often starting with a short executive summary and a brief description of focus and context, followed by methodology, the case study or thematic analysis, findings, and conclusions or implications. Conclusion-led reporting is similar in terms of its formality, but simply starts the other way around. From the conclusions drawn from the analyzed data, it works backward to tell the story through narrative, verbatim, and observational data of how these conclusions were reached. Both have a strong story line. The intent is analytic and explanatory.

Quite a different approach is to engage the reader in the experience and veracity of the case. Rather like constructing a portrait or editing a documentary film, this involves the sifting, constructing, re-ordering of frames, events and episodes to tell a coherent story primarily through interview excerpts, observations, vignettes, and critical incidents that depict what transpired in the case. Interpretation is indirect through the weaving of the data. The story can start at any point provided the underlying narrative structure is maintained to establish coherence ( House, 1980 , p. 116).

Different again, and from the other end of a continuum, is a highly interpretative account that may use similar ways of presenting data but weaves a story from the outset that is highly interpretative. Engaging metaphor, images, short stories, contradictions, paradoxes, and puzzles, it is invariably interesting to read and can be most persuasive. However, the evidence is less visible and therefore less open to alternative interpretations.

Even more persuasive is a case study that uses artistic forms to communicate the story of the case. Paintings, poetic form, drawings, photography, collage, and movement can all be adopted to report findings, whether the data was acquired using these forms or by other means. The arts-based inquiry movement ( Mullen & Finley, 2003 ) has contributed hugely to the validation and legitimation of artistic and creative ways of representing qualitative research findings. The journal Qualitative Inquiry contains many good examples, but see also Liamputtong & Rumbold (2008) . Such artistic forms of representation may not be for everyone or appropriate in some contexts, but they do have the power to engage an audience and the potential to facilitate use.

Generalization in Case Study Research

One of the potential limitations of case study often proposed is that it is impossible to generalize. This is not so. However, the way in which one generalizes from a case is different from that adopted in traditional forms of social science research that utilize large samples (randomly selected) and statistical procedures and which assume regularities in the social world that allow cause and effect to be determined. In this form of research inferences from data are stated as formal propositions that apply to all in the target population. See Donmoyer (1990) for an argument on the restricted nature of this form of generalization when considering single-case studies.

Making inferences from cases with a qualitative data set arises more from a process of interpretation in context, appealing to tacit and situated understanding for acceptance of their validity. Such inferences are possible where the context and experience of the case is richly described so the reader can recognize and connect with the events and experiences portrayed. There are two ways to examine how to reach these generalized understandings. One is to generalize from the case to other cases of a similar or dissimilar nature. The other is to see what we learn in-depth from the uniqueness of the single case itself.

Generalizing from the Single Case

A common approach to generalization and one most akin to a propositional form is cross-case generalization. In a collective or multi-site case study, each case is explored to see if issues that arise in one case also exist in other cases and what interconnecting themes there are between them. This kind of generalization has a degree of abstraction and potential for theorizing and is often welcomed by commissioners of research concerned that findings from the single case do not provide an adequate or “safe” basis for policy determination.

However, there are four additional ways to generalize from the single case, all of which draw more on tacit knowledge and recognition of context, although in different ways. In naturalistic generalization , first proposed by Stake (1978) , generalization is reached on the basis of recognition of similarities and differences to cases with which we are familiar. To enable such recognition, the case needs to feature rich description; people’s voices; and enough detail of time, place, and context to provide a vicarious experience to help readers discern what is similar and dissimilar to their own context ( Stake, 1978 ).

Situated generalization ( Simons, Kushner, Jones, & James, 2003 ) is close to the concept of naturalistic generalization in relying for its generality on retaining a connectedness with the context in which it first evolved. However, it has an extra dimension in a practice context. This notion of generalization was identified in an evaluation of a research project that engaged teachers in and with research. Here, in addition to the usual validity criteria to establish the warrant for the findings, the generalization was seen as dependable if trust existed between those who conducted the research (teachers, in this example) and those thinking about using it (other teachers). In other words, beyond the technical validity of the research, teachers considered using the findings in their own practice because they had confidence in those who generated them. This is a useful way to think about generalization if we wish research findings to improve professional practice.

The next two concepts of generalization— concept and process generalization —relate more to what you discover in making sense of the case. As you interpret and analyze, you begin to generate a theory of the case that makes sense of the whole. Concepts may be identified that make sense in the one case but have equal significance in other cases of a similar kind, even if the contexts are different.

It is the concept that generalizes, not the specific content or context. This may be similar to the process Donmoyer (2008) identifies of “intellectual generalization” (quoted by Butler-Kisber, 2010 , p. 15) to indicate the cognitive understanding one can gain from qualitative accounts even if settings are quite different.

The same is true for generalization of a process. It is possible to identify a significant process in one case (or several cases) that is transferable to other contexts, irrespective of the precise content and contexts of those other cases. An example here is the collaborative model for sustainable school self-evaluation I identified in researching school self-evaluation in a number of schools and countries ( Simons, 2002 ). Schools that successfully sustained school self-evaluation had an infrastructure that was collaborative at all stages of the evaluation process from design to conduct of the study, to analyzing the results and to reporting the findings. This ensured that the whole school was involved and that results were discussed and built into the ongoing development of school policies and practice. In other cases, different processes may be discovered that have applicability in a range of contexts. As with concept generalization, it is the process that generalizes not the substantive content or specific context.

Particularization

The forms of generalization discussed above are useful when we have to justify case study in a research or policy context. But the overarching justification for how we learn from case study is particularization —a rich portrayal of insights and understandings interpreted in the particular context. Several authors have made this point ( Stake, 1995 ; Flyvberg, 2006 ; Simons 2009 ). Stake puts it most sharply when he observes that “The real business of case study is particularization, not generalization” (p. 8), referring here to the main reason for studying the singular, which is to understand the uniqueness of the case itself.

My perspective (explored further in Simons, 1996 ; Simons, 2009 , p. 239; Simons & McCormack, 2007 ) is similar in that I believe the “real” strength of case study lies in the insights we gain from in-depth study of the particular. But I also argue for the universality of such insights—if we get it “right.” By which I mean that if we are able to capture and report the uniqueness, the essence, of the case in all its particularity and present this in a way we can all recognize, we will discover something of universal significance. This is something of a paradox. The more you learn in depth about the particularity of one person, situation, or context, the more likely you are to discover something universal. This process of reaching understanding has support both from the way in which many discoveries are made in science and in how we learn from artists, poets, and novelists, who reach us by communicating a recognizable truth about individuals, human relationships, and/or social contexts.

This concept of particularization is far from new, as the quotation from a preface to a book written in 1908 attests. Stephen Reynolds, the author of A Poor Man’s House , notes that the substance of the book was first recorded in a journal, kept for purposes of fiction, and in letters to one of his friends, but fiction proved an inappropriate medium. He felt that the life and the people were so much better than anything he could invent. The book therefore consists of the journal and letters drawn together to present a picture of a typical poor man’s house and life, much as we might draw together a range of data to present a case study. It is not the substance of the book that concerns us here but the methodological relevance to case study research. Reynolds notes that the conclusions expressed are tentative and possibly go beyond this man’s life, so he thought some explanation of the way he arrived at them was needed:

Educated people usually deal with the poor man’s life deductively; they reason from the general to the particular; and, starting with a theory, religious, philanthropic, political, or what not, they seek, and too easily find, among the millions of poor, specimens—very frequently abnormal—to illustrate their theories. With anything but human beings, that is an excellent method. Human beings, unfortunately, have individualities. They do what, theoretically, they ought not to do, and leave undone those things they ought to do. They are even said to possess souls—untrustworthy things beyond the reach of sociologists. The inductive method—reasoning from the particular to the general... should at least help to counterbalance the psychological superficiality of the deductive method. ( Reynolds, 1908 : preface) 1

Slightly overstated perhaps, but the point is well made. In our search for general laws, we not only lose sight of the uniqueness and humanity of individuals, but reduce them in the process, failing to present their experience in any “real” sense. What is astonishing about the quotation is that it was written over a century ago and yet many still argue today that you cannot generalize from the particular.

Going even further back, in 1798, Blake proclaimed that “To Generalize is to be an Idiot. To Particularize is the Alone Distinction of Merit.” In research, we may not wish to make such a strong distinction: these processes both have their uses in different kinds of research. But there is a major point here for the study of the particular that Wilson (2008) notes in commenting on Blake’s perception when he says: “Favouring the abstract over the concrete, one ‘sees all things only thro’ the narrow chinks of his cavern”’ (referring here to Blake’s The Marriage of Heaven and Hell [1793]; in Wilson, 2008 , p. 62). The danger Wilson is pointing to here is that abstraction relies heavily on what we know from our past understanding of things, and this may prevent us experiencing a concrete event directly or “apprehend[ing] a particular moment” ( Wilson, 2008 , p. 63).

Blake had a different mission, of course, than case researchers, and he was not himself free from abstractions, as Wilson points out, although he fought hard “to break through mental barriers to something unique and living” ( Wilson, 2008 , p. 65). It is this search for the “unique and living” and experiencing the “isness” of the particular that we should take from the Blake example to remind ourselves of the possibility of discovering something “new,” beyond our current understanding of the way things are.

Focusing on particularization does not diminish the usefulness of case study research for policy makers or practitioners. Grounded in recognizable experience, the potential is there to reach a range of audiences and to facilitate use of the findings. It may be more difficult for those who seek formal generalizations that seem to offer a safe basis for policy making to accept case study reports. However, particular stories often hold the key to why policies have or have not worked well in the past. It is not necessary to present long cases—a criticism frequently levelled—to demonstrate the story of the case. Such case stories can be most insightful for policy makers who, like many of us in everyday life, often draw inferences from a single instance or case, whatever the formal evidence presented. “I am reminded of the story of....”

The case for studying the particular to inform practice in professional contexts needs less persuasion because practitioners can recognize the content and context quite readily and make the inference to their own particular context ( Simons et al., 2003 ). In both sets of circumstances—policy and practice—it is more a question of whether the readers of our case research accept the validity of findings determined in this way, how they choose to learn, and our skill in telling the case study story.

Conclusion and Future Directions

In this chapter, I have presented an argument for case study research, making the case, in particular, for using qualitative methods to highlight what it is that qualitative case study research can bring to the study of social and educational programs. I outlined the various ways in which case study is commonly used before focusing directly on case study as a major mode of research inquiry, noting characteristics it shares with other qualitative methodologies, as well as itsdifference and the difficulties it is sometimes perceived to have. The chapter emphasizes the importance of thinking through what the case is, to be sure that the issues explored and the data generated do illuminate this case and not any other.

But there is still more to be done. In particular, I think we need to be more adventurous in how we craft and report the case. I suspect we may have been too cautious in the past in how we justified case study research, borrowing concepts from other disciplines and forms of educational research. More than 40 years on, it is time to take a greater risk—in demonstrating the intrinsic nature of case study and what it can offer to our understanding of human and social situations.

I have already drawn attention to the need to design the case, although this could be developed further to accentuate the uniqueness of the particular case. One way to do this is to feature individuals more in the design itself, not only to explore programs and policies through perspectives of key actors or groups and transactions between them, which to some extent happens already, but also to get them to characterize what makes the context unique. This is the reversal of many a design framework that starts with the logic of a program and takes forward the argument for personal evaluation ( Kushner, 2000 ), noted in the interlude on evaluation. Apart from this attention to design, there are three other issues I think we need to explore further: the warrant for creative methods in case study, more imaginative reporting; and how we learn from a study of the singular.

Warrant for More Creative Methods in Case Study Research

The promise that creative methods have for eliciting in-depth understanding and capturing the unusual, the idiosyncratic, the uniqueness of the case, was mentioned in the methods section. Yet, in case study research, particularly in program and policy contexts, we have few good examples of the use of artistic approaches for eliciting and interpreting data, although more, as acknowledged later, for presenting it. This may be because case study research is often conducted in academic or policy environments, where propositional ways of knowing are more valued.

Using creative and artistic forms in generating and interpreting case study data offers a form of evidence that acknowledges experiential understanding in illuminating the uniqueness of the case. The question is how to establish the warrant for this way of knowing and persuade others of its virtue. The answer is simple. By demonstrating the use of these methods in action, by arguing for a different form of validity that matches the intrinsic nature of the method, and, above all, by good examples.

Representing Findings to Engage Audiences in Learning

In evaluative and research policy contexts, where case study is often the main mode of inquiry or part of a broader study, case study reports often take a formal structure or sometimes, where the context is receptive, a portrayal or interpretative form. But, too often, the qualitative is an add-on to a story told by other means or reduced to issues in which the people who gave rise to the data are no longer seen. However, there are many ways to put them center stage.

Tell good stories and tell them well. Or, let key actors tell their own stories. Explore the different ways technology can help. Make video clips that demonstrate events in context, illustrate interactions between people, give voice to participants—show the reality of the program, in other words. Use graphics to summarize key issues and interactive, cartoon technology, as seen on some TED presentations, to summarize and visually show the complexity of the case. Video diaries were mentioned in the methods section: seeing individuals tell their tales directly is a powerful way of communicating, unhindered by “our” sense making. Tell photo stories. Let the photos convey the narrative, but make sure the structure of the narrative is evident to ensure coherence. These are just the beginnings. Those skilled in information technology could no doubt stretch our imagination further.

One problem and a further question concerns our audiences. Will they accept these modes of communication? Maybe not, in some contexts. However, there are three points I wish to leave you with. First, do not presume that they won’t. If people are fully present in the story and the complexity is not diminished, those reading, watching, or hearing about the case will get the message. If you are worried about how commissioners might respond, remember that they are no different from any other stakeholder or participant when it comes to how they learn from human experience. Witness the reference to Okri (1997) earlier about how we learn.

Second, when you detect that the context requires a more formal presentation of findings, respond according to expectation but also include elements of other forms of presentation. Nudge a little in the direction of creativity. Third, simply take a chance, that risk I spoke about earlier. Challenge the status quo. Find situations and contexts where you can fully represent the qualitative nature of the experience in the cases you study with creative forms of interpretation and representation. And let the audience decide.

Learning from a Study of the Singular

Finally, to return to the issue of “generalization” in case study that worries some audiences. I pointed out in the generalization section several ways in which it is possible to generalize from case study research, not in a formal propositional sense or from a case to a population, but by retaining a connection with the context in which the generalization first arose—that is, to realize in-depth understanding in context in different circumstances and situations. However, I also emphasized that, in many instances, it is particularization from which we learn. That is the point of the singular case study, and it is an art to perceive and craft the case in ways that we can.

Acknowledgments

Parts of this chapter build on ideas first explored in Simons, 2009 .

I am grateful to Bob Williams for pointing out the relevance of this quotation from Reynolds to remind us that “there is nothing new under the sun” and that we sometimes continue to engage endlessly in debates that have been well rehearsed before.

Adams, T. ( 2012 ) ‘ Olympics 2012: Team GB falters but London shines bright on opening day ’, Observer, 29.07.12.

Google Scholar

Google Preview

Bassey, M. ( 1999 ). Case study research in educational settings . Buckingham: Open University Press.

Bellah, R. N. , Madsen, R. , Sullivan, W. M. , Swidler, A. , & Tipton, S. M. ( 1985 ). Habits of the heart . London: Harper and Row.

Blake, W. (1798– 1809 ). Annotations to Sir Joshua Reynolds’s Discourses , pp. xvii–xcviii (c. 1798–1809) repr. In Complete Writings , ed.   Geoffrey Keynes   (1957). ‘Discourse II,’ annotations to Sir Joshua Reynolds, Discourses (c. 1808) .

Butler-Kisber, L. ( 2010 ). Qualitative inquiry: Thematic, narrative and arts-informed perspectives . London: Sage.

Cancienne, M. B. , & Snowber, C. N. ( 2003 ). Writing rhythm: Movement as method.   Qualitative Inquiry , 9 (2), 237–253.

Caulley, D. N. ( 2008 ). Making qualitative research reports less boring: The techniques of writing creative nonfiction.   Qualitative Inquiry , 14 (3) pp. 424–449.

Chadderton, C. , & Torrance, H. ( 2011 ). Case study. In B. Somekh & C. Lewin (Eds.), Theory and methods in social research . (2nd ed. pp 53–60). London and Thousand Oaks, CA: Sage.

Clandinin, D. J. & Connelly, F. M. ( 2000 ) Narrative inquiry: Experience and story in qualitative research. Ist edn. SanFrancisco: Jossey Bass Publishers.

Collier, J., Jr. ( 1967 ). Visual anthropology: Photography as a research method . New York: Holt, Reinhart, & Winston.

Denzin, N. K. & Lincoln, Y. S. (Eds.) ( 1994 ) Handbook of Qualitative Research , London and Thousand Oaks, CA:Sage

Donmoyer, R. ( 1990 ). Generalization and the single case study. In E. W. Eisner & A. Peshkin (Eds.), Qualitative inquiry in education (pp. 175–200). New York: Teachers College Press.

Donmoyer, R. ( 2008 ). Generalizability. In L. M. Givens (Ed.), The Sage encyclopedia of qualitative inquiry (vol. 2, pp. 371–372). Thousand Oaks, CA: Sage.

Duke, S. (2007). A narrative case study evaluation of the role of the Nurse Consultant in palliative care. PhD thesis, University of Southampton, England.

Elliott, J. (2008). Dance mirrors: Embodying, actualizing and operationalizing a dance experience in a healthcare context. PhD thesis, University of Ulster, Belfast.

Frank. A. ( 1997 ). Enacting illness stories: When, what, why. In H. L. Nelson (Ed.), Stories and their limits (pp. 31–49). London: Routledge.

Flyvberg, B. ( 2006 ). Five misunderstandings about case-study research.   Qualitative Inquiry , 12 (2), 219–245.

Gomm, R. , Hammersley, M. , & Foster, P. (Eds.). ( 2004 ). Case study method: Key issues, key texts . [First published in 2000]. London and Thousand Oaks, CA: Sage.

Greene, J. C. ( 2000 ). Understanding social programs through evaluation. In N. K. Denzin and Y. S. Lincoln (Eds.), The handbook of qualitative research (2nd ed., pp. 981–999). Thousand Oaks, CA: Sage.

Greene, J. C. ( 2007 ). Mixing methods in social inquiry . San Francisco: Jossey Bass.

Greenhalgh, T. , & Worrall, J. G. ( 1997 ). From EBM to CSM: The evolution of context-sensitive medicine.   Journal of Evaluation in Clinical Practice , 3 (2), 105–108.

Harrington, W. ( 2003 ). What journalism can offer ethnography.   Qualitative Inquiry . 9 (1), 90–114.

Heron, J. ( 1992 ). Feeling and personhood . Sage: London.

Heron, J. ( 1999 ). The complete facilitator’s handbook . London: Kogan Page.

House, E. R. ( 1980 ). Evaluating with validity . London, Beverly Hills, CA: Sage.

House, E. R. ( 1993 ). Professional evaluation: Social impact and political consequences . Newbury Park and London: Sage

Kushner, S. ( 2000 ). Personalizing evaluation . London and Thousand Oaks, CA: Sage.

Liamputtong, P. , & Rumbold, J. (Eds.). ( 2008 ). Knowing differently: Arts-based and collaborative research methods . New York: Nova Science Publishers.

MacDonald, B. ( 1976 ). Evaluation and the control of education. In D. Tawney (Ed.), Curriculum evaluation today: Trends and implications . Schools Council Research Studies (pp. 125– 136). London: Macmillan.

Matarasso, F. ( 2012 ). West Bromwich Operatic Society: fine art of musical theatre . West Bromwich, UK: Multistory.

Merriam, S. B. ( 1988 ). Case study research in education: A qualitative Approach . San Francisco: Jossey Bass.

Mullen, C. A. , & Finley, S. (Eds.). ( 2003 ). Arts-based approaches to qualitative inquiry [Special Issue].   Qualitative Inquiry , 9 (2), 165–329.

Okri, B. ( 1997 ). A way of being free . London: Phoenix.

Parlett, M. , & Hamilton, D. ( 1976 ). Evaluation as illumination: A new approach to the study of innovatory programmes. In G. Glass (Ed.), Evaluation studies review annual, I (pp. 140–157). [First published in 1972 as Occasional Paper 9, Centre for Research in the Educational Sciences, University of Edinburgh.] Beverly Hills: CA: Sage.

Platt, J. ( 2007 ). Case study. In W. Outhwaite & S. P. Turner (Eds.), The Sage handbook of social science methodology (pp. 100–118). London: Sage.

Prosser, J. ( 2000 ). The moral maze of image ethics. In H. Simons & R. Usher (Eds.), Situated ethics in educational research (pp. 116–132). London and New York: Routledge/Falmer.

Ragin, C. C. ( 1992 ). Cases of “What is a case?” In C. C. Ragin & H. S. Becker (Eds.), What is a case?: Exploring the foundations of social inquiry (pp. 1–17). Cambridge: Cambridge University Press.

Reynolds, S. S. (1908). A Poor Man’s House . The Project Guttenberg eBook, July 25, 2008 [eBook#26126]. Accessed February 26, 2013, http.//www.gutenberg.org

Richardson, L. ( 1994 ). Writing as a form of inquiry. In N. K. Denzin & Y. S. Lincoln (Eds.), Handbook of Qualitative Research (pp. 516–529). London: Sage.

Richardson, L. ( 1997 ). Fields of play (Constructing an academic life) . New Brunswick, NJ: Rutgers University Press.

Rugang, L. (2006). Chinese culture in globalisation: A multi-modal case study on visual discourse. PhD thesis, University of Southampton, England.

Seale, C. ( 1999 ). The quality of qualitative research . London and Thousand Oaks, CA: Sage.

Sewell, K. ( 2011 ). Researching sensitive issues: A critical appraisal of “draw and write” as a data collection technique in eliciting children’s perceptions.   International Journal of Research Methods in Education 34(2), pp.175–191.

Shaw, I. , & Gould, N. ( 2001 ). Qualitative research in social work: Context and method . London: Sage.

Silverman, D. ( 2000 ). Doing qualitative research: A practical handbook . London and Thousand Oaks. CA: Sage.

Silverman, D. (Ed.). ( 2004 ). Qualitative research: Theory, methods and practice (2nd ed.). London and Thousand Oaks, CA: Sage.

Simons, H. ( 1971 ). Innovation and the case study of schools.   Cambridge Journal of Education , 3 , 118–123.

Simons, H. (Ed.). ( 1980 ). Towards a science of the singular: Essays about case study in educational research and evaluation . Occasional Papers No. 10. Norwich, UK: Centre for Applied Research, University of East Anglia.

Simons, H. ( 1987 ). Getting to know schools in a democracy: The politics and process of evaluation . Lewes, UK: Falmer Press.

Simons, H. ( 1996 ). The paradox of case study.   Cambridge Journal of Education , 26 (2), 225–240.

Simons, H. ( 2000 ). Damned if you do, damned if you don’t: ethical and political dilemmas in evaluation. In H. Simons & R. Usher (Eds.) Situated ethics in educational research (pp.39– 55) London and New York: Routledge/Falmer.

Simons, H. ( 2002 ). School self-evaluation in a democracy. In D. Nevo (Ed.), School-based evaluation: An international perspective . Advances in Program Evaluation. London: Sage.

Simons, H. ( 2006 ). Ethics and evaluation. In I. F. Shaw , J. C. Greene , & M. M. Mark (Eds.), The international handbook of evaluation (pp. 243–265). London and Thousand Oaks, CA Sage.

Simons, H. ( 2009 ). Case study research in practice . London: Sage.

Simons, H. (2010). Democratic evaluation: Theory and practice. Paper prepared for Virtual Evaluation Conference, University of the Witwatersrand, Johannesburg, South Africa, May, 2010.

Simons, H. , Kushner, S. , Jones, K. , & James, D. ( 2003 ). From evidence-based practice to practice-based evidence: The idea of situated generalization.   Research Papers in Education: Policy and Practice , 18 (4), 347–364.

Simons, H. , & McCormack, B. ( 2007 ). Integrating arts-based inquiry in evaluation methodology.   Qualitative Inquiry , 13 (2) 292–311.

Simons, H. , & Usher, R. (Eds.). ( 2000 ). Situated ethics in educational research . London and New York: Routledge/Falmer.

Smith, L. M. , & Pohland, P. A. ( 1974 ). Education, technology, and the rural highlands. In R. H. P. Kraft ., L. M. Smith ., P. A. Pohland ., C. J. Brauner , & C. Gjerde (Eds.), Four evaluation examples: Anthropological, economic, narrative and portrayal (pp. 5–54), AERA Monograph Series on Curriculum Evaluation 7. Chicago: Rand McNally.

Sparkes, A. ( 2002 ). Telling tales in sport and physical activity: A qualitative journey . Champaign, IL: Human Kinetics Press.

Sparkes, A. C. , & Douglas, K. ( 2007 ). Making the case for poetic representations: An example in action.   The Sport Psychologist , 21(2) , 170–190.

Spouse, J. ( 2000 ). Talking pictures: Investigating personal knowledge though illuminating artwork.   Nursing Times Research Journal , 5 (4), 253–261.

Stake, R. E. ( 1978 ) The case study method in social inquiry.   Educational Researcher , 7(2), 5–9.

Stake, R. E. ( 1995 ). The art of case study research . Thousand Oaks, CA and London: Sage.

Stake, R. E. ( 2010 ). Qualitative research: Studying how things work . New York & London: Guildford Press.

Stenhouse, L. ( 1978 ). Case study and case records: Towards a contemporary history of education.   British Educational Research Journal , 4 (2), 21–39.

Thomas, G. ( 2011 b)). A typology for the case study in social science following a review of definition, discourse and structure.   Qualitative Inquiry , 17 (6) , 511–521.

Thomas, G. ( 2011 a). How to do your case study: A guide for students and researchers . London: Sage.

Walker, R. ( 1993 ). Finding a silent voice for the researcher: Using photographs in evaluation and research. In M. Schratz (Ed.), Qualitative voices in educational research (pp. 72–92). Lewes, UK: Falmer Press.

Wilson, E. G. ( 2008 ). Against happiness . New York: Sarah Crichton Books.

Yin, R. K. ( 2004 ). Case study research: Design and methods . Thousand Oaks, CA and London: Sage.

Zucker, D. M. ( 2001 ). Using case study methodology in nursing research.   Qualitative Report , 6 (2) June.

• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

Case Study Evaluation: Past, Present and Future Challenges: Volume 15

Table of contents, case study evaluation: past, present and future challenges, advances in program evaluation, copyright page, list of contributors, introduction, case study, methodology and educational evaluation: a personal view.

This chapter gives one version of the recent history of evaluation case study. It looks back over the emergence of case study as a sociological method, developed in the early years of the 20th Century and celebrated and elaborated by the Chicago School of urban sociology at Chicago University, starting throughout the 1920s and 1930s. Some of the basic methods, including constant comparison, were generated at that time. Only partly influenced by this methodological movement, an alliance between an Illinois-based team in the United States and a team at the University of East Anglia in the United Kingdom recast the case method as a key tool for the evaluation of social and educational programmes.

Letters from a Headmaster ☆ Originally published in Simons, H. (Ed.) (1980). Towards a Science of the Singular: Essays about Case Study in Educational Research and Evaluation. Occasional Papers No. 10. Norwich, UK: Centre for Applied Research, University of East Anglia.

Story telling and educational understanding ☆ previously published in occasional papers #12, evaluation centre, university of western michigan, 1978..

The full ‘storytelling’ paper was written in 1978 and was influential in its time. It is reprinted here, introduced by an Author's reflection on it in 2014. The chapter describes the author’s early disenchantment with traditional approaches to educational research.

He regards educational research as, at best, a misnomer, since little of it is preceded by a search . Entitled educational researchers often fancy themselves as scientists at work. But those whom they attempt to describe are often artists at work. Statistical methodologies enable educational researchers to measure something, but their measurements can neither capture nor explain splendid teaching.

Since such a tiny fraction of what is published in educational research journals influences school practitioners, professional researchers should risk trying alternative approaches to uncovering what is going on in schools.

Story telling is posited as a possible key to producing insights that inform and ultimately improve educational practice. It advocates openness to broad inquiry into the culture of the educational setting.

Case Study as Antidote to the Literal

Much programme and policy evaluation yields to the pressure to report on the productivity of programmes and is perforce compliant with the conditions of contract. Too often the view of these evaluations is limited to a literal reading of the analytical challenge. If we are evaluating X we look critically at X1, X2 and X3. There might be cause for embracing adjoining data sources such as W1 and Y1. This ignores frequent realities that an evaluation specification is only an approximate starting point for an unpredictable journey into comprehensive understanding; that the specification represents only that which is wanted by the sponsor, and not all that may be needed ; and that the contractual specification too often insists on privileging the questions and concerns of a few. Case study evaluation proves an alternative that allows for the less-than-literal in the form of analysis of contingencies – how people, phenomena and events may be related in dynamic ways, how context and action have only a blurred dividing line and how what defines the case as a case may only emerge late in the study.

Thinking about Case Studies in 3-D: Researching the NHS Clinical Commissioning Landscape in England

What is our unit of analysis and by implication what are the boundaries of our cases? This is a question we grapple with at the start of every new project. We observe that case studies are often referred to in an unreflective manner and are often conflated with geographical location. Neat units of analysis and clearly bounded cases usually do not reflect the messiness encountered during qualitative fieldwork. Others have puzzled over these questions. We briefly discuss work to problematise the use of households as units of analysis in the context of apartheid South Africa and then consider work of other anthropologists engaged in multi-site ethnography. We have found the notion of ‘following’ chains, paths and threads across sites to be particularly insightful.

We present two examples from our work studying commissioning in the English National Health Service (NHS) to illustrate our struggles with case studies. The first is a study of Practice-based Commissioning groups and the second is a study of the early workings of Clinical Commissioning Groups. In both instances we show how ideas of what constituted our unit of analysis and the boundaries of our cases became less clear as our research progressed. We also discuss pressures we experienced to add more case studies to our projects. These examples illustrate the primacy for us of understanding interactions between place, local history and rapidly developing policy initiatives. Understanding cases in this way can be challenging in a context where research funders hold different views of what constitutes a case.

The Case for Evaluating Process and Worth: Evaluation of a Programme for Carers and People with Dementia

A case study methodology was applied as a major component of a mixed-methods approach to the evaluation of a mobile dementia education and support service in the Bega Valley Shire, New South Wales, Australia. In-depth interviews with people with dementia (PWD), their carers, programme staff, family members and service providers and document analysis including analysis of client case notes and client database were used.

The strengths of the case study approach included: (i) simultaneous evaluation of programme process and worth, (ii) eliciting the theory of change and addressing the problem of attribution, (iii) demonstrating the impact of the programme on earlier steps identified along the causal pathway (iv) understanding the complexity of confounding factors, (v) eliciting the critical role of the social, cultural and political context, (vi) understanding the importance of influences contributing to differences in programme impact for different participants and (vii) providing insight into how programme participants experience the value of the programme including unintended benefits.

The broader case of the collective experience of dementia and as part of this experience, the impact of a mobile programme of support and education, in a predominately rural area grew from the investigation of the programme experience of ‘individual cases’ of carers and PWD. Investigation of living conditions, relationships, service interactions through observation and increased depth of interviews with service providers and family members would have provided valuable perspectives and thicker description of the case for increased understanding of the case and strength of the evaluation.

The Collapse of “Primary Care” in Medical Education: A Case Study of Michigan’s Community/University Health Partnerships Project

This chapter describes a case study of a social change project in medical education (primary care), in which the critical interpretive evaluation methodology I sought to use came up against the “positivist” approach preferred by senior figures in the medical school who commissioned the evaluation.

I describe the background to the study and justify the evaluation approach and methods employed in the case study – drawing on interviews, document analysis, survey research, participant observation, literature reviews, and critical incidents – one of which was the decision by the medical school hierarchy to restrict my contact with the lay community in my official evaluation duties. The use of critical ethnography also embraced wider questions about circuits of power and the social and political contexts within which the “social change” effort occurred.

Central to my analysis is John Gaventa’s theory of power as “the internalization of values that inhibit consciousness and participation while encouraging powerlessness and dependency.” Gaventa argued, essentially, that the evocation of power has as much to do with preventing decisions as with bringing them about. My chosen case illustrated all three dimensions of power that Gaventa originally uncovered in his portrait of self-interested Appalachian coal mine owners: (1) communities were largely excluded from decision making power; (2) issues were avoided or suppressed; and (3) the interests of the oppressed went largely unrecognized.

The account is auto-ethnographic, hence the study is limited by my abilities, biases, and subject positions. I reflect on these in the chapter.

The study not only illustrates the unique contribution of case study as a research methodology but also its low status in the positivist paradigm adhered to by many doctors. Indeed, the tension between the potential of case study to illuminate the complexities of community engagement through thick description and the rejection of this very method as inherently “flawed” suggests that medical education may be doomed to its neoliberal fate for some time to come.

‘Lead’ Standard Evaluation

This is a personal narrative, but I trust not a self-regarding one. For more years than I care to remember I have been working in the field of curriculum (or ‘program’) evaluation. The field by any standards is dispersed and fragmented, with variously ascribed purposes, roles, implicit values, political contexts, and social research methods. Attempts to organize this territory into an ‘evaluation theory tree’ (e.g. Alkin, M., & Christie, C. (2003). An evaluation theory tree. In M. Alkin (Ed.), Evaluation roots: Tracing theorists’ views and influences (pp. 12–65). Thousand Oaks, CA: Sage) have identified broad types or ‘branches’, but the migration of specific characteristics (like ‘case study’) or individual practitioners across the boundaries has tended to undermine the analysis at the level of detail, and there is no suggestion that it represents a cladistic taxonomy. There is, however, general agreement that the roots of evaluation practice tap into a variety of cultural sources, being grounded bureaucratically in (potentially conflicting) doctrines of accountability and methodologically in discipline-based or pragmatically eclectic formats for systematic social enquiry.

In general, this diversity is not treated as problematic. The professional evaluation community has increasingly taken the view (‘let all the flowers grow’) that evaluation models can be deemed appropriate across a wide spectrum, with their appropriateness determined by the nature of the task and its context, including in relation to hybrid studies using mixed models or displaying what Geertz (Geertz, C. (1980/1993). Blurred genres: The refiguration of social thought. The American Scholar , 49(2), 165–179) called ‘blurred genres’. However, from time to time historic tribal rivalries re-emerge as particular practitioners feel the need to defend their modus operandi (and thereby their livelihood) against paradigm shifts or governments and other sponsors of program evaluation seeking for ideological reasons to prioritize certain types of study at the expense of others. The latter possibility poses a potential threat that needs to be taken seriously by evaluators within the broad tradition showcased in this volume, interpretive qualitative case studies of educational programs that combine naturalistic description (often ‘thick’; Geertz, C. (1973). Thick description: Towards an interpretive theory of culture. In The interpretation of culture (pp. 3–30). New York, NY: Basic Books.) description with a values-orientated analysis of their implications. Such studies are more likely to seek inspiration from anthropology or critical discourse analysis than from the randomly controlled trials familiar in medical research or laboratory practice in the physical sciences, despite the impressive rigour of the latter in appropriate contexts. It is the risk of ideological allegiance that I address in this chapter.

Freedom from the Rubric

Twice-told tales how public inquiry could inform n of 1 case study research.

This chapter considers the usefulness and validity of public inquiries as a source of data and preliminary interpretation for case study research. Using two contrasting examples – the Bristol Inquiry into excess deaths in a children’s cardiac surgery unit and the Woolf Inquiry into a breakdown of governance at the London School of Economics (LSE) – I show how academics can draw fruitfully on, and develop further analysis from, the raw datasets, published summaries and formal judgements of public inquiries.

Academic analysis of public inquiries can take two broad forms, corresponding to the two main approaches to individual case study defined by Stake: instrumental (selecting the public inquiry on the basis of pre-defined theoretical features and using the material to develop and test theoretical propositions) and intrinsic (selecting the public inquiry on the basis of the particular topic addressed and using the material to explore questions about what was going on and why).

The advantages of a public inquiry as a data source for case study research typically include a clear and uncontested focus of inquiry; the breadth and richness of the dataset collected; the exceptional level of support available for the tasks of transcribing, indexing, collating, summarising and so on; and the expert interpretations and insights of the inquiry’s chair (with which the researcher may or may not agree). A significant disadvantage is that whilst the dataset collected for a public inquiry is typically ‘rich’, it has usually been collected under far from ideal research conditions. Hence, while public inquiries provide a potentially rich resource for researchers, those who seek to use public inquiry data for research must justify their choice on both ethical and scientific grounds.

Evaluation as the Co-Construction of Knowledge: Case Studies of Place-Based Leadership and Public Service Innovation

This chapter introduces the notion of the ‘Innovation Story’ as a methodological approach to public policy evaluation, which builds in greater opportunity for learning and reflexivity.

The Innovation Story is an adaptation of the case study approach and draws on participatory action research traditions. It is a structured narrative that describes a particular public policy innovation in the personalised contexts in which it is experienced by innovators. Its construction involves a discursive process through which involved actors tell their story, explain it to others, listen to their questions and co-construct knowledge of change together.

The approach was employed to elaborate five case studies of place-based leadership and public service innovation in the United Kingdom, The Netherlands and Mexico. The key findings are that spaces in which civic leaders come together from different ‘realms’ of leadership in a locality (community, business, professional managers and political leaders) can become innovation zones that foster inventive behaviour. Much depends on the quality of civic leadership, and its capacity to foster genuine dialogue and co-responsibility. This involves the evaluation seeking out influential ideas from below the level of strategic management, and documenting leadership activities of those who are skilled at ‘boundary crossing’ – for example, communicating between sectors.

The evaluator can be a key player in this process, as a convenor of safe spaces for actors to come together to discuss and deliberate before returning to practice. Our approach therefore argues for a particular awareness of the political nature of policy evaluation in terms of negotiating these spaces, and the need for politically engaged evaluators who are skilled in facilitating collective learning processes.

Evaluation Noir: The Other Side of the Experience

What are the boundaries of a case study, and what should new evaluators do when these boundaries are breached? How does a new evaluator interpret the breakdown of communication, how do new evaluators protect themselves when the evaluation fails? This chapter discusses the journey of an evaluator new to the field of qualitative evaluative inquiry. Integrating the perspective of a senior evaluator, the authors reflect on three key experiences that informed the new evaluator. The authors hope to provide a rare insight into case study practice as emotional issues turn out to be just as complex as the methodology used.

About the Editors

About the authors.

• Jill Russell
• Trisha Greenhalgh
• Saville Kushner

We’re listening — tell us what you think

Something didn’t work….

Report bugs here

All feedback is valuable

Please share your general feedback

Join us on our journey

Platform update page.

Visit emeraldpublishing.com/platformupdate to discover the latest news and updates

Questions & More Information

Answers to the most commonly asked questions here

Search form

• Table of Contents
• Troubleshooting Guide
• A Model for Getting Started
• Justice Action Toolkit
• Best Change Processes
• Databases of Best Practices
• Online Courses
• Ask an Advisor
• Subscribe to eNewsletter
• Community Stories
• YouTube Channel
• About the Tool Box
• How to Use the Tool Box
• Privacy Statement
• Workstation/Check Box Sign-In
• Online Training Courses
• Capacity Building Training
• Training Curriculum - Order Now
• Community Check Box Evaluation System
• Build Your Toolbox
• Facilitation of Community Processes
• Community Health Assessment and Planning
• Section 1. A Framework for Program Evaluation: A Gateway to Tools

Chapter 36 Sections

• Section 2. Community-based Participatory Research
• Section 3. Understanding Community Leadership, Evaluators, and Funders: What Are Their Interests?
• Section 4. Choosing Evaluators
• Section 5. Developing an Evaluation Plan
• Section 6. Participatory Evaluation
• Main Section

This section is adapted from the article "Recommended Framework for Program Evaluation in Public Health Practice," by Bobby Milstein, Scott Wetterhall, and the CDC Evaluation Working Group.

Around the world, there exist many programs and interventions developed to improve conditions in local communities. Communities come together to reduce the level of violence that exists, to work for safe, affordable housing for everyone, or to help more students do well in school, to give just a few examples.

But how do we know whether these programs are working? If they are not effective, and even if they are, how can we improve them to make them better for local communities? And finally, how can an organization make intelligent choices about which promising programs are likely to work best in their community?

Over the past years, there has been a growing trend towards the better use of evaluation to understand and improve practice.The systematic use of evaluation has solved many problems and helped countless community-based organizations do what they do better.

Despite an increased understanding of the need for - and the use of - evaluation, however, a basic agreed-upon framework for program evaluation has been lacking. In 1997, scientists at the United States Centers for Disease Control and Prevention (CDC) recognized the need to develop such a framework. As a result of this, the CDC assembled an Evaluation Working Group comprised of experts in the fields of public health and evaluation. Members were asked to develop a framework that summarizes and organizes the basic elements of program evaluation. This Community Tool Box section describes the framework resulting from the Working Group's efforts.

Before we begin, however, we'd like to offer some definitions of terms that we will use throughout this section.

By evaluation , we mean the systematic investigation of the merit, worth, or significance of an object or effort. Evaluation practice has changed dramatically during the past three decades - new methods and approaches have been developed and it is now used for increasingly diverse projects and audiences.

Throughout this section, the term program is used to describe the object or effort that is being evaluated. It may apply to any action with the goal of improving outcomes for whole communities, for more specific sectors (e.g., schools, work places), or for sub-groups (e.g., youth, people experiencing violence or HIV/AIDS). This definition is meant to be very broad.

Examples of different types of programs include:

• Direct service interventions (e.g., a program that offers free breakfast to improve nutrition for grade school children)
• Community mobilization efforts (e.g., organizing a boycott of California grapes to improve the economic well-being of farm workers)
• Research initiatives (e.g., an effort to find out whether inequities in health outcomes based on race can be reduced)
• Surveillance systems (e.g., whether early detection of school readiness improves educational outcomes)
• Advocacy work (e.g., a campaign to influence the state legislature to pass legislation regarding tobacco control)
• Social marketing campaigns (e.g., a campaign in the Third World encouraging mothers to breast-feed their babies to reduce infant mortality)
• Infrastructure building projects (e.g., a program to build the capacity of state agencies to support community development initiatives)
• Training programs (e.g., a job training program to reduce unemployment in urban neighborhoods)
• Administrative systems (e.g., an incentive program to improve efficiency of health services)

Program evaluation - the type of evaluation discussed in this section - is an essential organizational practice for all types of community health and development work. It is a way to evaluate the specific projects and activities community groups may take part in, rather than to evaluate an entire organization or comprehensive community initiative.

Stakeholders refer to those who care about the program or effort. These may include those presumed to benefit (e.g., children and their parents or guardians), those with particular influence (e.g., elected or appointed officials), and those who might support the effort (i.e., potential allies) or oppose it (i.e., potential opponents). Key questions in thinking about stakeholders are: Who cares? What do they care about?

This section presents a framework that promotes a common understanding of program evaluation. The overall goal is to make it easier for everyone involved in community health and development work to evaluate their efforts.

Why evaluate community health and development programs?

The type of evaluation we talk about in this section can be closely tied to everyday program operations. Our emphasis is on practical, ongoing evaluation that involves program staff, community members, and other stakeholders, not just evaluation experts. This type of evaluation offers many advantages for community health and development professionals.

For example, it complements program management by:

• Helping to clarify program plans
• Improving communication among partners
• Gathering the feedback needed to improve and be accountable for program effectiveness

It's important to remember, too, that evaluation is not a new activity for those of us working to improve our communities. In fact, we assess the merit of our work all the time when we ask questions, consult partners, make assessments based on feedback, and then use those judgments to improve our work. When the stakes are low, this type of informal evaluation might be enough. However, when the stakes are raised - when a good deal of time or money is involved, or when many people may be affected - then it may make sense for your organization to use evaluation procedures that are more formal, visible, and justifiable.

How do you evaluate a specific program?

Before your organization starts with a program evaluation, your group should be very clear about the answers to the following questions:.

• What will be evaluated?
• What criteria will be used to judge program performance?
• What standards of performance on the criteria must be reached for the program to be considered successful?
• What evidence will indicate performance on the criteria relative to the standards?
• What conclusions about program performance are justified based on the available evidence?

To clarify the meaning of each, let's look at some of the answers for Drive Smart, a hypothetical program begun to stop drunk driving.

• Drive Smart, a program focused on reducing drunk driving through public education and intervention.
• The number of community residents who are familiar with the program and its goals
• The number of people who use "Safe Rides" volunteer taxis to get home
• The percentage of people who report drinking and driving
• The reported number of single car night time crashes (This is a common way to try to determine if the number of people who drive drunk is changing)
• 80% of community residents will know about the program and its goals after the first year of the program
• The number of people who use the "Safe Rides" taxis will increase by 20% in the first year
• The percentage of people who report drinking and driving will decrease by 20% in the first year
• The reported number of single car night time crashes will decrease by 10 % in the program's first two years
• A random telephone survey will demonstrate community residents' knowledge of the program and changes in reported behavior
• Logs from "Safe Rides" will tell how many people use their services
• Information on single car night time crashes will be gathered from police records
• Are the changes we have seen in the level of drunk driving due to our efforts, or something else? Or (if no or insufficient change in behavior or outcome,)
• Should Drive Smart change what it is doing, or have we just not waited long enough to see results?

The following framework provides an organized approach to answer these questions.

A framework for program evaluation

Program evaluation offers a way to understand and improve community health and development practice using methods that are useful, feasible, proper, and accurate. The framework described below is a practical non-prescriptive tool that summarizes in a logical order the important elements of program evaluation.

The framework contains two related dimensions:

• Steps in evaluation practice, and
• Standards for "good" evaluation.

The six connected steps of the framework are actions that should be a part of any evaluation. Although in practice the steps may be encountered out of order, it will usually make sense to follow them in the recommended sequence. That's because earlier steps provide the foundation for subsequent progress. Thus, decisions about how to carry out a given step should not be finalized until prior steps have been thoroughly addressed.

However, these steps are meant to be adaptable, not rigid. Sensitivity to each program's unique context (for example, the program's history and organizational climate) is essential for sound evaluation. They are intended to serve as starting points around which community organizations can tailor an evaluation to best meet their needs.

• Engage stakeholders
• Describe the program
• Focus the evaluation design
• Gather credible evidence
• Justify conclusions
• Ensure use and share lessons learned

Understanding and adhering to these basic steps will improve most evaluation efforts.

The second part of the framework is a basic set of standards to assess the quality of evaluation activities. There are 30 specific standards, organized into the following four groups:

• Feasibility

These standards help answer the question, "Will this evaluation be a 'good' evaluation?" They are recommended as the initial criteria by which to judge the quality of the program evaluation efforts.

Engage Stakeholders

Stakeholders are people or organizations that have something to gain or lose from what will be learned from an evaluation, and also in what will be done with that knowledge. Evaluation cannot be done in isolation. Almost everything done in community health and development work involves partnerships - alliances among different organizations, board members, those affected by the problem, and others. Therefore, any serious effort to evaluate a program must consider the different values held by the partners. Stakeholders must be part of the evaluation to ensure that their unique perspectives are understood. When stakeholders are not appropriately involved, evaluation findings are likely to be ignored, criticized, or resisted.

However, if they are part of the process, people are likely to feel a good deal of ownership for the evaluation process and results. They will probably want to develop it, defend it, and make sure that the evaluation really works.

That's why this evaluation cycle begins by engaging stakeholders. Once involved, these people will help to carry out each of the steps that follows.

Three principle groups of stakeholders are important to involve:

• People or organizations involved in program operations may include community members, sponsors, collaborators, coalition partners, funding officials, administrators, managers, and staff.
• People or organizations served or affected by the program may include clients, family members, neighborhood organizations, academic institutions, elected and appointed officials, advocacy groups, and community residents. Individuals who are openly skeptical of or antagonistic toward the program may also be important to involve. Opening an evaluation to opposing perspectives and enlisting the help of potential program opponents can strengthen the evaluation's credibility.

Likewise, individuals or groups who could be adversely or inadvertently affected by changes arising from the evaluation have a right to be engaged. For example, it is important to include those who would be affected if program services were expanded, altered, limited, or ended as a result of the evaluation.

• Primary intended users of the evaluation are the specific individuals who are in a position to decide and/or do something with the results.They shouldn't be confused with primary intended users of the program, although some of them should be involved in this group. In fact, primary intended users should be a subset of all of the stakeholders who have been identified. A successful evaluation will designate primary intended users, such as program staff and funders, early in its development and maintain frequent interaction with them to be sure that the evaluation specifically addresses their values and needs.

The amount and type of stakeholder involvement will be different for each program evaluation. For instance, stakeholders can be directly involved in designing and conducting the evaluation. They can be kept informed about progress of the evaluation through periodic meetings, reports, and other means of communication.

It may be helpful, when working with a group such as this, to develop an explicit process to share power and resolve conflicts . This may help avoid overemphasis of values held by any specific stakeholder.

Describe the Program

A program description is a summary of the intervention being evaluated. It should explain what the program is trying to accomplish and how it tries to bring about those changes. The description will also illustrate the program's core components and elements, its ability to make changes, its stage of development, and how the program fits into the larger organizational and community environment.

How a program is described sets the frame of reference for all future decisions about its evaluation. For example, if a program is described as, "attempting to strengthen enforcement of existing laws that discourage underage drinking," the evaluation might be very different than if it is described as, "a program to reduce drunk driving by teens." Also, the description allows members of the group to compare the program to other similar efforts, and it makes it easier to figure out what parts of the program brought about what effects.

Moreover, different stakeholders may have different ideas about what the program is supposed to achieve and why. For example, a program to reduce teen pregnancy may have some members who believe this means only increasing access to contraceptives, and other members who believe it means only focusing on abstinence.

Evaluations done without agreement on the program definition aren't likely to be very useful. In many cases, the process of working with stakeholders to develop a clear and logical program description will bring benefits long before data are available to measure program effectiveness.

There are several specific aspects that should be included when describing a program.

Statement of need

A statement of need describes the problem, goal, or opportunity that the program addresses; it also begins to imply what the program will do in response. Important features to note regarding a program's need are: the nature of the problem or goal, who is affected, how big it is, and whether (and how) it is changing.

Expectations

Expectations are the program's intended results. They describe what the program has to accomplish to be considered successful. For most programs, the accomplishments exist on a continuum (first, we want to accomplish X... then, we want to do Y...). Therefore, they should be organized by time ranging from specific (and immediate) to broad (and longer-term) consequences. For example, a program's vision, mission, goals, and objectives , all represent varying levels of specificity about a program's expectations.

Activities are everything the program does to bring about changes. Describing program components and elements permits specific strategies and actions to be listed in logical sequence. This also shows how different program activities, such as education and enforcement, relate to one another. Describing program activities also provides an opportunity to distinguish activities that are the direct responsibility of the program from those that are conducted by related programs or partner organizations. Things outside of the program that may affect its success, such as harsher laws punishing businesses that sell alcohol to minors, can also be noted.

Resources include the time, talent, equipment, information, money, and other assets available to conduct program activities. Reviewing the resources a program has tells a lot about the amount and intensity of its services. It may also point out situations where there is a mismatch between what the group wants to do and the resources available to carry out these activities. Understanding program costs is a necessity to assess the cost-benefit ratio as part of the evaluation.

Stage of development

A program's stage of development reflects its maturity. All community health and development programs mature and change over time. People who conduct evaluations, as well as those who use their findings, need to consider the dynamic nature of programs. For example, a new program that just received its first grant may differ in many respects from one that has been running for over a decade.

At least three phases of development are commonly recognized: planning , implementation , and effects or outcomes . In the planning stage, program activities are untested and the goal of evaluation is to refine plans as much as possible. In the implementation phase, program activities are being field tested and modified; the goal of evaluation is to see what happens in the "real world" and to improve operations. In the effects stage, enough time has passed for the program's effects to emerge; the goal of evaluation is to identify and understand the program's results, including those that were unintentional.

A description of the program's context considers the important features of the environment in which the program operates. This includes understanding the area's history, geography, politics, and social and economic conditions, and also what other organizations have done. A realistic and responsive evaluation is sensitive to a broad range of potential influences on the program. An understanding of the context lets users interpret findings accurately and assess their generalizability. For example, a program to improve housing in an inner-city neighborhood might have been a tremendous success, but would likely not work in a small town on the other side of the country without significant adaptation.

Logic model

A logic model synthesizes the main program elements into a picture of how the program is supposed to work. It makes explicit the sequence of events that are presumed to bring about change. Often this logic is displayed in a flow-chart, map, or table to portray the sequence of steps leading to program results.

Creating a logic model allows stakeholders to improve and focus program direction. It reveals assumptions about conditions for program effectiveness and provides a frame of reference for one or more evaluations of the program. A detailed logic model can also be a basis for estimating the program's effect on endpoints that are not directly measured. For example, it may be possible to estimate the rate of reduction in disease from a known number of persons experiencing the intervention if there is prior knowledge about its effectiveness.

The breadth and depth of a program description will vary for each program evaluation. And so, many different activities may be part of developing that description. For instance, multiple sources of information could be pulled together to construct a well-rounded description. The accuracy of an existing program description could be confirmed through discussion with stakeholders. Descriptions of what's going on could be checked against direct observation of activities in the field. A narrow program description could be fleshed out by addressing contextual factors (such as staff turnover, inadequate resources, political pressures, or strong community participation) that may affect program performance.

Focus the Evaluation Design

By focusing the evaluation design, we mean doing advance planning about where the evaluation is headed, and what steps it will take to get there. It isn't possible or useful for an evaluation to try to answer all questions for all stakeholders; there must be a focus. A well-focused plan is a safeguard against using time and resources inefficiently.

Depending on what you want to learn, some types of evaluation will be better suited than others. However, once data collection begins, it may be difficult or impossible to change what you are doing, even if it becomes obvious that other methods would work better. A thorough plan anticipates intended uses and creates an evaluation strategy with the greatest chance to be useful, feasible, proper, and accurate.

Among the issues to consider when focusing an evaluation are:

Purpose refers to the general intent of the evaluation. A clear purpose serves as the basis for the design, methods, and use of the evaluation. Taking time to articulate an overall purpose will stop your organization from making uninformed decisions about how the evaluation should be conducted and used.

There are at least four general purposes for which a community group might conduct an evaluation:

• To gain insight .This happens, for example, when deciding whether to use a new approach (e.g., would a neighborhood watch program work for our community?) Knowledge from such an evaluation will provide information about its practicality. For a developing program, information from evaluations of similar programs can provide the insight needed to clarify how its activities should be designed.
• To improve how things get done .This is appropriate in the implementation stage when an established program tries to describe what it has done. This information can be used to describe program processes, to improve how the program operates, and to fine-tune the overall strategy. Evaluations done for this purpose include efforts to improve the quality, effectiveness, or efficiency of program activities.
• To determine what the effects of the program are . Evaluations done for this purpose examine the relationship between program activities and observed consequences. For example, are more students finishing high school as a result of the program? Programs most appropriate for this type of evaluation are mature programs that are able to state clearly what happened and who it happened to. Such evaluations should provide evidence about what the program's contribution was to reaching longer-term goals such as a decrease in child abuse or crime in the area. This type of evaluation helps establish the accountability, and thus, the credibility, of a program to funders and to the community.
• Empower program participants (for example, being part of an evaluation can increase community members' sense of control over the program);
• Supplement the program (for example, using a follow-up questionnaire can reinforce the main messages of the program);
• Promote staff development (for example, by teaching staff how to collect, analyze, and interpret evidence); or
• Contribute to organizational growth (for example, the evaluation may clarify how the program relates to the organization's mission).

Users are the specific individuals who will receive evaluation findings. They will directly experience the consequences of inevitable trade-offs in the evaluation process. For example, a trade-off might be having a relatively modest evaluation to fit the budget with the outcome that the evaluation results will be less certain than they would be for a full-scale evaluation. Because they will be affected by these tradeoffs, intended users have a right to participate in choosing a focus for the evaluation. An evaluation designed without adequate user involvement in selecting the focus can become a misguided and irrelevant exercise. By contrast, when users are encouraged to clarify intended uses, priority questions, and preferred methods, the evaluation is more likely to focus on things that will inform (and influence) future actions.

Uses describe what will be done with what is learned from the evaluation. There is a wide range of potential uses for program evaluation. Generally speaking, the uses fall in the same four categories as the purposes listed above: to gain insight, improve how things get done, determine what the effects of the program are, and affect participants. The following list gives examples of uses in each category.

Some specific examples of evaluation uses

To gain insight:.

• Assess needs and wants of community members
• Identify barriers to use of the program
• Learn how to best describe and measure program activities

To improve how things get done:

• Refine plans for introducing a new practice
• Determine the extent to which plans were implemented
• Improve educational materials
• Enhance cultural competence
• Verify that participants' rights are protected
• Set priorities for staff training
• Make mid-course adjustments
• Clarify communication
• Determine if client satisfaction can be improved
• Compare costs to benefits
• Find out which participants benefit most from the program
• Mobilize community support for the program

To determine what the effects of the program are:

• Assess skills development by program participants
• Compare changes in behavior over time
• Decide where to allocate new resources
• Document the level of success in accomplishing objectives
• Demonstrate that accountability requirements are fulfilled
• Use information from multiple evaluations to predict the likely effects of similar programs

To affect participants:

• Reinforce messages of the program
• Stimulate dialogue and raise awareness about community issues
• Broaden consensus among partners about program goals
• Teach evaluation skills to staff and other stakeholders
• Gather success stories
• Support organizational change and improvement

The evaluation needs to answer specific questions . Drafting questions encourages stakeholders to reveal what they believe the evaluation should answer. That is, what questions are more important to stakeholders? The process of developing evaluation questions further refines the focus of the evaluation.

The methods available for an evaluation are drawn from behavioral science and social research and development. Three types of methods are commonly recognized. They are experimental, quasi-experimental, and observational or case study designs. Experimental designs use random assignment to compare the effect of an intervention between otherwise equivalent groups (for example, comparing a randomly assigned group of students who took part in an after-school reading program with those who didn't). Quasi-experimental methods make comparisons between groups that aren't equal (e.g. program participants vs. those on a waiting list) or use of comparisons within a group over time, such as in an interrupted time series in which the intervention may be introduced sequentially across different individuals, groups, or contexts. Observational or case study methods use comparisons within a group to describe and explain what happens (e.g., comparative case studies with multiple communities).

No design is necessarily better than another. Evaluation methods should be selected because they provide the appropriate information to answer stakeholders' questions, not because they are familiar, easy, or popular. The choice of methods has implications for what will count as evidence, how that evidence will be gathered, and what kind of claims can be made. Because each method option has its own biases and limitations, evaluations that mix methods are generally more robust.

Over the course of an evaluation, methods may need to be revised or modified. Circumstances that make a particular approach useful can change. For example, the intended use of the evaluation could shift from discovering how to improve the program to helping decide about whether the program should continue or not. Thus, methods may need to be adapted or redesigned to keep the evaluation on track.

Agreements summarize the evaluation procedures and clarify everyone's roles and responsibilities. An agreement describes how the evaluation activities will be implemented. Elements of an agreement include statements about the intended purpose, users, uses, and methods, as well as a summary of the deliverables, those responsible, a timeline, and budget.

The formality of the agreement depends upon the relationships that exist between those involved. For example, it may take the form of a legal contract, a detailed protocol, or a simple memorandum of understanding. Regardless of its formality, creating an explicit agreement provides an opportunity to verify the mutual understanding needed for a successful evaluation. It also provides a basis for modifying procedures if that turns out to be necessary.

As you can see, focusing the evaluation design may involve many activities. For instance, both supporters and skeptics of the program could be consulted to ensure that the proposed evaluation questions are politically viable. A menu of potential evaluation uses appropriate for the program's stage of development could be circulated among stakeholders to determine which is most compelling. Interviews could be held with specific intended users to better understand their information needs and timeline for action. Resource requirements could be reduced when users are willing to employ more timely but less precise evaluation methods.

Gather Credible Evidence

Credible evidence is the raw material of a good evaluation. The information learned should be seen by stakeholders as believable, trustworthy, and relevant to answer their questions. This requires thinking broadly about what counts as "evidence." Such decisions are always situational; they depend on the question being posed and the motives for asking it. For some questions, a stakeholder's standard for credibility could demand having the results of a randomized experiment. For another question, a set of well-done, systematic observations such as interactions between an outreach worker and community residents, will have high credibility. The difference depends on what kind of information the stakeholders want and the situation in which it is gathered.

Context matters! In some situations, it may be necessary to consult evaluation specialists. This may be especially true if concern for data quality is especially high. In other circumstances, local people may offer the deepest insights. Regardless of their expertise, however, those involved in an evaluation should strive to collect information that will convey a credible, well-rounded picture of the program and its efforts.

Having credible evidence strengthens the evaluation results as well as the recommendations that follow from them. Although all types of data have limitations, it is possible to improve an evaluation's overall credibility. One way to do this is by using multiple procedures for gathering, analyzing, and interpreting data. Encouraging participation by stakeholders can also enhance perceived credibility. When stakeholders help define questions and gather data, they will be more likely to accept the evaluation's conclusions and to act on its recommendations.

The following features of evidence gathering typically affect how credible it is seen as being:

Indicators translate general concepts about the program and its expected effects into specific, measurable parts.

Examples of indicators include:

• The program's capacity to deliver services
• The participation rate
• The level of client satisfaction
• The amount of intervention exposure (how many people were exposed to the program, and for how long they were exposed)
• Changes in participant behavior
• Changes in community conditions or norms
• Changes in the environment (e.g., new programs, policies, or practices)
• Longer-term changes in population health status (e.g., estimated teen pregnancy rate in the county)

Indicators should address the criteria that will be used to judge the program. That is, they reflect the aspects of the program that are most meaningful to monitor. Several indicators are usually needed to track the implementation and effects of a complex program or intervention.

One way to develop multiple indicators is to create a "balanced scorecard," which contains indicators that are carefully selected to complement one another. According to this strategy, program processes and effects are viewed from multiple perspectives using small groups of related indicators. For instance, a balanced scorecard for a single program might include indicators of how the program is being delivered; what participants think of the program; what effects are observed; what goals were attained; and what changes are occurring in the environment around the program.

Another approach to using multiple indicators is based on a program logic model, such as we discussed earlier in the section. A logic model can be used as a template to define a full spectrum of indicators along the pathway that leads from program activities to expected effects. For each step in the model, qualitative and/or quantitative indicators could be developed.

Indicators can be broad-based and don't need to focus only on a program's long -term goals. They can also address intermediary factors that influence program effectiveness, including such intangible factors as service quality, community capacity, or inter -organizational relations. Indicators for these and similar concepts can be created by systematically identifying and then tracking markers of what is said or done when the concept is expressed.

In the course of an evaluation, indicators may need to be modified or new ones adopted. Also, measuring program performance by tracking indicators is only one part of evaluation, and shouldn't be confused as a basis for decision making in itself. There are definite perils to using performance indicators as a substitute for completing the evaluation process and reaching fully justified conclusions. For example, an indicator, such as a rising rate of unemployment, may be falsely assumed to reflect a failing program when it may actually be due to changing environmental conditions that are beyond the program's control.

Sources of evidence in an evaluation may be people, documents, or observations. More than one source may be used to gather evidence for each indicator. In fact, selecting multiple sources provides an opportunity to include different perspectives about the program and enhances the evaluation's credibility. For instance, an inside perspective may be reflected by internal documents and comments from staff or program managers; whereas clients and those who do not support the program may provide different, but equally relevant perspectives. Mixing these and other perspectives provides a more comprehensive view of the program or intervention.

The criteria used to select sources should be clearly stated so that users and other stakeholders can interpret the evidence accurately and assess if it may be biased. In addition, some sources provide information in narrative form (for example, a person's experience when taking part in the program) and others are numerical (for example, how many people were involved in the program). The integration of qualitative and quantitative information can yield evidence that is more complete and more useful, thus meeting the needs and expectations of a wider range of stakeholders.

Quality refers to the appropriateness and integrity of information gathered in an evaluation. High quality data are reliable and informative. It is easier to collect if the indicators have been well defined. Other factors that affect quality may include instrument design, data collection procedures, training of those involved in data collection, source selection, coding, data management, and routine error checking. Obtaining quality data will entail tradeoffs (e.g. breadth vs. depth); stakeholders should decide together what is most important to them. Because all data have limitations, the intent of a practical evaluation is to strive for a level of quality that meets the stakeholders' threshold for credibility.

Quantity refers to the amount of evidence gathered in an evaluation. It is necessary to estimate in advance the amount of information that will be required and to establish criteria to decide when to stop collecting data - to know when enough is enough. Quantity affects the level of confidence or precision users can have - how sure we are that what we've learned is true. It also partly determines whether the evaluation will be able to detect effects. All evidence collected should have a clear, anticipated use.

By logistics , we mean the methods, timing, and physical infrastructure for gathering and handling evidence. People and organizations also have cultural preferences that dictate acceptable ways of asking questions and collecting information, including who would be perceived as an appropriate person to ask the questions. For example, some participants may be unwilling to discuss their behavior with a stranger, whereas others are more at ease with someone they don't know. Therefore, the techniques for gathering evidence in an evaluation must be in keeping with the cultural norms of the community. Data collection procedures should also ensure that confidentiality is protected.

Justify Conclusions

The process of justifying conclusions recognizes that evidence in an evaluation does not necessarily speak for itself. Evidence must be carefully considered from a number of different stakeholders' perspectives to reach conclusions that are well -substantiated and justified. Conclusions become justified when they are linked to the evidence gathered and judged against agreed-upon values set by the stakeholders. Stakeholders must agree that conclusions are justified in order to use the evaluation results with confidence.

The principal elements involved in justifying conclusions based on evidence are:

Standards reflect the values held by stakeholders about the program. They provide the basis to make program judgments. The use of explicit standards for judgment is fundamental to sound evaluation. In practice, when stakeholders articulate and negotiate their values, these become the standards to judge whether a given program's performance will, for instance, be considered "successful," "adequate," or "unsuccessful."

Analysis and synthesis

Analysis and synthesis are methods to discover and summarize an evaluation's findings. They are designed to detect patterns in evidence, either by isolating important findings (analysis) or by combining different sources of information to reach a larger understanding (synthesis). Mixed method evaluations require the separate analysis of each evidence element, as well as a synthesis of all sources to examine patterns that emerge. Deciphering facts from a given body of evidence involves deciding how to organize, classify, compare, and display information. These decisions are guided by the questions being asked, the types of data available, and especially by input from stakeholders and primary intended users.

Interpretation

Interpretation is the effort to figure out what the findings mean. Uncovering facts about a program's performance isn't enough to make conclusions. The facts must be interpreted to understand their practical significance. For example, saying, "15 % of the people in our area witnessed a violent act last year," may be interpreted differently depending on the situation. For example, if 50% of community members had watched a violent act in the last year when they were surveyed five years ago, the group can suggest that, while still a problem, things are getting better in the community. However, if five years ago only 7% of those surveyed said the same thing, community organizations may see this as a sign that they might want to change what they are doing. In short, interpretations draw on information and perspectives that stakeholders bring to the evaluation. They can be strengthened through active participation or interaction with the data and preliminary explanations of what happened.

Judgments are statements about the merit, worth, or significance of the program. They are formed by comparing the findings and their interpretations against one or more selected standards. Because multiple standards can be applied to a given program, stakeholders may reach different or even conflicting judgments. For instance, a program that increases its outreach by 10% from the previous year may be judged positively by program managers, based on standards of improved performance over time. Community members, however, may feel that despite improvements, a minimum threshold of access to services has still not been reached. Their judgment, based on standards of social equity, would therefore be negative. Conflicting claims about a program's quality, value, or importance often indicate that stakeholders are using different standards or values in making judgments. This type of disagreement can be a catalyst to clarify values and to negotiate the appropriate basis (or bases) on which the program should be judged.

Recommendations

Recommendations are actions to consider as a result of the evaluation. Forming recommendations requires information beyond just what is necessary to form judgments. For example, knowing that a program is able to increase the services available to battered women doesn't necessarily translate into a recommendation to continue the effort, particularly when there are competing priorities or other effective alternatives. Thus, recommendations about what to do with a given intervention go beyond judgments about a specific program's effectiveness.

If recommendations aren't supported by enough evidence, or if they aren't in keeping with stakeholders' values, they can really undermine an evaluation's credibility. By contrast, an evaluation can be strengthened by recommendations that anticipate and react to what users will want to know.

Three things might increase the chances that recommendations will be relevant and well-received:

• Sharing draft recommendations
• Soliciting reactions from multiple stakeholders
• Presenting options instead of directive advice

Justifying conclusions in an evaluation is a process that involves different possible steps. For instance, conclusions could be strengthened by searching for alternative explanations from the ones you have chosen, and then showing why they are unsupported by the evidence. When there are different but equally well supported conclusions, each could be presented with a summary of their strengths and weaknesses. Techniques to analyze, synthesize, and interpret findings might be agreed upon before data collection begins.

Ensure Use and Share Lessons Learned

It is naive to assume that lessons learned in an evaluation will necessarily be used in decision making and subsequent action. Deliberate effort on the part of evaluators is needed to ensure that the evaluation findings will be used appropriately. Preparing for their use involves strategic thinking and continued vigilance in looking for opportunities to communicate and influence. Both of these should begin in the earliest stages of the process and continue throughout the evaluation.

The elements of key importance to be sure that the recommendations from an evaluation are used are:

Design refers to how the evaluation's questions, methods, and overall processes are constructed. As discussed in the third step of this framework (focusing the evaluation design), the evaluation should be organized from the start to achieve specific agreed-upon uses. Having a clear purpose that is focused on the use of what is learned helps those who will carry out the evaluation to know who will do what with the findings. Furthermore, the process of creating a clear design will highlight ways that stakeholders, through their many contributions, can improve the evaluation and facilitate the use of the results.

Preparation

Preparation refers to the steps taken to get ready for the future uses of the evaluation findings. The ability to translate new knowledge into appropriate action is a skill that can be strengthened through practice. In fact, building this skill can itself be a useful benefit of the evaluation. It is possible to prepare stakeholders for future use of the results by discussing how potential findings might affect decision making.

For example, primary intended users and other stakeholders could be given a set of hypothetical results and asked what decisions or actions they would make on the basis of this new knowledge. If they indicate that the evidence presented is incomplete or irrelevant and that no action would be taken, then this is an early warning sign that the planned evaluation should be modified. Preparing for use also gives stakeholders more time to explore both positive and negative implications of potential results and to identify different options for program improvement.

Feedback is the communication that occurs among everyone involved in the evaluation. Giving and receiving feedback creates an atmosphere of trust among stakeholders; it keeps an evaluation on track by keeping everyone informed about how the evaluation is proceeding. Primary intended users and other stakeholders have a right to comment on evaluation decisions. From a standpoint of ensuring use, stakeholder feedback is a necessary part of every step in the evaluation. Obtaining valuable feedback can be encouraged by holding discussions during each step of the evaluation and routinely sharing interim findings, provisional interpretations, and draft reports.

Follow-up refers to the support that many users need during the evaluation and after they receive evaluation findings. Because of the amount of effort required, reaching justified conclusions in an evaluation can seem like an end in itself. It is not . Active follow-up may be necessary to remind users of the intended uses of what has been learned. Follow-up may also be required to stop lessons learned from becoming lost or ignored in the process of making complex or political decisions. To guard against such oversight, it may be helpful to have someone involved in the evaluation serve as an advocate for the evaluation's findings during the decision -making phase.

Facilitating the use of evaluation findings also carries with it the responsibility to prevent misuse. Evaluation results are always bounded by the context in which the evaluation was conducted. Some stakeholders, however, may be tempted to take results out of context or to use them for different purposes than what they were developed for. For instance, over-generalizing the results from a single case study to make decisions that affect all sites in a national program is an example of misuse of a case study evaluation.

Similarly, program opponents may misuse results by overemphasizing negative findings without giving proper credit for what has worked. Active follow-up can help to prevent these and other forms of misuse by ensuring that evidence is only applied to the questions that were the central focus of the evaluation.

Dissemination

Dissemination is the process of communicating the procedures or the lessons learned from an evaluation to relevant audiences in a timely, unbiased, and consistent fashion. Like other elements of the evaluation, the reporting strategy should be discussed in advance with intended users and other stakeholders. Planning effective communications also requires considering the timing, style, tone, message source, vehicle, and format of information products. Regardless of how communications are constructed, the goal for dissemination is to achieve full disclosure and impartial reporting.

Along with the uses for evaluation findings, there are also uses that flow from the very process of evaluating. These "process uses" should be encouraged. The people who take part in an evaluation can experience profound changes in beliefs and behavior. For instance, an evaluation challenges staff members to act differently in what they are doing, and to question assumptions that connect program activities with intended effects.

Evaluation also prompts staff to clarify their understanding of the goals of the program. This greater clarity, in turn, helps staff members to better function as a team focused on a common end. In short, immersion in the logic, reasoning, and values of evaluation can have very positive effects, such as basing decisions on systematic judgments instead of on unfounded assumptions.

Additional process uses for evaluation include:

• By defining indicators, what really matters to stakeholders becomes clear
• It helps make outcomes matter by changing the reinforcements connected with achieving positive results. For example, a funder might offer "bonus grants" or "outcome dividends" to a program that has shown a significant amount of community change and improvement.

Standards for "good" evaluation

There are standards to assess whether all of the parts of an evaluation are well -designed and working to their greatest potential. The Joint Committee on Educational Evaluation developed "The Program Evaluation Standards" for this purpose. These standards, designed to assess evaluations of educational programs, are also relevant for programs and interventions related to community health and development.

The program evaluation standards make it practical to conduct sound and fair evaluations. They offer well-supported principles to follow when faced with having to make tradeoffs or compromises. Attending to the standards can guard against an imbalanced evaluation, such as one that is accurate and feasible, but isn't very useful or sensitive to the context. Another example of an imbalanced evaluation is one that would be genuinely useful, but is impossible to carry out.

The following standards can be applied while developing an evaluation design and throughout the course of its implementation. Remember, the standards are written as guiding principles, not as rigid rules to be followed in all situations.

The 30 more specific standards are grouped into four categories:

The utility standards are:

• Stakeholder Identification : People who are involved in (or will be affected by) the evaluation should be identified, so that their needs can be addressed.
• Evaluator Credibility : The people conducting the evaluation should be both trustworthy and competent, so that the evaluation will be generally accepted as credible or believable.
• Information Scope and Selection : Information collected should address pertinent questions about the program, and it should be responsive to the needs and interests of clients and other specified stakeholders.
• Values Identification: The perspectives, procedures, and rationale used to interpret the findings should be carefully described, so that the bases for judgments about merit and value are clear.
• Report Clarity: Evaluation reports should clearly describe the program being evaluated, including its context, and the purposes, procedures, and findings of the evaluation. This will help ensure that essential information is provided and easily understood.
• Report Timeliness and Dissemination: Significant midcourse findings and evaluation reports should be shared with intended users so that they can be used in a timely fashion.
• Evaluation Impact: Evaluations should be planned, conducted, and reported in ways that encourage follow-through by stakeholders, so that the evaluation will be used.

Feasibility Standards

The feasibility standards are to ensure that the evaluation makes sense - that the steps that are planned are both viable and pragmatic.

The feasibility standards are:

• Practical Procedures: The evaluation procedures should be practical, to keep disruption of everyday activities to a minimum while needed information is obtained.
• Political Viability : The evaluation should be planned and conducted with anticipation of the different positions or interests of various groups. This should help in obtaining their cooperation so that possible attempts by these groups to curtail evaluation operations or to misuse the results can be avoided or counteracted.
• Cost Effectiveness: The evaluation should be efficient and produce enough valuable information that the resources used can be justified.

Propriety Standards

The propriety standards ensure that the evaluation is an ethical one, conducted with regard for the rights and interests of those involved. The eight propriety standards follow.

• Service Orientation : Evaluations should be designed to help organizations effectively serve the needs of all of the targeted participants.
• Formal Agreements : The responsibilities in an evaluation (what is to be done, how, by whom, when) should be agreed to in writing, so that those involved are obligated to follow all conditions of the agreement, or to formally renegotiate it.
• Rights of Human Subjects : Evaluation should be designed and conducted to respect and protect the rights and welfare of human subjects, that is, all participants in the study.
• Human Interactions : Evaluators should respect basic human dignity and worth when working with other people in an evaluation, so that participants don't feel threatened or harmed.
• Complete and Fair Assessment : The evaluation should be complete and fair in its examination, recording both strengths and weaknesses of the program being evaluated. This allows strengths to be built upon and problem areas addressed.
• Disclosure of Findings : The people working on the evaluation should ensure that all of the evaluation findings, along with the limitations of the evaluation, are accessible to everyone affected by the evaluation, and any others with expressed legal rights to receive the results.
• Conflict of Interest: Conflict of interest should be dealt with openly and honestly, so that it does not compromise the evaluation processes and results.
• Fiscal Responsibility : The evaluator's use of resources should reflect sound accountability procedures and otherwise be prudent and ethically responsible, so that expenditures are accounted for and appropriate.

Accuracy Standards

The accuracy standards ensure that the evaluation findings are considered correct.

There are 12 accuracy standards:

• Program Documentation: The program should be described and documented clearly and accurately, so that what is being evaluated is clearly identified.
• Context Analysis: The context in which the program exists should be thoroughly examined so that likely influences on the program can be identified.
• Described Purposes and Procedures: The purposes and procedures of the evaluation should be monitored and described in enough detail that they can be identified and assessed.
• Defensible Information Sources: The sources of information used in a program evaluation should be described in enough detail that the adequacy of the information can be assessed.
• Valid Information: The information gathering procedures should be chosen or developed and then implemented in such a way that they will assure that the interpretation arrived at is valid.
• Reliable Information : The information gathering procedures should be chosen or developed and then implemented so that they will assure that the information obtained is sufficiently reliable.
• Systematic Information: The information from an evaluation should be systematically reviewed and any errors found should be corrected.
• Analysis of Quantitative Information: Quantitative information - data from observations or surveys - in an evaluation should be appropriately and systematically analyzed so that evaluation questions are effectively answered.
• Analysis of Qualitative Information: Qualitative information - descriptive information from interviews and other sources - in an evaluation should be appropriately and systematically analyzed so that evaluation questions are effectively answered.
• Justified Conclusions: The conclusions reached in an evaluation should be explicitly justified, so that stakeholders can understand their worth.
• Impartial Reporting: Reporting procedures should guard against the distortion caused by personal feelings and biases of people involved in the evaluation, so that evaluation reports fairly reflect the evaluation findings.
• Metaevaluation: The evaluation itself should be evaluated against these and other pertinent standards, so that it is appropriately guided and, on completion, stakeholders can closely examine its strengths and weaknesses.

Applying the framework: Conducting optimal evaluations

There is an ever-increasing agreement on the worth of evaluation; in fact, doing so is often required by funders and other constituents. So, community health and development professionals can no longer question whether or not to evaluate their programs. Instead, the appropriate questions are:

• What is the best way to evaluate?
• What are we learning from the evaluation?
• How will we use what we learn to become more effective?

The framework for program evaluation helps answer these questions by guiding users to select evaluation strategies that are useful, feasible, proper, and accurate.

To use this framework requires quite a bit of skill in program evaluation. In most cases there are multiple stakeholders to consider, the political context may be divisive, steps don't always follow a logical order, and limited resources may make it difficult to take a preferred course of action. An evaluator's challenge is to devise an optimal strategy, given the conditions she is working under. An optimal strategy is one that accomplishes each step in the framework in a way that takes into account the program context and is able to meet or exceed the relevant standards.

This framework also makes it possible to respond to common concerns about program evaluation. For instance, many evaluations are not undertaken because they are seen as being too expensive. The cost of an evaluation, however, is relative; it depends upon the question being asked and the level of certainty desired for the answer. A simple, low-cost evaluation can deliver information valuable for understanding and improvement.

Rather than discounting evaluations as a time-consuming sideline, the framework encourages evaluations that are timed strategically to provide necessary feedback. This makes it possible to make evaluation closely linked with everyday practices.

Another concern centers on the perceived technical demands of designing and conducting an evaluation. However, the practical approach endorsed by this framework focuses on questions that can improve the program.

Finally, the prospect of evaluation troubles many staff members because they perceive evaluation methods as punishing ("They just want to show what we're doing wrong."), exclusionary ("Why aren't we part of it? We're the ones who know what's going on."), and adversarial ("It's us against them.") The framework instead encourages an evaluation approach that is designed to be helpful and engages all interested stakeholders in a process that welcomes their participation.

Evaluation is a powerful strategy for distinguishing programs and interventions that make a difference from those that don't. It is a driving force for developing and adapting sound strategies, improving existing programs, and demonstrating the results of investments in time and other resources. It also helps determine if what is being done is worth the cost.

This recommended framework for program evaluation is both a synthesis of existing best practices and a set of standards for further improvement. It supports a practical approach to evaluation based on steps and standards that can be applied in almost any setting. Because the framework is purposefully general, it provides a stable guide to design and conduct a wide range of evaluation efforts in a variety of specific program areas. The framework can be used as a template to create useful evaluation plans to contribute to understanding and improvement. The Magenta Book - Guidance for Evaluation  provides additional information on requirements for good evaluation, and some straightforward steps to make a good evaluation of an intervention more feasible, read The Magenta Book - Guidance for Evaluation.

Online Resources

Are You Ready to Evaluate your Coalition? prompts 15 questions to help the group decide whether your coalition is ready to evaluate itself and its work.

The  American Evaluation Association Guiding Principles for Evaluators  helps guide evaluators in their professional practice.

CDC Evaluation Resources  provides a list of resources for evaluation, as well as links to professional associations and journals.

Chapter 11: Community Interventions in the "Introduction to Community Psychology" explains professionally-led versus grassroots interventions, what it means for a community intervention to be effective, why a community needs to be ready for an intervention, and the steps to implementing community interventions.

The  Comprehensive Cancer Control Branch Program Evaluation Toolkit  is designed to help grantees plan and implement evaluations of their NCCCP-funded programs, this toolkit provides general guidance on evaluation principles and techniques, as well as practical templates and tools.

Developing an Effective Evaluation Plan  is a workbook provided by the CDC. In addition to information on designing an evaluation plan, this book also provides worksheets as a step-by-step guide.

EvaluACTION , from the CDC, is designed for people interested in learning about program evaluation and how to apply it to their work. Evaluation is a process, one dependent on what you’re currently doing and on the direction in which you’d like go. In addition to providing helpful information, the site also features an interactive Evaluation Plan & Logic Model Builder, so you can create customized tools for your organization to use.

Evaluating Your Community-Based Program  is a handbook designed by the American Academy of Pediatrics covering a variety of topics related to evaluation.

GAO Designing Evaluations  is a handbook provided by the U.S. Government Accountability Office with copious information regarding program evaluations.

The CDC's  Introduction to Program Evaluation for Publilc Health Programs: A Self-Study Guide  is a "how-to" guide for planning and implementing evaluation activities. The manual, based on CDC’s Framework for Program Evaluation in Public Health, is intended to assist with planning, designing, implementing and using comprehensive evaluations in a practical way.

McCormick Foundation Evaluation Guide  is a guide to planning an organization’s evaluation, with several chapters dedicated to gathering information and using it to improve the organization.

A Participatory Model for Evaluating Social Programs from the James Irvine Foundation.

Practical Evaluation for Public Managers  is a guide to evaluation written by the U.S. Department of Health and Human Services.

Penn State Program Evaluation  offers information on collecting different forms of data and how to measure different community markers.

Program Evaluaton  information page from Implementation Matters.

The Program Manager's Guide to Evaluation  is a handbook provided by the Administration for Children and Families with detailed answers to nine big questions regarding program evaluation.

Program Planning and Evaluation  is a website created by the University of Arizona. It provides links to information on several topics including methods, funding, types of evaluation, and reporting impacts.

User-Friendly Handbook for Program Evaluation  is a guide to evaluations provided by the National Science Foundation.  This guide includes practical information on quantitative and qualitative methodologies in evaluations.

W.K. Kellogg Foundation Evaluation Handbook  provides a framework for thinking about evaluation as a relevant and useful program tool. It was originally written for program directors with direct responsibility for the ongoing evaluation of the W.K. Kellogg Foundation.

Print Resources

This Community Tool Box section is an edited version of:

CDC Evaluation Working Group. (1999). (Draft). Recommended framework for program evaluation in public health practice . Atlanta, GA: Author.

The article cites the following references:

Adler. M., &  Ziglio, E. (1996). Gazing into the oracle: the delphi method and its application to social policy and community health and development. London: Jessica Kingsley Publishers.

Barrett, F.   Program Evaluation: A Step-by-Step Guide.  Sunnycrest Press, 2013. This practical manual includes helpful tips to develop evaluations, tables illustrating evaluation approaches, evaluation planning and reporting templates, and resources if you want more information.

Basch, C., Silepcevich, E., Gold, R., Duncan, D., & Kolbe, L. (1985).   Avoiding type III errors in health education program evaluation: a case study . Health Education Quarterly. 12(4):315-31.

Bickman L, & Rog, D. (1998). Handbook of applied social research methods. Thousand Oaks, CA: Sage Publications.

Boruch, R.  (1998).  Randomized controlled experiments for evaluation and planning. In Handbook of applied social research methods, edited by Bickman L., & Rog. D. Thousand Oaks, CA: Sage Publications: 161-92.

Centers for Disease Control and Prevention DoHAP. Evaluating CDC HIV prevention programs: guidance and data system . Atlanta, GA: Centers for Disease Control and Prevention, Division of HIV/AIDS Prevention, 1999.

Centers for Disease Control and Prevention. Guidelines for evaluating surveillance systems. Morbidity and Mortality Weekly Report 1988;37(S-5):1-18.

Centers for Disease Control and Prevention. Handbook for evaluating HIV education . Atlanta, GA: Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Division of Adolescent and School Health, 1995.

Cook, T., & Campbell, D. (1979). Quasi-experimentation . Chicago, IL: Rand McNally.

Cook, T.,& Reichardt, C. (1979).  Qualitative and quantitative methods in evaluation research . Beverly Hills, CA: Sage Publications.

Cousins, J.,& Whitmore, E. (1998).   Framing participatory evaluation. In Understanding and practicing participatory evaluation , vol. 80, edited by E Whitmore. San Francisco, CA: Jossey-Bass: 5-24.

Chen, H. (1990).  Theory driven evaluations . Newbury Park, CA: Sage Publications.

de Vries, H., Weijts, W., Dijkstra, M., & Kok, G. (1992).  The utilization of qualitative and quantitative data for health education program planning, implementation, and evaluation: a spiral approach . Health Education Quarterly.1992; 19(1):101-15.

Dyal, W. (1995).  Ten organizational practices of community health and development: a historical perspective . American Journal of Preventive Medicine;11(6):6-8.

Eddy, D. (1998). Performance measurement: problems and solutions . Health Affairs;17 (4):7-25.Harvard Family Research Project. Performance measurement. In The Evaluation Exchange, vol. 4, 1998, pp. 1-15.

Eoyang,G., & Berkas, T. (1996).  Evaluation in a complex adaptive system . Edited by (we don´t have the names), (1999): Taylor-Powell E, Steele S, Douglah M. Planning a program evaluation. Madison, Wisconsin: University of Wisconsin Cooperative Extension.

Fawcett, S.B., Paine-Andrews, A., Fancisco, V.T., Schultz, J.A., Richter, K.P, Berkley-Patton, J., Fisher, J., Lewis, R.K., Lopez, C.M., Russos, S., Williams, E.L., Harris, K.J., & Evensen, P. (2001). Evaluating community initiatives for health and development. In I. Rootman, D. McQueen, et al. (Eds.),  Evaluating health promotion approaches . (pp. 241-277). Copenhagen, Denmark: World Health Organization - Europe.

Fawcett , S., Sterling, T., Paine-, A., Harris, K., Francisco, V. et al. (1996).  Evaluating community efforts to prevent cardiovascular diseases . Atlanta, GA: Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion.

Fetterman, D.,, Kaftarian, S., & Wandersman, A. (1996).  Empowerment evaluation: knowledge and tools for self-assessment and accountability . Thousand Oaks, CA: Sage Publications.

Frechtling, J.,& Sharp, L. (1997).  User-friendly handbook for mixed method evaluations . Washington, DC: National Science Foundation.

Goodman, R., Speers, M., McLeroy, K., Fawcett, S., Kegler M., et al. (1998).  Identifying and defining the dimensions of community capacity to provide a basis for measurement . Health Education and Behavior;25(3):258-78.

Greene, J.  (1994). Qualitative program evaluation: practice and promise . In Handbook of Qualitative Research, edited by NK Denzin and YS Lincoln. Thousand Oaks, CA: Sage Publications.

Haddix, A., Teutsch. S., Shaffer. P., & Dunet. D. (1996). Prevention effectiveness: a guide to decision analysis and economic evaluation . New York, NY: Oxford University Press.

Hennessy, M.  Evaluation. In Statistics in Community health and development , edited by Stroup. D.,& Teutsch. S. New York, NY: Oxford University Press, 1998: 193-219

Henry, G. (1998). Graphing data. In Handbook of applied social research methods , edited by Bickman. L., & Rog.  D.. Thousand Oaks, CA: Sage Publications: 527-56.

Henry, G. (1998).  Practical sampling. In Handbook of applied social research methods , edited by  Bickman. L., & Rog. D.. Thousand Oaks, CA: Sage Publications: 101-26.

Institute of Medicine. Improving health in the community: a role for performance monitoring . Washington, DC: National Academy Press, 1997.

Joint Committee on Educational Evaluation, James R. Sanders (Chair). The program evaluation standards: how to assess evaluations of educational programs . Thousand Oaks, CA: Sage Publications, 1994.

Kaplan,  R., & Norton, D.  The balanced scorecard: measures that drive performance . Harvard Business Review 1992;Jan-Feb71-9.

Kar, S. (1989). Health promotion indicators and actions . New York, NY: Springer Publications.

Knauft, E. (1993).   What independent sector learned from an evaluation of its own hard-to -measure programs . In A vision of evaluation, edited by ST Gray. Washington, DC: Independent Sector.

Koplan, J. (1999)  CDC sets millennium priorities . US Medicine 4-7.

Lipsy, M. (1998).  Design sensitivity: statistical power for applied experimental research . In Handbook of applied social research methods, edited by Bickman, L., & Rog, D. Thousand Oaks, CA: Sage Publications. 39-68.

Lipsey, M. (1993). Theory as method: small theories of treatments . New Directions for Program Evaluation;(57):5-38.

Lipsey, M. (1997).  What can you build with thousands of bricks? Musings on the cumulation of knowledge in program evaluation . New Directions for Evaluation; (76): 7-23.

Love, A.  (1991).  Internal evaluation: building organizations from within . Newbury Park, CA: Sage Publications.

Miles, M., & Huberman, A. (1994).  Qualitative data analysis: a sourcebook of methods . Thousand Oaks, CA: Sage Publications, Inc.

National Quality Program. (1999).  National Quality Program , vol. 1999. National Institute of Standards and Technology.

National Quality Program . Baldridge index outperforms S&P 500 for fifth year, vol. 1999.

National Quality Program , 1999.

National Quality Program. Health care criteria for performance excellence , vol. 1999. National Quality Program, 1998.

Newcomer, K.  Using statistics appropriately. In Handbook of Practical Program Evaluation, edited by Wholey,J.,  Hatry, H., & Newcomer. K. San Francisco, CA: Jossey-Bass, 1994: 389-416.

Patton, M. (1990).  Qualitative evaluation and research methods . Newbury Park, CA: Sage Publications.

Patton, M (1997).  Toward distinguishing empowerment evaluation and placing it in a larger context . Evaluation Practice;18(2):147-63.

Patton, M. (1997).  Utilization-focused evaluation . Thousand Oaks, CA: Sage Publications.

Perrin, B. Effective use and misuse of performance measurement . American Journal of Evaluation 1998;19(3):367-79.

Perrin, E, Koshel J. (1997).  Assessment of performance measures for community health and development, substance abuse, and mental health . Washington, DC: National Academy Press.

Phillips, J. (1997).  Handbook of training evaluation and measurement methods . Houston, TX: Gulf Publishing Company.

Poreteous, N., Sheldrick B., & Stewart P. (1997).  Program evaluation tool kit: a blueprint for community health and development management . Ottawa, Canada: Community health and development Research, Education, and Development Program, Ottawa-Carleton Health Department.

Posavac, E., & Carey R. (1980).  Program evaluation: methods and case studies . Prentice-Hall, Englewood Cliffs, NJ.

Preskill, H. & Torres R. (1998).  Evaluative inquiry for learning in organizations . Thousand Oaks, CA: Sage Publications.

Public Health Functions Project. (1996). The public health workforce: an agenda for the 21st century . Washington, DC: U.S. Department of Health and Human Services, Community health and development Service.

Public Health Training Network. (1998).  Practical evaluation of public health programs . CDC, Atlanta, GA.

Reichardt, C., & Mark M. (1998).  Quasi-experimentation . In Handbook of applied social research methods, edited by L Bickman and DJ Rog. Thousand Oaks, CA: Sage Publications, 193-228.

Rossi, P., & Freeman H.  (1993).  Evaluation: a systematic approach . Newbury Park, CA: Sage Publications.

Rush, B., & Ogbourne A. (1995).  Program logic models: expanding their role and structure for program planning and evaluation . Canadian Journal of Program Evaluation;695 -106.

Sanders, J. (1993).  Uses of evaluation as a means toward organizational effectiveness. In A vision of evaluation , edited by ST Gray. Washington, DC: Independent Sector.

Schorr, L. (1997).   Common purpose: strengthening families and neighborhoods to rebuild America . New York, NY: Anchor Books, Doubleday.

Scriven, M. (1998) . A minimalist theory of evaluation: the least theory that practice requires . American Journal of Evaluation.

Shadish, W., Cook, T., Leviton, L. (1991).  Foundations of program evaluation . Newbury Park, CA: Sage Publications.

Shadish, W. (1998).   Evaluation theory is who we are. American Journal of Evaluation:19(1):1-19.

Shulha, L., & Cousins, J. (1997).  Evaluation use: theory, research, and practice since 1986 . Evaluation Practice.18(3):195-208

Sieber, J. (1998).   Planning ethically responsible research . In Handbook of applied social research methods, edited by L Bickman and DJ Rog. Thousand Oaks, CA: Sage Publications: 127-56.

Steckler, A., McLeroy, K., Goodman, R., Bird, S., McCormick, L. (1992).  Toward integrating qualitative and quantitative methods: an introduction . Health Education Quarterly;191-8.

Taylor-Powell, E., Rossing, B., Geran, J. (1998). Evaluating collaboratives: reaching the potential. Madison, Wisconsin: University of Wisconsin Cooperative Extension.

Teutsch, S.  A framework for assessing the effectiveness of disease and injury prevention . Morbidity and Mortality Weekly Report: Recommendations and Reports Series 1992;41 (RR-3 (March 27, 1992):1-13.

Torres, R., Preskill, H., Piontek, M., (1996).   Evaluation strategies for communicating and reporting: enhancing learning in organizations . Thousand Oaks, CA: Sage Publications.

Trochim, W. (1999).  Research methods knowledge base , vol.

United Way of America. Measuring program outcomes: a practical approach . Alexandria, VA: United Way of America, 1996.

U.S. General Accounting Office. Case study evaluations . GAO/PEMD-91-10.1.9. Washington, DC: U.S. General Accounting Office, 1990.

U.S. General Accounting Office. Designing evaluations . GAO/PEMD-10.1.4. Washington, DC: U.S. General Accounting Office, 1991.

U.S. General Accounting Office. Managing for results: measuring program results that are under limited federal control . GAO/GGD-99-16. Washington, DC: 1998.

U.S. General Accounting Office. Prospective evaluation methods: the prosepctive evaluation synthesis . GAO/PEMD-10.1.10. Washington, DC: U.S. General Accounting Office, 1990.

U.S. General Accounting Office. The evaluation synthesis . Washington, DC: U.S. General Accounting Office, 1992.

U.S. General Accounting Office. Using statistical sampling . Washington, DC: U.S. General Accounting Office, 1992.

Wandersman, A., Morrissey, E., Davino, K., Seybolt, D., Crusto, C., et al. Comprehensive quality programming and accountability: eight essential strategies for implementing successful prevention programs . Journal of Primary Prevention 1998;19(1):3-30.

Weiss, C. (1995). Nothing as practical as a good theory: exploring theory-based evaluation for comprehensive community initiatives for families and children . In New Approaches to Evaluating Community Initiatives, edited by Connell, J. Kubisch, A. Schorr, L.  & Weiss, C.  New York, NY, NY: Aspin Institute.

Weiss, C. (1998).  Have we learned anything new about the use of evaluation? American Journal of Evaluation;19(1):21-33.

Weiss, C. (1997).  How can theory-based evaluation make greater headway? Evaluation Review 1997;21(4):501-24.

W.K. Kellogg Foundation. (1998). The W.K. Foundation Evaluation Handbook . Battle Creek, MI: W.K. Kellogg Foundation.

Wong-Reiger, D.,& David, L. (1995).  Using program logic models to plan and evaluate education and prevention programs. In Evaluation Methods Sourcebook II, edited by Love. A.J. Ottawa, Ontario: Canadian Evaluation Society.

Wholey, S., Hatry, P., & Newcomer, E. .  Handbook of Practical Program Evaluation.  Jossey-Bass, 2010. This book serves as a comprehensive guide to the evaluation process and its practical applications for sponsors, program managers, and evaluators.

Yarbrough,  B., Lyn, M., Shulha, H., Rodney K., & Caruthers, A. (2011).  The Program Evaluation Standards: A Guide for Evalualtors and Evaluation Users Third Edition . Sage Publications.

Yin, R. (1988).  Case study research: design and methods . Newbury Park, CA: Sage Publications.

• Business Essentials
• Leadership & Management
• Credential of Leadership, Impact, and Management in Business (CLIMB)
• Entrepreneurship & Innovation
• Digital Transformation
• Finance & Accounting
• Business in Society
• For Organizations
• Support Portal
• Media Coverage
• Founding Donors
• Leadership Team

• Harvard Business School →
• HBS Online →
• Business Insights →

Business Insights

Harvard Business School Online's Business Insights Blog provides the career insights you need to achieve your goals and gain confidence in your business skills.

• Career Development
• Communication
• Decision-Making
• Earning Your MBA
• Negotiation
• News & Events
• Productivity
• Staff Spotlight
• Student Profiles
• Work-Life Balance
• AI Essentials for Business
• Alternative Investments
• Business Analytics
• Business Strategy
• Business and Climate Change
• Design Thinking and Innovation
• Digital Marketing Strategy
• Disruptive Strategy
• Economics for Managers
• Entrepreneurship Essentials
• Financial Accounting
• Global Business
• Launching Tech Ventures
• Leadership Principles
• Leadership, Ethics, and Corporate Accountability
• Leading Change and Organizational Renewal
• Leading with Finance
• Management Essentials
• Negotiation Mastery
• Organizational Leadership
• Power and Influence for Positive Impact
• Strategy Execution
• Sustainable Business Strategy
• Sustainable Investing
• Winning with Digital Platforms

5 Benefits of Learning Through the Case Study Method

• 28 Nov 2023

While several factors make HBS Online unique —including a global Community and real-world outcomes —active learning through the case study method rises to the top.

In a 2023 City Square Associates survey, 74 percent of HBS Online learners who also took a course from another provider said HBS Online’s case method and real-world examples were better by comparison.

Here’s a primer on the case method, five benefits you could gain, and how to experience it for yourself.

Access your free e-book today.

What Is the Harvard Business School Case Study Method?

The case study method , or case method , is a learning technique in which you’re presented with a real-world business challenge and asked how you’d solve it. After working through it yourself and with peers, you’re told how the scenario played out.

HBS pioneered the case method in 1922. Shortly before, in 1921, the first case was written.

“How do you go into an ambiguous situation and get to the bottom of it?” says HBS Professor Jan Rivkin, former senior associate dean and chair of HBS's master of business administration (MBA) program, in a video about the case method . “That skill—the skill of figuring out a course of inquiry to choose a course of action—that skill is as relevant today as it was in 1921.”

Originally developed for the in-person MBA classroom, HBS Online adapted the case method into an engaging, interactive online learning experience in 2014.

In HBS Online courses , you learn about each case from the business professional who experienced it. After reviewing their videos, you’re prompted to take their perspective and explain how you’d handle their situation.

You then get to read peers’ responses, “star” them, and comment to further the discussion. Afterward, you learn how the professional handled it and their key takeaways.

HBS Online’s adaptation of the case method incorporates the famed HBS “cold call,” in which you’re called on at random to make a decision without time to prepare.

“Learning came to life!” said Sheneka Balogun , chief administration officer and chief of staff at LeMoyne-Owen College, of her experience taking the Credential of Readiness (CORe) program . “The videos from the professors, the interactive cold calls where you were randomly selected to participate, and the case studies that enhanced and often captured the essence of objectives and learning goals were all embedded in each module. This made learning fun, engaging, and student-friendly.”

If you’re considering taking a course that leverages the case study method, here are five benefits you could experience.

5 Benefits of Learning Through Case Studies

1. take new perspectives.

The case method prompts you to consider a scenario from another person’s perspective. To work through the situation and come up with a solution, you must consider their circumstances, limitations, risk tolerance, stakeholders, resources, and potential consequences to assess how to respond.

Taking on new perspectives not only can help you navigate your own challenges but also others’. Putting yourself in someone else’s situation to understand their motivations and needs can go a long way when collaborating with stakeholders.

2. Hone Your Decision-Making Skills

Another skill you can build is the ability to make decisions effectively . The case study method forces you to use limited information to decide how to handle a problem—just like in the real world.

Throughout your career, you’ll need to make difficult decisions with incomplete or imperfect information—and sometimes, you won’t feel qualified to do so. Learning through the case method allows you to practice this skill in a low-stakes environment. When facing a real challenge, you’ll be better prepared to think quickly, collaborate with others, and present and defend your solution.

3. Become More Open-Minded

As you collaborate with peers on responses, it becomes clear that not everyone solves problems the same way. Exposing yourself to various approaches and perspectives can help you become a more open-minded professional.

When you’re part of a diverse group of learners from around the world, your experiences, cultures, and backgrounds contribute to a range of opinions on each case.

On the HBS Online course platform, you’re prompted to view and comment on others’ responses, and discussion is encouraged. This practice of considering others’ perspectives can make you more receptive in your career.

“You’d be surprised at how much you can learn from your peers,” said Ratnaditya Jonnalagadda , a software engineer who took CORe.

In addition to interacting with peers in the course platform, Jonnalagadda was part of the HBS Online Community , where he networked with other professionals and continued discussions sparked by course content.

“You get to understand your peers better, and students share examples of businesses implementing a concept from a module you just learned,” Jonnalagadda said. “It’s a very good way to cement the concepts in one's mind.”

4. Enhance Your Curiosity

One byproduct of taking on different perspectives is that it enables you to picture yourself in various roles, industries, and business functions.

“Each case offers an opportunity for students to see what resonates with them, what excites them, what bores them, which role they could imagine inhabiting in their careers,” says former HBS Dean Nitin Nohria in the Harvard Business Review . “Cases stimulate curiosity about the range of opportunities in the world and the many ways that students can make a difference as leaders.”

Through the case method, you can “try on” roles you may not have considered and feel more prepared to change or advance your career .

5. Build Your Self-Confidence

Finally, learning through the case study method can build your confidence. Each time you assume a business leader’s perspective, aim to solve a new challenge, and express and defend your opinions and decisions to peers, you prepare to do the same in your career.

According to a 2022 City Square Associates survey , 84 percent of HBS Online learners report feeling more confident making business decisions after taking a course.

“Self-confidence is difficult to teach or coach, but the case study method seems to instill it in people,” Nohria says in the Harvard Business Review . “There may well be other ways of learning these meta-skills, such as the repeated experience gained through practice or guidance from a gifted coach. However, under the direction of a masterful teacher, the case method can engage students and help them develop powerful meta-skills like no other form of teaching.”

How to Experience the Case Study Method

If the case method seems like a good fit for your learning style, experience it for yourself by taking an HBS Online course. Offerings span seven subject areas, including:

• Business essentials
• Leadership and management
• Entrepreneurship and innovation
• Finance and accounting
• Business in society

No matter which course or credential program you choose, you’ll examine case studies from real business professionals, work through their challenges alongside peers, and gain valuable insights to apply to your career.

Are you interested in discovering how HBS Online can help advance your career? Explore our course catalog and download our free guide —complete with interactive workbook sections—to determine if online learning is right for you and which course to take.

About the Author

Case Study Research Method in Psychology

Saul Mcleod, PhD

Editor-in-Chief for Simply Psychology

BSc (Hons) Psychology, MRes, PhD, University of Manchester

Saul Mcleod, PhD., is a qualified psychology teacher with over 18 years of experience in further and higher education. He has been published in peer-reviewed journals, including the Journal of Clinical Psychology.

Learn about our Editorial Process

Olivia Guy-Evans, MSc

Associate Editor for Simply Psychology

BSc (Hons) Psychology, MSc Psychology of Education

Olivia Guy-Evans is a writer and associate editor for Simply Psychology. She has previously worked in healthcare and educational sectors.

On This Page:

Case studies are in-depth investigations of a person, group, event, or community. Typically, data is gathered from various sources using several methods (e.g., observations & interviews).

The case study research method originated in clinical medicine (the case history, i.e., the patient’s personal history). In psychology, case studies are often confined to the study of a particular individual.

The information is mainly biographical and relates to events in the individual’s past (i.e., retrospective), as well as to significant events that are currently occurring in his or her everyday life.

The case study is not a research method, but researchers select methods of data collection and analysis that will generate material suitable for case studies.

Freud (1909a, 1909b) conducted very detailed investigations into the private lives of his patients in an attempt to both understand and help them overcome their illnesses.

This makes it clear that the case study is a method that should only be used by a psychologist, therapist, or psychiatrist, i.e., someone with a professional qualification.

There is an ethical issue of competence. Only someone qualified to diagnose and treat a person can conduct a formal case study relating to atypical (i.e., abnormal) behavior or atypical development.

 Famous Case Studies

• Anna O – One of the most famous case studies, documenting psychoanalyst Josef Breuer’s treatment of “Anna O” (real name Bertha Pappenheim) for hysteria in the late 1800s using early psychoanalytic theory.
• Little Hans – A child psychoanalysis case study published by Sigmund Freud in 1909 analyzing his five-year-old patient Herbert Graf’s house phobia as related to the Oedipus complex.
• Bruce/Brenda – Gender identity case of the boy (Bruce) whose botched circumcision led psychologist John Money to advise gender reassignment and raise him as a girl (Brenda) in the 1960s.
• Genie Wiley – Linguistics/psychological development case of the victim of extreme isolation abuse who was studied in 1970s California for effects of early language deprivation on acquiring speech later in life.
• Phineas Gage – One of the most famous neuropsychology case studies analyzes personality changes in railroad worker Phineas Gage after an 1848 brain injury involving a tamping iron piercing his skull.

Clinical Case Studies

• Studying the effectiveness of psychotherapy approaches with an individual patient
• Assessing and treating mental illnesses like depression, anxiety disorders, PTSD
• Neuropsychological cases investigating brain injuries or disorders

Child Psychology Case Studies

• Studying psychological development from birth through adolescence
• Cases of learning disabilities, autism spectrum disorders, ADHD
• Effects of trauma, abuse, deprivation on development

Types of Case Studies

• Explanatory case studies : Used to explore causation in order to find underlying principles. Helpful for doing qualitative analysis to explain presumed causal links.
• Exploratory case studies : Used to explore situations where an intervention being evaluated has no clear set of outcomes. It helps define questions and hypotheses for future research.
• Descriptive case studies : Describe an intervention or phenomenon and the real-life context in which it occurred. It is helpful for illustrating certain topics within an evaluation.
• Multiple-case studies : Used to explore differences between cases and replicate findings across cases. Helpful for comparing and contrasting specific cases.
• Intrinsic : Used to gain a better understanding of a particular case. Helpful for capturing the complexity of a single case.
• Collective : Used to explore a general phenomenon using multiple case studies. Helpful for jointly studying a group of cases in order to inquire into the phenomenon.

Where Do You Find Data for a Case Study?

There are several places to find data for a case study. The key is to gather data from multiple sources to get a complete picture of the case and corroborate facts or findings through triangulation of evidence. Most of this information is likely qualitative (i.e., verbal description rather than measurement), but the psychologist might also collect numerical data.

1. Primary sources

• Interviews – Interviewing key people related to the case to get their perspectives and insights. The interview is an extremely effective procedure for obtaining information about an individual, and it may be used to collect comments from the person’s friends, parents, employer, workmates, and others who have a good knowledge of the person, as well as to obtain facts from the person him or herself.
• Observations – Observing behaviors, interactions, processes, etc., related to the case as they unfold in real-time.
• Documents & Records – Reviewing private documents, diaries, public records, correspondence, meeting minutes, etc., relevant to the case.

2. Secondary sources

• News/Media – News coverage of events related to the case study.
• Academic articles – Journal articles, dissertations etc. that discuss the case.
• Government reports – Official data and records related to the case context.
• Books/films – Books, documentaries or films discussing the case.

3. Archival records

Searching historical archives, museum collections and databases to find relevant documents, visual/audio records related to the case history and context.

Public archives like newspapers, organizational records, photographic collections could all include potentially relevant pieces of information to shed light on attitudes, cultural perspectives, common practices and historical contexts related to psychology.

4. Organizational records

Organizational records offer the advantage of often having large datasets collected over time that can reveal or confirm psychological insights.

Of course, privacy and ethical concerns regarding confidential data must be navigated carefully.

However, with proper protocols, organizational records can provide invaluable context and empirical depth to qualitative case studies exploring the intersection of psychology and organizations.

• Organizational/industrial psychology research : Organizational records like employee surveys, turnover/retention data, policies, incident reports etc. may provide insight into topics like job satisfaction, workplace culture and dynamics, leadership issues, employee behaviors etc.
• Clinical psychology : Therapists/hospitals may grant access to anonymized medical records to study aspects like assessments, diagnoses, treatment plans etc. This could shed light on clinical practices.
• School psychology : Studies could utilize anonymized student records like test scores, grades, disciplinary issues, and counseling referrals to study child development, learning barriers, effectiveness of support programs, and more.

How do I Write a Case Study in Psychology?

Follow specified case study guidelines provided by a journal or your psychology tutor. General components of clinical case studies include: background, symptoms, assessments, diagnosis, treatment, and outcomes. Interpreting the information means the researcher decides what to include or leave out. A good case study should always clarify which information is the factual description and which is an inference or the researcher’s opinion.

1. Introduction

• Provide background on the case context and why it is of interest, presenting background information like demographics, relevant history, and presenting problem.
• Compare briefly to similar published cases if applicable. Clearly state the focus/importance of the case.

2. Case Presentation

• Describe the presenting problem in detail, including symptoms, duration,and impact on daily life.
• Include client demographics like age and gender, information about social relationships, and mental health history.
• Describe all physical, emotional, and/or sensory symptoms reported by the client.
• Use patient quotes to describe the initial complaint verbatim. Follow with full-sentence summaries of relevant history details gathered, including key components that led to a working diagnosis.
• Summarize clinical exam results, namely orthopedic/neurological tests, imaging, lab tests, etc. Note actual results rather than subjective conclusions. Provide images if clearly reproducible/anonymized.
• Clearly state the working diagnosis or clinical impression before transitioning to management.

3. Management and Outcome

• Indicate the total duration of care and number of treatments given over what timeframe. Use specific names/descriptions for any therapies/interventions applied.
• Present the results of the intervention,including any quantitative or qualitative data collected.
• For outcomes, utilize visual analog scales for pain, medication usage logs, etc., if possible. Include patient self-reports of improvement/worsening of symptoms. Note the reason for discharge/end of care.

4. Discussion

• Analyze the case, exploring contributing factors, limitations of the study, and connections to existing research.
• Analyze the effectiveness of the intervention,considering factors like participant adherence, limitations of the study, and potential alternative explanations for the results.
• Identify any questions raised in the case analysis and relate insights to established theories and current research if applicable. Avoid definitive claims about physiological explanations.
• Offer clinical implications, and suggest future research directions.

5. Additional Items

• Thank specific assistants for writing support only. No patient acknowledgments.
• References should directly support any key claims or quotes included.
• Use tables/figures/images only if substantially informative. Include permissions and legends/explanatory notes.
• Provides detailed (rich qualitative) information.
• Provides insight for further research.
• Permitting investigation of otherwise impractical (or unethical) situations.

Case studies allow a researcher to investigate a topic in far more detail than might be possible if they were trying to deal with a large number of research participants (nomothetic approach) with the aim of ‘averaging’.

Because of their in-depth, multi-sided approach, case studies often shed light on aspects of human thinking and behavior that would be unethical or impractical to study in other ways.

Research that only looks into the measurable aspects of human behavior is not likely to give us insights into the subjective dimension of experience, which is important to psychoanalytic and humanistic psychologists.

Case studies are often used in exploratory research. They can help us generate new ideas (that might be tested by other methods). They are an important way of illustrating theories and can help show how different aspects of a person’s life are related to each other.

The method is, therefore, important for psychologists who adopt a holistic point of view (i.e., humanistic psychologists ).

Limitations

• Lacking scientific rigor and providing little basis for generalization of results to the wider population.
• Researchers’ own subjective feelings may influence the case study (researcher bias).
• Difficult to replicate.
• Time-consuming and expensive.
• The volume of data, together with the time restrictions in place, impacted the depth of analysis that was possible within the available resources.

Because a case study deals with only one person/event/group, we can never be sure if the case study investigated is representative of the wider body of “similar” instances. This means the conclusions drawn from a particular case may not be transferable to other settings.

Because case studies are based on the analysis of qualitative (i.e., descriptive) data , a lot depends on the psychologist’s interpretation of the information she has acquired.

This means that there is a lot of scope for Anna O , and it could be that the subjective opinions of the psychologist intrude in the assessment of what the data means.

For example, Freud has been criticized for producing case studies in which the information was sometimes distorted to fit particular behavioral theories (e.g., Little Hans ).

This is also true of Money’s interpretation of the Bruce/Brenda case study (Diamond, 1997) when he ignored evidence that went against his theory.

Breuer, J., & Freud, S. (1895).  Studies on hysteria . Standard Edition 2: London.

Curtiss, S. (1981). Genie: The case of a modern wild child .

Diamond, M., & Sigmundson, K. (1997). Sex Reassignment at Birth: Long-term Review and Clinical Implications. Archives of Pediatrics & Adolescent Medicine , 151(3), 298-304

Freud, S. (1909a). Analysis of a phobia of a five year old boy. In The Pelican Freud Library (1977), Vol 8, Case Histories 1, pages 169-306

Freud, S. (1909b). Bemerkungen über einen Fall von Zwangsneurose (Der “Rattenmann”). Jb. psychoanal. psychopathol. Forsch ., I, p. 357-421; GW, VII, p. 379-463; Notes upon a case of obsessional neurosis, SE , 10: 151-318.

Harlow J. M. (1848). Passage of an iron rod through the head.  Boston Medical and Surgical Journal, 39 , 389–393.

Harlow, J. M. (1868).  Recovery from the Passage of an Iron Bar through the Head .  Publications of the Massachusetts Medical Society. 2  (3), 327-347.

Money, J., & Ehrhardt, A. A. (1972).  Man & Woman, Boy & Girl : The Differentiation and Dimorphism of Gender Identity from Conception to Maturity. Baltimore, Maryland: Johns Hopkins University Press.

Money, J., & Tucker, P. (1975). Sexual signatures: On being a man or a woman.

Further Information

• Case Study Approach
• Case Study Method
• Enhancing the Quality of Case Studies in Health Services Research
• “We do things together” A case study of “couplehood” in dementia
• Using mixed methods for evaluating an integrative approach to cancer care: a case study

Related Articles

Research Methodology

Qualitative Data Coding

What Is a Focus Group?

Cross-Cultural Research Methodology In Psychology

What Is Internal Validity In Research?

Research Methodology , Statistics

What Is Face Validity In Research? Importance & How To Measure

Criterion Validity: Definition & Examples

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• SAGE Open Nurs
• v.7; Jan-Dec 2021

Case Study Analysis as an Effective Teaching Strategy: Perceptions of Undergraduate Nursing Students From a Middle Eastern Country

Vidya seshan.

1 Maternal and Child Health Department, College of Nursing, Sultan Qaboos University, P.O. Box 66 Al-Khoudh, Postal Code 123, Muscat, Oman

Gerald Amandu Matua

2 Fundamentals and Administration Department, College of Nursing, Sultan Qaboos University, P.O. Box 66 Al-Khoudh, Postal Code 123, Muscat, Oman

Divya Raghavan

Judie arulappan, iman al hashmi, erna judith roach, sheeba elizebath sunderraj, emi john prince.

3 Griffith University, Nathan Campus, Queensland 4111

Background: Case study analysis is an active, problem-based, student-centered, teacher-facilitated teaching strategy preferred in undergraduate programs as they help the students in developing critical thinking skills. Objective: It determined the effectiveness of case study analysis as an effective teacher-facilitated strategy in an undergraduate nursing program. Methodology: A descriptive qualitative research design using focus group discussion method guided the study. The sample included undergraduate nursing students enrolled in the Maternal Health Nursing Course during the Academic Years 2017 and 2018. The researcher used a purposive sampling technique and a total of 22 students participated in the study, through five (5) focus groups, with each focus group comprising between four to six nursing students. Results: In total, nine subthemes emerged from the three themes. The themes were “Knowledge development”, “Critical thinking and Problem solving”, and “Communication and Collaboration”. Regarding “Knowledge development”, the students perceived case study analysis method as contributing toward deeper understanding of the course content thereby helping to reduce the gap between theory and practice especially during clinical placement. The “Enhanced critical thinking ability” on the other hand implies that case study analysis increased student's ability to think critically and aroused problem-solving interest in the learners. The “Communication and Collaboration” theme implies that case study analysis allowed students to share their views, opinions, and experiences with others and this enabled them to communicate better with others and to respect other's ideas which further enhanced their team building capacities. Conclusion: This method is effective for imparting professional knowledge and skills in undergraduate nursing education and it results in deeper level of learning and helps in the application of theoretical knowledge into clinical practice. It also broadened students’ perspectives, improved their cooperation capacity and their communication with each other. Finally, it enhanced student's judgment and critical thinking skills which is key for their success.

Introduction/Background

Recently, educators started to advocate for teaching modalities that not only transfer knowledge ( Shirani Bidabadi et al., 2016 ), but also foster critical and higher-order thinking and student-centered learning ( Wang & Farmer, 2008 ; Onweh & Akpan, 2014). Therefore, educators need to utilize proven teaching strategies to produce positive outcomes for learners (Onweh & Akpan, 2014). Informed by this view point, a teaching strategy is considered effective if it results in purposeful learning ( Centra, 1993 ; Sajjad, 2010 ) and allows the teacher to create situations that promote appropriate learning (Braskamp & Ory, 1994) to achieve the desired outcome ( Hodges et al., 2020 ). Since teaching methods impact student learning significantly, educators need to continuously test the effectives of their teaching strategies to ensure desired learning outcomes for their students given today's dynamic learning environments ( Farashahi & Tajeddin, 2018 ).

In this study, the researchers sought to study the effectiveness of case study analysis as an active, problem-based, student-centered, teacher-facilitated strategy in a baccalaureate-nursing program. This choice of teaching method is supported by the fact that nowadays, active teaching-learning is preferred in undergraduate programs because, they not only make students more powerful actors in professional life ( Bean, 2011 ; Yang et al., 2013 ), but they actually help learners to develop critical thinking skills ( Clarke, 2010 ). In fact, students who undergo such teaching approaches usually become more resourceful in integrating theory with practice, especially as they solve their case scenarios ( Chen et al., 2019 ; Farashahi & Tajeddin, 2018 ; Savery, 2019 ).

Review of Literature

As a pedagogical strategy, case studies allow the learner to integrate theory with real-life situations as they devise solutions to the carefully designed scenarios ( Farashahi & Tajeddin, 2018 ; Hermens & Clarke, 2009). Another important known observation is that case-study-based teaching exposes students to different cases, decision contexts and the environment to experience teamwork and interpersonal relations as “they learn by doing” thus benefiting from possibilities that traditional lectures hardly create ( Farashahi & Tajeddin, 2018 ; Garrison & Kanuka, 2004 ).

Another merit associated with case study method of teaching is the fact that students can apply and test their perspectives and knowledge in line with the tenets of Kolb et al.'s (2014) “experiential learning model”. This model advocates for the use of practical experience as the source of one's learning and development. Proponents of case study-based teaching note that unlike passive lectures where student input is limited, case studies allow them to draw from their own experience leading to the development of higher-order thinking and retention of knowledge.

Case scenario-based teaching also encourages learners to engage in reflective practice as they cooperate with others to solve the cases and share views during case scenario analysis and presentation ( MsDade, 1995 ).

This method results in “idea marriage” as learners articulate their views about the case scenario. This “idea marriage” phenomenon occurs through knowledge transfer from one situation to another as learners analyze scenarios, compare notes with each other, and develop multiple perspectives of the case scenario. In fact, recent evidence shows that authentic case-scenarios help learners to acquire problem solving and collaborative capabilities, including the ability to express their own views firmly and respectfully, which is vital for future success in both professional and personal lives ( Eronen et al., 2019 ; Yajima & Takahashi, 2017 ). In recognition of this higher education trend toward student-focused learning, educators are now increasingly expected to incorporate different strategies in their teaching.

This study demonstrated that when well implemented, educators can use active learning strategies like case study analysis to aid critical thinking, problem-solving, and collaborative capabilities in undergraduate students. This study is significant because the findings will help educators in the country and in the region to incorporate active learning strategies such as case study analysis to aid critical thinking, problem-solving, and collaborative capabilities in undergraduate students. Besides, most studies on the case study method in nursing literature mostly employ quantitative methods. The shortage of published research on the case study method in the Arabian Gulf region and the scanty use of qualitative methods further justify why we adopted the focus group method for inquiry.

A descriptive qualitative research design using focus group discussion method guided the study. The authors chose this method because it is not only inexpensive, flexible, stimulating but it is also known to help with information recall and results in rich data ( Matua et al., 2014 ; Streubert & Carpenter, 2011 ). Furthermore, as evidenced in the literature, the focus group discussion method is often used when there is a need to gain an in-depth understanding of poorly understood phenomena as the case in our study. The choice of this method is further supported by the scarcity of published research related to the use of case study analysis as a teaching strategy in the Middle Eastern region, thereby further justifying the need for an exploratory research approach for our study.

As a recommended strategy, the researchers generated data from information-rich purposively selected group of baccalaureate nursing students who had experienced both traditional lectures and cased-based teaching approaches. The focus group interviews allowed the study participants to express their experiences and perspectives in their own words. In addition, the investigators integrated participants’ self-reported experiences with their own observations and this enhanced the study findings ( Morgan & Bottorff, 2010 ; Nyumba et al., 2018 ; Parker & Tritter, 2006 ).

Eligibility Criteria

In order to be eligible to participate in the study, the participants had to:

• be a baccalaureate nursing student in College of Nursing, Sultan Qaboos University
• register for Maternity Nursing Course in 2017 and 2018.
• attend all the Case Study Analysis sessions in the courses before the study.
• show a willingness to participate in the study voluntarily and share their views freely.

The population included the undergraduate nursing students enrolled in the Maternal Health Nursing Course during the Academic Years 2017 and 2018.

The researcher used a purposive sampling technique to choose participants who were capable of actively participating and discussing their views in the focus group interviews. This technique enabled the researchers to select participants who could provide rich information and insights about case study analysis method as an effective teaching strategy. The final study sample included baccalaureate nursing students who agreed to participate in the study by signing a written informed consent. In total, twenty-two (22) students participated in the study, through five focus groups, with each focus group comprising between four and six students. The number of participants was determined by the stage at which data saturation was reached. The point of data saturation is when no new information emerges from additional participants interviewed ( Saunders et al., 2018 ).Focus group interviews were stopped once data saturation was achieved. Qualitative research design with focus group discussion allowed the researchers to generate data from information-rich purposively selected group of baccalaureate nursing students who had experienced both traditional lectures and case-based teaching approaches. The focus group interviews allowed the study participants to express their perspectives in their own words. In addition, the investigators enhanced the study findings by integrating participants’ self-reported experiences with the researchers’ own observations and notes during the study.

The study took place at College of Nursing; Sultan Qaboos University, Oman's premier public university, in Muscat. This is the only setting chosen for the study. The participants are the students who were enrolled in Maternal Health Nursing course during 2017 and 2018. The interviews occurred in the teaching rooms after official class hours. Students who did not participate in the study learnt the course content using the traditional lecture based method.

Ethical Considerations

Permission to conduct the study was granted by the College Research and Ethics Committee (XXXX). Prior to the interviews, each participant was informed about the purpose, benefits as well as the risks associated with participating in the study and clarifications were made by the principal researcher. After completing this ethical requirement, each student who accepted to participate in the study proceeded to sign an informed consent form signifying that their participation in the focus group interview was entirely voluntary and based on free will.

The anonymity of study participants and confidentiality of their data was upheld throughout the focus group interviews and during data analysis. To enhance confidentiality and anonymity of the data, each participant was assigned a unique code number which was used throughout data analysis and reporting phases. To further assure the confidentiality of the research data and anonymity of the participants, all research-related data were kept safe, under lock and key and through digital password protection, with unhindered access only available to the research team.

Research Intervention

In Fall 2017 and Spring 2018 semesters, as a method of teaching Maternal Health Nursing course, all students participated in two group-based case study analysis exercises which were implemented in the 7 th and 13 th weeks. This was done after the students were introduced to the case study method using a sample case study prior to the study. The instructor explained to the students how to solve the sample problem, including how to accomplish the role-specific competencies in the courses through case study analysis. In both weeks, each group consisting of six to seven students was assigned to different case scenarios to analyze and work on, after which they presented their collective solution to the case scenarios to the larger class of 40 students. The case scenarios used in both weeks were peer-reviewed by the researchers prior to the study.

Pilot Study

A group of three students participated as a pilot group for the study. However, the students who participated in the pilot study were not included in the final study as is general the principle with qualitative inquiry because of possible prior exposure “contamination”. The purpose of piloting was to gather data to provide guidance for a substantive study focusing on testing the data collection procedure, the interview process including the sequence and number of questions and probes and recording equipment efficacy. After the pilot phase, the lessons learned from the pilot were incorporated to ensure smooth operations during the actual focus group interview ( Malmqvist et al., 2019 .

Data Collection

The focus group interviews took place after the target population was exposed to case study analysis method in Maternal Health Nursing course during the Fall 2017 and Spring 2018 semesters. Before data collection began, the research team pilot tested the focus group interview guide to ensure that all the guide questions were clear and well understood by study participants.

In total, five (5) focus groups participated in the study, with each group comprising between four and six students. The focus group interviews lasted between 60 and 90 min. In addition to the interview guide questions, participants’ responses to unanswered questions were elicited using prompts to facilitate information flow whenever required. As a best practice, all the interviews were audio-recorded in addition to extensive field notes taken by one of the researchers. The focus group interviews continued until data saturation occurred in all the five (5) focus groups.

Credibility

In this study, participant's descriptions were digitally audio recorded to ensure that no information was lost. In order to ensure that the results are accurate, verbatim transcriptions of the audio recordings were done supported by interview notes. Furthermore, interpretations of the researcher were verified and supported with existing literature with oversight from the research team.

Transferability

The researcher provided a detailed description about the study settings, participants, sampling technique, and the process of data collection and analyses. The researcher used verbatim quotes from various participants to aid the transferability of the results.

Dependability

The researcher ensured that the research process is clearly documented, traceable, and logical to achieve dependability of the research findings. Furthermore, the researcher transparently described the research steps, procedures and process from the start of the research project to the reporting of the findings.

Confirmability

In this study, confirmability of the study findings was achieved through the researcher's efforts to make the findings credible, dependable, and transferable.

Data Analysis

Data were analyzed manually after the lead researcher integrated the verbatim transcriptions with the extensive field notes to form the final data set. Data were analyzed thematically under three thematic areas of a) knowledge development; b) critical thinking and problem solving; and (c) communication and collaboration, which are linked to the study objectives. The researchers used the Six (6) steps approach to conduct a trustworthy thematic analysis: (1) familiarization with the research data, (2) generating initial codes, (3) searching for themes, (4) reviewing the themes, (5) defining and naming themes, (6) writing the report ( Nowell et al., 2017 ).

The analysis process started with each team member individually reading and re-reading the transcripts several times and then identifying meaning units linked to the three thematic areas. The co-authors then discussed in-depth the various meaning units linked to the thematic statements until consensus was reached and final themes emerged based on the study objectives.

A total of 22 undergraduate third-year baccalaureate nursing students who were enrolled in the Maternal Health Nursing Course during the Academic Years 2017 and 2018 participated in the study, through five focus groups, with each group comprising four to six students. Of these, 59% were females and 41% were males. In total, nine subthemes emerged from the three themes. Under knowledge development, emerged the subthemes, “ deepened understanding of content ; “ reduced gap between theory and practice” and “ improved test-taking ability ”. While under Critical thinking and problem solving, emerged the subthemes, “ enhanced critical thinking ability ” and “ heightened curiosity”. The third thematic area of communication and collaboration yielded, “ improved communication ability ”; “ enhanced team-building capacity ”; “ effective collaboration” and “ improved presentation skills ”, details of which are summarized in Table 1 .

Table 1.

Objective Linked Themes and Student Perceptions of Outcome Case Study Analysis.

Theme 1: Knowledge Development

In terms of knowledge development, students expressed delight at the inclusion of case study analysis as a method during their regular theory class. The first subtheme related to knowledge development that supports the adoption of the case study approach is its perceived benefit of ‘ deepened understanding of content ’ by the students as vividly described by this participant:

“ I was able to perform well in the in-course exams as this teaching method enhanced my understanding of the content rather than memorizing ” (FGD#3).

The second subtheme related to knowledge development was informed by participants’ observation that teaching them using case study analysis method ‘ reduced the gap between theory and practice’. This participant's claim stem from the realization that, a case study scenario his group analyzed in the previous week helped him and his colleagues to competently deal with a similar situation during clinical placement the following week, as articulated below:

“ You see when I was caring for mothers in antenatal unit, I could understand the condition better and could plan her care well because me and my group already analyzed a similar situation in class last week which the teacher gave us, this made our work easier in the ward”. (FGD#7).

Another student added that:

“ It was useful as what is taught in the theory class could be applied to the clinical cases.”

This ‘theory-practice’ connection was particularly useful in helping students to better understand how to manage patients with different health conditions. Interestingly, the students reported that they were more likely to link a correct nursing care plan to patients whose conditions were close to the case study scenarios they had already studied in class as herein affirmed:

“ …when in the hospital I felt I could perceive the treatment modality and plan for [a particular] nursing care well when I [had] discussed with my team members and referred the textbook resource while performing case study discussion”. (FGD#17).

In a similar way, another student added:

“…I could relate with the condition I have seen in the clinical area. So this has given me a chance to recall the condition and relate the theory to practice”. (FGD#2) .

The other subtheme closely related to case study scenarios as helping to deepen participant's understanding of the course content, is the notion that this teaching strategy also resulted in ‘ improved test taking-ability’ as this participant's verbatim statement confirms:

“ I could answer the questions related to the cases discussed [much] better during in-course exams. Also [the case scenarios] helped me a great deal to critically think and answer my exam papers” (FGD#11).

Theme 2: Critical Thinking and Problem Solving

In this subtheme, students found the case study analysis as an excellent method to learn disease conditions in the two courses. This perceived success with the case study approach is associated with the method's ability to ‘ enhance students’ critical thinking ability’ as this student declares:

“ This method of teaching increased my ability to think critically as the cases are the situations, where we need to think to solve the situation”. (FGD#5)

This enhanced critical thinking ability attributed to case study scenario analysis was also manifested during patient care where students felt it allowed them to experience a “ flow of patient care” leading to better patient management planning as would typically occur during case scenario analysis. In support of this finding, a participant mentioned that:

“ …I could easily connect the flow of patient care provided and hence was able to plan for [his] management as often required during case study discussion” (FGD#12)

Another subtheme linked with this theme is the “ heightened curiosity” associated with the case scenario discussions. It was clear from the findings that the cases aroused curiosity in the mind of the students. This heightened interest meant that during class discussion, baccalaureate nursing students became active learners, eager to discover the next set of action as herein affirmed:

“… from the beginning of discussion with the group, I was eager to find the answer to questions presented and wanted to learn the best way for patient management” (FGD#14)

Theme 3: Communication and Collaboration

In terms of its impact on student communication, the subtheme revealed that case study analysis resulted in “ improved communication ability” among the nursing students . This enhanced ability of students to exchange ideas with each other may be attributed to the close interaction required to discuss and solve their assigned case scenarios as described by the participant below:

“ as [case study analysis] was done in the way of group discussion, I felt me and my friends communicated more within the group as we discussed our condition. We also learnt from each other, and we became better with time.” (FGD#21).

The next subtheme further augments the notion that case study analysis activities helped to “ enhance team-building capacity” of students as this participant affirmatively narrates:

“ students have the opportunity to meet face to face to share their views, opinion, and their experience, as this build on the way they can communicate with each other and respect each other's opinions and enhance team-building”. (FGD#19).

Another subtheme revealed from the findings show that the small groups in which the case analysis occurs allowed the learners to have deeper and more focused conversations with one another, resulting in “ an effective collaboration between students” as herein declared:

“ We could collaborate effectively as we further went into a deep conversation on the case to solve”. (FGD#16).

Similarly, another student noted that:

“ …discussion of case scenarios helped us to prepare better for clinical postings and simulation lab experience” (FGD#5) .

A fourth subtheme related to communication found that students also identified that case study analysis resulted in “ improved presentation skills”. This is attributed in part to the preparation students have to go through as part of their routine case study discussion activities, which include organizing their presentations and justifying and integrating their ideas. Besides readying themselves for case presentations, the advice, motivation, and encouragement such students receive from their faculty members and colleagues makes them better presenters as confirmed below:

“ …teachers gave us enough time to prepare, hence I was able to present in front of the class regarding the finding from our group.” (FGD#16).

In this study, the researches explored learner's perspectives on how one of the active teaching strategies, case study analysis method impacted their knowledge development, critical thinking, and problem solving as well as communication and collaboration ability.

Knowledge Development

In terms of knowledge development, the nursing students perceived case study analysis as contributing toward: (a) deeper understanding of content, (b) reducing gap between theory and practice, and (c) improving test-taking ability. Deeper learning” implies better grasping and retention of course content. It may also imply a deeper understanding of course content combined with learner's ability to apply that understanding to new problems including grasping core competencies expected in future practice situations (Rickles et al., 2019; Rittle-Johnson et al., 2020 ). Deeper learning therefore occurs due to the disequilibrium created by the case scenario, which is usually different from what the learner already knows ( Hattie, 2017 ). Hence, by “forcing” students to compare and discuss various options in the quest to solve the “imbalance” embedded in case scenarios, students dig deeper in their current understanding of a given content including its application to the broader context ( Manalo, 2019 ). This movement to a deeper level of understanding arises from carefully crafted case scenarios that instructors use to stimulate learning in the desired area (Nottingham, 2017; Rittle-Johnson et al., 2020 ). The present study demonstrated that indeed such carefully crafted case study scenarios did encourage students to engage more deeply with course content. This finding supports the call by educators to adopt case study as an effective strategy.

Another finding that case study analysis method helps in “ reducing the gap between theory and practice ” implies that the method helps students to maintain a proper balance between theory and practice, where they can see how theoretical knowledge has direct practical application in the clinical area. Ajani and Moez (2011) argue that to enable students to link theory and practice effectively, nurse educators should introduce them to different aspects of knowledge and practice as with case study analysis. This dual exposure ensures that students are proficient in theory and clinical skills. This finding further amplifies the call for educators to adequately prepare students to match the demands and realities of modern clinical environments ( Hickey, 2010 ). This expectation can be met by ensuring that student's knowledge and skills that are congruent with hospital requirements ( Factor et al., 2017 ) through adoption of case study analysis method which allows integration of clinical knowledge in classroom discussion on regular basis.

The third finding, related to “improved test taking ability”, implies that case study analysis helped them to perform better in their examination, noting that their experience of going through case scenario analysis helped them to answer similar cases discussed in class much better during examinations. Martinez-Rodrigo et al. (2017) report similar findings in a study conducted among Spanish electrical engineering students who were introduced to problem-based cooperative learning strategies, which is similar to case study analysis method. Analysis of student's results showed that their grades and pass rates increased considerably compared to previous years where traditional lecture-based method was used. Similar results were reported by Bonney (2015) in an even earlier study conducted among biology students in Kings Borough community college students, in New York, United States. When student's performance in examination questions covered by case studies was compared with class-room discussions, and text-book reading, case study analysis approach was significantly more effective compared to traditional methods in aiding students’ performance in their examinations. This finding therefore further demonstrates that case study analysis method indeed improves student's test taking ability.

Critical Thinking and Problem Solving

In terms of critical thinking and problem-solving ability, the use of case study analysis resulted in two subthemes: (a) enhanced critical thinking ability and (b) heightened learner curiosity. The “ enhanced critical thinking ability” implies that case analysis increased student's ability to think critically as they navigated through the case scenarios. This observation agrees with the findings of an earlier questionnaire-based study conducted among 145 undergraduate business administration students at Chittagong University, Bangladesh, that showed 81% of respondents agree that case study analysis develops critical thinking ability and enables students to do better problem analysis ( Muhiuddin & Jahan, 2006 ). This observation agrees with the findings of an earlier study conducted among 145 undergraduate business administration students at Chittagong University, Bangladesh. The study showed that 81% of respondents agreed that case study analysis facilitated the development of critical thinking ability in the learners and enabled the students to perform better with problem analysis ( Muhiuddin & Jahan, 2006 ).

More recently, Suwono et al. (2017) found similar results in a quasi-experimental research conducted at a Malaysian university. The research findings showed that there was a significant difference in biological literacy and critical thinking skills between the students taught using socio-biological case-based learning and those taught using traditional lecture-based learning. The researchers concluded that case-based learning enhanced the biological literacy and critical thinking skills of the students. The current study adds to the existing pedagogical knowledge base that case study methodology can indeed help to deepen learner's critical thinking and problem solving ability.

The second subtheme related to “ heightened learner curiosity” seems to suggest that the case studies aroused problem-solving interest in learners. This observation agrees with two earlier studies by Tiwari et al. (2006) and Flanagan and McCausland (2007) who both reported that most students enjoyed case-based teaching. The authors add that the case study method also improved student's clinical reasoning, diagnostic interpretation of patient information as well as their ability to think logically when presented a challenge in the classroom and in the clinical area. Jackson and Ward (2012) similarly reported that first year engineering undergraduates experienced enhanced student motivation. The findings also revealed that the students venturing self-efficacy increased much like their awareness of the importance of key aspects of the course for their future careers. The authors conclude that the case-based method appears to motivate students to autonomously gather, analyze and present data to solve a given case. The researchers observed enhanced personal and collaborative efforts among the learners, including improved communication ability. Further still, learners were more willing to challenge conventional wisdom, and showed higher “softer” skills after exposure to case analysis based teaching method. These findings like that of the current study indicate that teaching using case based analysis approach indeed motivates students to engage more in their learning, there by resulting in deeper learning.

Communication and Collaboration

Case study analysis is also perceived to result in: (a) improved communication ability; (b) enhanced team -building capacity, (c) effective collaboration ability, and (d) enhanced presentation skills. The “ improved communication ability ” manifested in learners being better able to exchange ideas with peers, communicating their views more clearly and collaborating more effectively with their colleagues to address any challenges that arise. Fini et al. (2018) report comparable results in a study involving engineering students who were subjected to case scenario brainstorming activities about sustainability concepts and their implications in transportation engineering in selected courses. The results show that this intervention significantly improved student's communication skills besides their higher-order cognitive, self-efficacy and teamwork skills. The researchers concluded that involving students in brainstorming activities related to problem identification including their practical implications, is an effective teaching strategy. Similarly, a Korean study by Park and Choi (2018) that sought to analyze the effects of case-based communication training involving 112 sophomore nursing students concluded that case-based training program improved the students’ critical thinking ability and communication competence. This finding seems to support further the use of case based teaching as an effective teaching-learning strategy.

The “ enhanced team-building capacity” arose from the opportunity students had in sharing their views, opinions, and experiences where they learned to communicate with each other and respect each other's ideas which further enhance team building. Fini et al. (2018) similarly noted that increased teamwork levels were seen among their study respondents when the researchers subjected engineering students to case scenario based-brainstorming activities as occurs with case study analysis teaching. Likewise, Lairamore et al. (2013) report similar results in their study that showed that case study analysis method increased team work ability and readiness among students from five health disciplines in a US-based study.

The finding that case study analysis teaching method resulted in “ effective collaboration ability” among students manifested as students entered into deep conversation as they solved the case scenarios. Rezaee and Mosalanejad (2015) assert that such innovative learning strategies result in noticeable educational outcomes, such as greater satisfaction with and enjoyment of the learning process ( Wellmon et al., 2012 ). Further, positive attitudes toward learning and collaboration have been noted leading to deeper learning as students prepare for case discussions ( Rezaee & Mosalanejad, 2015 ). This results show that case study analysis can be utilized by educators to foster professional collaboration among their learners, which is one of the key expectations of new graduates today.

The finding associated with “improved presentation skills” is consistent with the results of a descriptive study in Saudi Arabia that compared case study and traditional lectures in the teaching of physiology course to undergraduate nursing students. The researchers found that case-based teaching improved student’ overall knowledge and performance in the course including facilitating the acquisition of skills compared to traditional lectures ( Majeed, 2014 ). Noblitt et al. (2010) report similar findings in their study that compares traditional presentation approach with the case study method for developing and improving student's oral communication skills. This finding extends our understanding that case study method improves learners’ presentation skills.

The study was limited to level third year nursing students belonging to only one college and the sample size, which might limit the transferability of the study findings to other settings.

Implications for Practice

These study findings add to the existing body of knowledge that places case study based teaching as a tested method that promotes perception learning where students’ senses are engaged as a result of the real-life and authentic clinical scenarios ( Malesela, 2009 ), resulting in deeper learning and achievement of long-lasting knowledge ( Fiscus, 2018 ). The students reported that case scenario discussions broadened their perspectives, improved their cooperation capacity and communication with each other. This teaching method, in turn, offers students an opportunity to enhance their judgment and critical thinking skills by applying theory into practice.

These skills are critically important because nurses need to have the necessary knowledge and skills to plan high quality care for their patients to achieve a speedy recovery. In order to attain this educational goal, nurse educators have to prepare students through different student- centered strategies. The findings of our study appear to show that when appropriately used, case-based teaching results in acquisition of disciplinary knowledge manifested by deepened understanding of course content, as well as reducing the gap between theory and practice and enhancing learner's test-taking-ability. The study also showed that cased based teaching enhanced learner's critical thinking ability and curiosity to seek and acquire a deeper knowledge. Finally, the study results indicate that case study analysis results in improved communication and enhanced team-building capacity, collaborative ability and improved oral communication and presentation skills. The study findings and related evidence from literature show that case study analysis is well- suited approach for imparting knowledge and skills in baccalaureate nursing education.

This study evaluated the usefulness of Case Study Analysis as a teaching strategy. We found that this method of teaching helps encourages deeper learning among students. For instructors, it provides the opportunity to tailor learning experiences for students to undertake in depth study in order to stimulate deeper understanding of the desired content. The researchers conclude that if the cases are carefully selected according to the level of the students, and are written realistically and creatively and the group discussions keep students well engaged, case study analysis method is more effective than other traditional lecture methods in facilitating deeper and transferable learning/skills acquisition in undergraduate courses.

Conflict of Interest: The authors declare no conflict of interest.

ORCID iD: Judie Arulappan https://orcid.org/0000-0003-2788-2755

• Ajani K., Moez S. (2011). Gap between knowledge and practice in nursing . Procedia-Social and Behavioral Sciences , 15 , 3927–3931. 10.1016/j.sbspro.2011.04.396 [ CrossRef ] [ Google Scholar ]
• Bean J. C. (2011). Engaging ideas: The professor’s guide to integrating writing critical thinking and active-learning in the classroom (2nd ed.). Jossey-Bass. [ Google Scholar ]
• Bonney K. M. (2015). Case study teaching method improves student performance and perceptions of learning gains . Journal of Microbiology & Biology Education , 16 ( 1 ), 21–28. 10.1128/jmbe.v16i1.846 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Braskamp L. A., Ory J. C. (1994). Assessing faculty work: Enhancing individual and institutional performance . Jossey-Bass Higher and Adult Education Series. Jossey-Bass Inc. [ Google Scholar ]
• Centra J. A. (1993). Reflective faculty evaluation: Enhancing teaching and determining faculty effectiveness . Jossey-Bass. [ Google Scholar ]
• Chen W., Shah U. V., Brechtelsbauer C. (2019). A framework for hands-on learning in chemical engineering education—training students with the end goal in mind . Education for Chemical Engineers , 28 , 25–29. 10.1016/j.ece.2019.03.002 [ CrossRef ] [ Google Scholar ]
• Clarke J. (2010). Student centered teaching methods in a Chinese setting . Nurse Education Today , 30 ( 1 ), 15–19. 10.1016/j.nedt.2009.05.009 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Eronen L., Kokko S., Sormunen K. (2019). Escaping the subject-based class: A Finnish case study of developing transversal competencies in a transdisciplinary course . The Curriculum Journal , 30 ( 3 ), 264–278. 10.1080/09585176.2019.1568271 [ CrossRef ] [ Google Scholar ]
• Factor E. M. R., Matienzo E. T., de Guzman A. B. (2017). A square peg in a round hole: Theory-practice gap from the lens of Filipino student nurses . Nurse Education Today , 57 , 82–87. 10.1016/j.nedt.2017.07.004 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Farashahi M., Tajeddin M. (2018). Effectiveness of teaching methods in business education: A comparison study on the learning outcomes of lectures, case studies and simulations . The International Journal of Management Education , 16 ( 1 ), 131–142. 10.1016/j.ijme.2018.01.003 [ CrossRef ] [ Google Scholar ]
• Fini E. H., Awadallah F., Parast M. M., Abu-Lebdeh T. (2018). The impact of project-based learning on improving student learning outcomes of sustainability concepts in transportation engineering courses . European Journal of Engineering Education , 43 ( 3 ), 473–488. 10.1080/03043797.2017.1393045 [ CrossRef ] [ Google Scholar ]
• Fiscus J. (2018). Reflection in Motion: A Case Study of Reflective Practice in the Composition Classroom [ Doctoral dissertation ]. Source: http://hdl.handle.net/1773/42299 [ Google Scholar ]
• Flanagan N. A., McCausland L. (2007). Teaching around the cycle: Strategies for teaching theory to undergraduate nursing students . Nursing Education Perspectives , 28 ( 6 ), 310–314. [ PubMed ] [ Google Scholar ]
• Garrison D. R., Kanuka H. (2004). Blended learning: Uncovering its transformative potential in higher education . The internet and higher education , 7 ( 2 ), 95–105. 10.1016/j.iheduc.2004.02.001 [ CrossRef ] [ Google Scholar ]
• Hattie J. (2017). Foreword . In Nottingham J. (Ed.), The learning challenge: How to guide your students through the learning pit to achieve deeper understanding . Corwin Press, p. xvii. [ Google Scholar ]
• Hermens A., Clarke E. (2009). Integrating blended teaching and learning to enhance graduate attributes . Education+ Training , 51 ( 5/6 ), 476–490. [ Google Scholar ]
• Hickey M. T. (2010). Baccalaureate nursing graduates’ perceptions of their clinical instructional experiences and preparation for practice . Journal of Professional Nursing , 26 ( 1 ), 35–41. 10.1016/j.profnurs.2009.03.001 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Hodges C., Moore S., Lockee B., Trust T., Bond A. (2020). The difference between emergency remote teaching and online learning . Educause review , 27 , 1–12. [ Google Scholar ]
• Jackson N. R., Ward A. E. (2012). Curiosity based learning: Impact study in 1st year electronics undergraduates. 2012 International Conference on Information Technology Based Higher Education and Training (ITHET), Istanbul, pp. 1–6. 10.1109/ITHET.2012.6246005. [ CrossRef ] [ Google Scholar ]
• Kolb A. Y., Kolb D. A., Passarelli A., Sharma G. (2014). On becoming an experiential educator: The educator role profile . Simulation & Gaming , 45 ( 2 ), 204–234. 10.1177/1046878114534383 [ CrossRef ] [ Google Scholar ]
• Lairamore C., George-Paschal L., McCullough K., Grantham M., Head D. (2013). A case-based interprofessional education forum improves students’ perspectives on the need for collaboration, teamwork, and communication . MedEdPORTAL, The Journal of Teaching and learning resources , 9 , 10.15766/mep_2374-8265.9484 [ CrossRef ] [ Google Scholar ]
• Majeed F. (2014). Effectiveness of case based teaching of physiology for nursing students . Journal of Taibah University Medical Sciences , 9 ( 4 ), 289–292. 10.1016/j.jtumed.2013.12.005 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Malesela J. M. (2009). Case study as a learning opportunity among nursing students in a university . Health SA Gesondheid (Online) , 14 ( 1 ), 33–38. 10.4102/hsag.v14i1.434 [ CrossRef ] [ Google Scholar ]
• Malmqvist J., Hellberg K., Möllås G., Rose R., Shevlin M. (2019). Conducting the pilot study: A neglected part of the research process? Methodological findings supporting the importance of piloting in qualitative research studies . International Journal of Qualitative Methods , 18 . 10.1177/1609406919878341 [ CrossRef ] [ Google Scholar ]
• Manalo E. (ed.). (2019). Deeper learning, dialogic learning, and critical thinking: Research-based strategies for the classroom . Routledge. [ Google Scholar ]
• Martinez-Rodrigo F., Herrero-De Lucas L. C., De Pablo S., Rey-Boue A. B. (2017). Using PBL to improve educational outcomes and student satisfaction in the teaching of DC/DC and DC/AC converters . IEEE Transactions on Education , 60 ( 3 ), 229–237. 10.1109/TE.2016.2643623 [ CrossRef ] [ Google Scholar ]
• Matua G. A., Seshan V., Akintola A. A., Thanka A. N. (2014). Strategies for providing effective feedback during preceptorship: Perspectives from an Omani Hospital . Journal of Nursing Education and Practice , 4 ( 10 ), 24. 10.5430/jnep.v4n10p24 [ CrossRef ] [ Google Scholar ]
• Morgan D. L., Bottorff J. L. (2010). Advancing our craft: Focus group methods and practice . Qualitative Health Research , 20 ( 5 ), 579–581. 10.1177/1049732310364625 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• MsDade S. A. (1995). Case study pedagogy to advance critical thinking . Teaching psychology , 22 ( 1 ), 9–10. 10.1207/s15328023top2201_3 [ CrossRef ] [ Google Scholar ]
• Muhiuddin G., Jahan N. (2006). Students’ perception towards case study as a method of learning in the field of business administration’ . The Chittagong University Journal of Business Administration , 21 , 25–41. [ Google Scholar ]
• Noblitt L., Vance D. E., Smith M. L. D. (2010). A comparison of case study and traditional teaching methods for improvement of oral communication and critical-thinking skills . Journal of College Science Teaching , 39 ( 5 ), 26–32. [ Google Scholar ]
• Nottingham J. (2017). The learning challenge: How to guide your students through the learning pit to achieve deeper understanding . Corwin Press. [ Google Scholar ]
• Nowell L. S., Norris J. M., White D. E., Moules N. J. (2017). Thematic analysis: Striving to meet the trustworthiness criteria . International Journal of Qualitative Methods , 16 ( 1 ). 10.1177/1609406917733847 [ CrossRef ] [ Google Scholar ]
• Nyumba T., Wilson K., Derrick C. J., Mukherjee N. (2018). The use of focus group discussion methodology: Insights from two decades of application in conservation . Methods in Ecology and evolution , 9 ( 1 ), 20–32. 10.1111/2041-210X.12860 [ CrossRef ] [ Google Scholar ]
• Onweh V. E., Akpan U. T. (2014). Instructional strategies and students academic performance in electrical installation in technical colleges in Akwa Ibom State: Instructional skills for structuring appropriate learning experiences for students . International Journal of Educational Administration and Policy Studies , 6 ( 5 ), 80–86. [ Google Scholar ]
• Park S. J., Choi H. S. (2018). The effect of case-based SBAR communication training program on critical thinking disposition, communication self-efficacy and communication competence of nursing students . Journal of the Korea Academia-Industrial Cooperation Society , 19 ( 11 ), 426–434. 10.5762/KAIS.2018.19.11.426 [ CrossRef ] [ Google Scholar ]
• Parker A., Tritter J. (2006). Focus group method and methodology: Current practice and recent debate . International Journal of Research & Method in Education , 29 ( 1 ), 23–37. 10.1080/01406720500537304 [ CrossRef ] [ Google Scholar ]
• Rezaee R., Mosalanejad L. (2015). The effects of case-based team learning on students’ learning, self-regulation and self-direction . Global Journal of Health Science , 7 ( 4 ), 295. 10.5539/gjhs.v7n4p295 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Rickles J., Zeiser K. L., Yang R., O’Day J., Garet M. S. (2019). Promoting deeper learning in high school: Evidence of opportunities and outcomes . Educational Evaluation and Policy Analysis , 41 ( 2 ), 214–234. [ Google Scholar ]
• Rittle-Johnson B., Star J. R., Durkin K., Loehr A. (2020). Compare and discuss to promote deeper learning. Deeper learning, dialogic learning, and critical thinking: Research-based strategies for the classroom . Routlegde, p. 48. 10.4324/9780429323058-4 [ CrossRef ] [ Google Scholar ]
• Sajjad S. (2010). Effective teaching methods at higher education level . Pakistan Journal of Special Education , 11 , 29–43. [ Google Scholar ]
• Saunders B., Sim J., Kingstone T., Baker S., Waterfield J., Bartlam B., Burroughs H., Jinks C. (2018). Saturation in qualitative research: Exploring its conceptualization and operationalization . Quality & Quantity , 52 ( 4 ), 1893–1907. 10.1007/s11135-017-0574-8 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Savery J. R. (2019). Comparative pedagogical models of problem based learning . The Wiley Handbook of Problem Based Learning , 81–104. 10.1002/9781119173243.ch4 [ CrossRef ] [ Google Scholar ]
• Shirani Bidabadi N., Nasr Isfahani A., Rouhollahi A., Khalili R. (2016). Effective teaching methods in higher education: Requirements and barriers . Journal of Advances in Medical Education & Professionalism , 4 ( 4 ), 170–178. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Streubert H. J., Carpenter D. R. (2011). Qualitative research in nursing: Advancing the humanistic imperative . Wolters Kluwer. [ Google Scholar ]
• Suwono H., Pratiwi H. E., Susanto H., Susilo H. (2017). Enhancement of students’ biological literacy and critical thinking of biology through socio-biological case-based learning . JurnalPendidikan IPA Indonesia , 6 ( 2 ), 213–220. 10.15294/jpii.v6i2.9622 [ CrossRef ] [ Google Scholar ]
• Tiwari A., Lai P., So M., Yuen K. (2006). A comparison of the effects of problem-based learning and lecturing on the development of students’ critical thinking . Medical Education , 40 ( 6 ), 547–554. 10.1111/j.1365-2929.2006.02481.x [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Wang V., Farmer L. (2008). Adult teaching methods in China and bloom's taxonomy . International Journal for the Scholarship of Teaching and Learning , 2 ( 2 ), n2. 10.20429/ijsotl.2008 [ CrossRef ] [ Google Scholar ]
• Wellmon R., Gilin B., Knauss L., Linn M. I. (2012). Changes in student attitudes toward interprofessional learning and collaboration arising from a case-based educational experience . Journal of Allied Health , 41 ( 1 ), 26–34. [ PubMed ] [ Google Scholar ]
• Yajima K., Takahashi S. (2017). Development of evaluation system of AL students . Procedia Computer Science , 112 , 1388–1395. 10.1016/j.procs.2017.08.056 [ CrossRef ] [ Google Scholar ]
• Yang W. P., Chao C. S. C., Lai W. S., Chen C. H., Shih Y. L., Chiu G. L. (2013). Building a bridge for nursing education and clinical care in Taiwan—using action research and confucian tradition to close the gap . Nurse Education Today , 33 ( 3 ), 199–204. 10.1016/j.nedt.2012.02.016 [ PubMed ] [ CrossRef ] [ Google Scholar ]

• MONITORING AND EVALUATION APPROACHES
• Learning Center

Monitoring and evaluation (M&E) are two essential components of project management that help organizations assess the progress and effectiveness of their programs. Monitoring and evaluation approaches are essential for any organization for measuring the progress and success of any project or program. Evaluation approaches have often been developed to address specific evaluation questions or challenges and they refer to an integrated package of methods and processes.

Table of contents

Results-based monitoring and evaluation approach

Participatory monitoring and evaluation approach, theory-based evaluation approach.

• Utilization-focused evaluation approach

M&E for learning

• Gender-responsive evaluation

Case study evaluation approach

Process monitoring and evaluation approach, impact evaluation approach.

• Evaluation approaches versus evaluation methods

Conclusion on monitoring and evaluation approaches

This approach involves setting specific, measurable, achievable, relevant, and time-bound (SMART) indicators for a project and tracking progress against these indicators. It emphasizes the importance of measuring outcomes and impact rather than just activities. Results-based monitoring and evaluation (M&E) approaches can provide the insight needed to evaluate performance and strategy. Results-based M&E involves collecting and analyzing data to assess the impact of programs and identify areas for improvement. It helps organizations understand where they need to focus their resources, and allows them to ensure that projects are meeting established goals. Results-based M&E is an invaluable tool for ensuring efficiency, effectiveness and accountability in any organization’s operations. Read more .

This approach involves involving stakeholders, including beneficiaries, in the monitoring and evaluation process. It can help ensure that the evaluation is sensitive to the needs of those who are intended to benefit from the project. It provides an insight into the progress of the program or project and helps to identify problems that need immediate attention. Participatory monitoring and evaluation approaches help to ensure that all stakeholders are engaged in the evaluation process, bringing a wider perspective and enabling more effective feedback. Through this method, progress and impact can be better understood, allowing for better decisions in order to reach desired outcomes. Participatory approaches are therefore an important part of monitoring and evaluation for any project or program. Read more .

This approach involves examining the underlying theory of change that a project is based on to determine whether the assumptions about how the project will work are valid. It can help identify what changes are likely to occur and how they can be measured. The Theory-based Evaluation approach is a powerful monitoring and evaluation tool that can help organizations make informed decisions about their programs and services. This approach focuses on the underlying theories of change that drive program implementation and outcomes, and helps to identify and address gaps in the program’s effectiveness. It also serves as a way to measure the progress of a program and its impact on the target population. Theory-based evaluation is a comprehensive approach that considers both qualitative and quantitative data, and is useful for understanding the complex relationships between program activities and outcomes. It is an important tool for organizations to ensure that their programs are achieving their intended goals and objectives. Read more.

Utilisation-focused evaluation approach

The Utilisation-focused Evaluation approach is an effective tool for monitoring and evaluation users. It is a user-oriented approach that focuses on the utilisation of evaluation results by intended users and stakeholders. This approach encourages users to be actively involved in the evaluation process, from planning to implementation to reporting. It enables users to assess the impact of the evaluation results on their decision-making and practice. The Utilisation-focused Evaluation approach also encourages users to use the results for further improvement and refinement of their strategies and practices. This approach helps users to identify areas for improvement and to develop strategies to address them. In addition, it helps users to determine the most effective ways to use the evaluation results in order to achieve their desired outcomes. Read more.

Monitoring and Evaluation (M&E) for learning is an approach that prioritizes learning and program improvement, as opposed to solely focusing on accountability and reporting to external stakeholders. It is an iterative process that involves continuous monitoring, feedback, and reflection to enable learning and adaptation. By engaging stakeholders in the evaluation process, M&E for learning can identify strengths, weaknesses, and areas for improvement, and use this information to guide program design and implementation. Ultimately, the goal of M&E for learning is to create a culture of continuous learning within organizations, where learning and adaptation are integrated into every aspect of program design and implementation. Read more .

Gender Responsive Evaluation

A gender-responsive evaluation is an approach to understanding the impacts of a project, policy, or program on women, men and gender diverse populations. It is a valuable tool to assess how different gender groups are affected by a particular project, as well as how to ensure that the project meets its objectives in a way that is equitable and beneficial to all genders. Gender-responsive evaluations also provide useful information on how different gender groups interact and participate in projects or policies, which can help identify any potential inequities in access or outcomes.  Read more .

The case study evaluation approach is a powerful tool for monitoring and evaluating the success of a program or initiative. It allows researchers to look at the impact of a program from multiple perspectives, including the behavior of participants and the effectiveness of interventions. By using a case study evaluation approach, researchers can develop a comprehensive picture of the program’s strengths and weaknesses, identify areas for improvement, and make recommendations for future action. This approach is particularly useful for programs that involve multiple stakeholders, as it allows for the examination of both individual and collective outcomes. Furthermore, it is a valuable tool for assessing the program’s effectiveness over time, as it enables researchers to compare the results of different interventions and track changes in program outcomes. Read more.

This approach focuses on how a project is implemented, rather than the outcomes. It can help identify problems in project implementation, such as delays or budget overruns, and make recommendations for improvement. The process monitoring and evaluation approach is a systematic way of tracking and assessing the progress of a project or program. It involves regularly collecting, analyzing, and interpreting data to determine the effectiveness of a program and to identify areas for improvement. Monitoring and evaluation are two distinct but related functions. Monitoring is the continuous collection of information to track the progress of a program or project over time. Evaluation, on the other hand, is the periodic assessment of a program or project to determine its effectiveness and impact. The process monitoring and evaluation approach provides a comprehensive understanding of the program’s strengths and weaknesses, enabling decision-makers to make informed decisions about how to improve the program and ensure its success. Read more.

This approach involves assessing the causal impact of a project on its beneficiaries or the wider community. It can help determine whether a project has achieved its intended outcomes and whether the benefits outweigh the costs. The impact evaluation approach is a monitoring and evaluation technique used to assess the outcomes of a program or intervention. This approach helps to identify the changes that have occurred due to the program or intervention and measure the effectiveness of the program. It is used to evaluate the impact of the program on the target population, such as whether the program has achieved its desired objectives. The impact evaluation approach helps to identify areas of improvement and assess the cost-effectiveness of the program. It also helps to determine whether the program has met its goals and objectives, and if not, what changes should be made in order to achieve the desired results. This approach is a valuable tool for organizations to assess the success of their programs and interventions. Read more.

Evaluation Approaches versus Evaluation Methods

Evaluation approaches and evaluation methods are both used to assess the effectiveness and impact of programs, policies, or interventions. However, they refer to different aspects of the evaluation process.

Evaluation approaches refer to the overall framework or perspective that guides the evaluation. They define the philosophical, theoretical, and methodological principles that underpin the evaluation.

Evaluation methods, on the other hand, are the specific techniques and tools used to collect and analyze data to evaluate the program. Methods can be quantitative (e.g., surveys, experiments, statistical analysis) or qualitative (e.g., interviews, focus groups, content analysis), and may vary depending on the evaluation approach used.

In summary, evaluation approaches define the overall framework and principles that guide the evaluation, while evaluation methods are the specific techniques and tools used to collect and analyze data to evaluate the program.

An effective monitoring and evaluation approach can help to identify whether an organization’s goals are being achieved in a timely manner.

Overall, organizations can use one or more of these approaches to monitoring and evaluation, depending on the needs of their project and the resources available to them. Although there are many different types of monitoring and evaluation approaches available, they all share the same goal – to understand the impact of an organization’s programs and projects on its stakeholders.

This is so detailed and simple to understand. Thanks EvalCommunity for your contribution towards monitoring and evaluation. I always love your resources, thank you!

Khadar Mahad

I Comment is Only To Say You Thanks How To Prepare In This Lesson

Leave a Comment Cancel Reply

Your email address will not be published.

Login with your Social Account

How strong is my resume.

Only 2% of resumes land interviews.

Land a better, higher-paying career

Jobs for You

Business development associate.

• United States

Director of Finance and Administration

• Bosnia and Herzegovina

Request for Information – Collecting Information on Potential Partners for Local Works Evaluation

• Washington, USA

Principal Field Monitors

Technical expert (health, wash, nutrition, education, child protection, hiv/aids, supplies), survey expert, data analyst, team leader, usaid-bha performance evaluation consultant.

• International Rescue Committee

Manager II, Institutional Support Program Implementation

Senior human resources associate, energy and environment analyst – usaid bureau for latin america and the caribbean, intern- international project and proposal support, ispi, deputy chief of party, senior accounting associate, services you might be interested in, useful guides ....

How to Create a Strong Resume

Monitoring And Evaluation Specialist Resume

Resume Length for the International Development Sector

Types of Evaluation

Monitoring, Evaluation, Accountability, and Learning (MEAL)

LAND A JOB REFERRAL IN 2 WEEKS (NO ONLINE APPS!)

Sign Up & To Get My Free Referral Toolkit Now:

• Browse All Articles
• Newsletter Sign-Up

PerformanceEvaluation →

No results found in working knowledge.

• Were any results found in one of the other content buckets on the left?
• Try removing some search filters.
• Use different search filters.

A .gov website belongs to an official government organization in the United States.

A lock ( ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

• Public Health Counsel Partners
• About the Public Health Law Program
• Tribal Public Health
• Publications and Resources
• Public Health Law News
• Training and Educational Resources
• Technical Assistance
• Coroner and Medical Examiner Laws

Competencies for Legal Evaluation Studies

The Legal Epidemiology Competency Model includes three major domains: 1) general legal epidemiology competencies, 2) legal mapping, and 3) legal evaluation. Domain 3 focuses on legal evaluation including designing projects that study potential associations between health and law.

Additional information

• Figure 1—Domain 1: General Legal Epidemiology Competencies
• Figure 1—Domain 2: Competencies for Legal Mapping Studies
• Figure 2—Three Stages of Legal Epidemiology Career Development

Figure 1—Domain 3: Competencies for Legal Evaluation Studies

• Public Health Infrastructure Center
• Public Health Law Program

Public Health Law

The Public Health Law Program works to improve the health of the public by developing law-related tools and providing legal technical assistance to public health practitioners and policy makers in state, tribal, local, and territorial (STLT) jurisdictions.

Health Care Providers

Public health.

• Open access
• Published: 10 May 2024

Translating global evidence into local implementation through technical assistance: a realist evaluation of the Bloomberg philanthropies initiative for global Road safety

• Rachel Neill   ORCID: orcid.org/0000-0002-1110-5479 1 ,
• Angélica López Hernández 1 ,
• Adam D. Koon 1 &
• Abdulgafoor M. Bachani 1  

Globalization and Health volume  20 , Article number:  42 ( 2024 ) Cite this article

219 Accesses

Metrics details

Traffic-related crashes are a leading cause of premature death and disability. The safe systems approach is an evidence-informed set of innovations to reduce traffic-related injuries and deaths. First developed in Sweden, global health actors are adapting the model to improve road safety in low- and middle-income countries via technical assistance (TA) programs; however, there is little evidence on road safety TA across contexts. This study investigated how, why, and under what conditions technical assistance influenced evidence-informed road safety in Accra (Ghana), Bogotá (Colombia), and Mumbai (India), using a case study of the Bloomberg Philanthropies Initiative for Global Road Safety (BIGRS).

We conducted a realist evaluation with a multiple case study design to construct a program theory. Key informant interviews were conducted with 68 government officials, program staff, and other stakeholders. Documents were utilized to trace the evolution of the program. We used a retroductive analysis approach, drawing on the diffusion of innovation theory and guided by the context-mechanism-outcome approach to realist evaluation.

TA can improve road safety capabilities and increase the uptake of evidence-informed interventions. Hands-on capacity building tailored to specific implementation needs improved implementers’ understanding of new approaches. BIGRS generated novel, city-specific analytics that shifted the focus toward vulnerable road users. BIGRS and city officials launched pilots that brought evidence-informed approaches. This built confidence by demonstrating successful implementation and allowing government officials to gauge public perception. But pilots had to scale within existing city and national contexts. City champions, governance structures, existing political prioritization, and socio-cultural norms influenced scale-up.

The program theory emphasizes the interaction of trust, credibility, champions and their authority, governance structures, political prioritization, and the implement-ability of international evidence in creating the conditions for road safety change. BIGRS continues to be a vehicle for improving road safety at scale and developing coalitions that assist governments in fulfilling their role as stewards of population well-being. Our findings improve understanding of the complex role of TA in translating evidence-informed interventions to country-level implementation and emphasize the importance of context-sensitive TA to increase impact.

Road traffic crashes are the leading cause of death for persons aged 5–29 years age [ 1 ], and the 12th leading cause of deaths overall [ 2 ]. Road traffic mortality is three times higher in low-income countries than high-income countries (HICs), despite low-income countries having less than 1% of global motor vehicles [ 2 ]. Over half of traffic-related deaths are vulnerable road users (e.g., pedestrians, cyclists, and motorcyclists) [ 2 ].

Attention to road safety has grown, supported by evidence on the severity of the problem and solutions [ 3 ]. Successive ‘Decades of Action for Road Safety’ have raised awareness, and new institutions have improved policy cohesion and civil society mobilization [ 3 ]. The global road safety community has also cohered around a consensus-based solution – the safe system approach – developed in Sweden and increasingly applied globally. The safe system approach is a human-centered, proactive approach that shifts the focus of road safety from preventing crashes and improving road user behavior to preventing deaths and injuries while accounting for human error [ 4 ]. Despite global momentum, there is limited implementation of the safe system approach in low- and middle-income countries (LMICs) [ 3 , 5 , 6 ]. Global road safety programs emphasize the adaptation of the safe systems model to LMICs [ 5 , 7 ], even though the implementation context in LMICs varies significantly [ 8 ].

The role of technical assistance

Technical assistance (TA) is one way to increase the uptake of the safe system approach and other evidence-informed interventions. TA is a capacity-building process to design and/or improve the quality, effectiveness, and efficiency of programs and policies, [ 9 ]. Multi-country TA programs seek to translate the safe system approach to LMICs to reduce traffic-related injuries and mortality. The Bloomberg Philanthropies Initiative for Global Road Safety (BIGRS) is one of the largest and longest-standing multi-country road safety TA programs. This analysis concerns BIGRS Phase II, which provided a common package of TA interventions to ten LMIC city governments from 2014 to 2019. By the end of Phase II, cities differed considerably on the scale and scope of implementation.

BIGRS’ differential experiences across LMIC cities present an empirical case study on the feasibility of adapting common technical approaches across divergent contexts and the TA’s role. How much influence does TA have? What is the role of context in shaping TA providers’ and recipients’ agency?

A diverse body of scholarship concerns these questions and can guide empirical inquiry. Diffusion of innovation theory describes the process of transferring an evidence-informed intervention from one setting to another [ 10 , 11 ] and has been used to explore TA effectiveness [ 9 ]. Diffusion of innovation theory focuses on intervention characteristics, intervention adaptation, and how adaptation influences adoption and fidelity [ 10 , 11 , 12 ]. Greenhalgh’s Determinants of Diffusion, Dissemination, and Implementation of Innovations in Health Service Delivery and Organization Conceptual Model builds on diffusion of innovation theory by mapping considerations that influence the uptake of innovations. These include credibility, personal relationships, effective communication,translation of the innovation to meet end-users needs, and support to adopters [ 12 ]. More broadly, social science theories consider the role of structural context (e.g., laws, social norms, and governance) and pragmatic implementation contexts (e.g., individuals, relationships, and organizational cultures) in determining adaptation, and implementation [ 13 ]. These literature bring different perspectives to explain change through the interaction of interventions, actors, and context.

However, there is limited application of this literature to understand TA, especially road safety TA. A growing body of case studies describes what works and does not work for improving road safety in LMICs [ 14 , 15 , 16 ]. Limited research emphasizes political will, intervention tailoring, human and financial resources for dissemination [ 17 ], the best practice exchange [ 18 ], technology transfer [ 19 ], and the power of multi-sectoral coalitions [ 20 ] to translate road safety evidence into practice. However, despite the existence of several multi-million-dollar road safety TA and funding programs [ 21 , 22 , 23 , 24 , 25 ], we did not identify any evidence on the role of TA in supporting or inhibiting road safety improvements – a key evidence gap.

Study objective

This study aims to improve understanding of if, how, why, and under what conditions TA programs strengthen evidence-informed road safety programs in LMICs. We do this via a realist evaluation with a multiple case study design of BIGRS’ implementation, comparing how common TA interventions interacted with contextual factors to produce differential observable outcomes in Accra, Bogotá, and Mumbai. These findings are distilled into a program theory that provides insight into how ‘global’ approaches are translated to country-level implementation and can be used to guide TA’s design and implementation.

Realist evaluation connects theories of ‘how the world works’ with ‘how a program works’ to explain how interventions trigger mechanisms in different contexts [ 26 ]. We used a realist evaluation methodology [ 27 ] to identify how, why, and under what conditions TA can strengthen evidence-informed road safety, with a multiple case study design to improve understanding of how BIGRS worked in diverse contexts [ 28 , 29 ] [ 26 ]. This methodology was selected to identify the underlying mechanisms driving the program’s differential outcomes in different contexts [ 27 ].

• Realist evaluation

Programs are theories about how something works. They are embedded into open systems and adaptively interact with the context. Intervention outcomes result from engagement between program actors and contexts [ 27 ]. An intervention-context-mechanism-outcome pattern (ICMO) represents this [ 27 ] (Table  1 ).

We adhered to the Realist and Meta-narrative Evidence Syntheses: Evolving Standards (RAMSES) II reporting guidelines for realist evaluation to guide design, data collection, and analysis [ 26 ], provided in Additional File 1 . The study protocol is in Additional File 2 .

Study setting

Bloomberg Philanthropies’ BIGRS Phase Two was implemented in Accra, Addis Ababa, Bandung, Bangkok, Bogotá, Fortaleza, Mumbai, Ho Chi Min, Sao Paulo, and Shanghai from 2014 to 2019 and is the focus of this study. BIGRS is currently in its third phase and has scaled up to 27 cities and two states across Latin America, sub-Saharan Africa, and Asia.

Cities applied for BIGRS-supported TA by submitting a proposal that demonstrated their commitment to and plans for road safety. This is important because it meant that cities demonstrated a common commitment and desired TA, at least in theory. Funding for interventions (e.g., re-designing an intersection or mass media campaigns) came from city governments.

BIGRS’ TA came with a technical agenda – aligned to the safe system approach – on how road safety should be improved. BIGRS’ scope was tailored to city needs within pre-existing parameters and excluding funding for capital construction. To provide TA, BIGRS seconded staff into leading road safety agencies to build institutional capacity for change. Embedded staff supported BIGRS interventions, provided direct TA to city counterparts, and often, provided cross-cutting support to city officials. In addition, seven international partner organizations managed technical activities. Partners and embedded staff were aligned to technical areas and often worked with different counterparts (e.g., an enforcement partner working with the police, an infrastructure partner working with an engineering unit).

Case design and sampling

A multiple case study design was utilized; see Additional file 3 for details. Only cities continuing into BIGRS Phase Three were eligible for selection to ensure access to informants. We purposefully selected three cities – Accra, Ghana; Bogotá, Colombia; and Mumbai, India – with different baseline characteristics described in Table  2 .

Data collection

Key informant interviews (KIIs) were our primary data source, and documents were secondary. Program documents were used to build an initial program theory, develop the interview guides, and follow ‘hunches’ about how an intervention worked in a context [ 35 , 36 ]. We also snowballed documents from interviews to confirm and triangulate interview findings.

Key informant interviews

We used a theoretical sampling approach to select informants based on their ICMO potential [ 27 ]. We iteratively sampled informants until saturation – when interviews provided no new insights [ 37 ]. Table  3 describes the KII sample. Road safety governance models influenced the balance of KII types. Road safety governance in Accra and Mumbai is more diffused than Bogotá, which had fewer government informants. KIIs also varied across cities due to differential access to informants. To overcome this disadvantage, we triangulated findings with the document review.

Interviews were conducted from January 2020 to November 2022 by two members of the research team with doctoral-level training in qualitative methods. Participants were contacted via email and invited to a one-hour interview on barriers and enablers to BIGRS and mechanisms associated with program outcomes. Interviews in Mumbai were conducted via Zoom. Interviews in Bogotá and Accra were conducted in-person and on Zoom.

A realist approach to interviewing was used to build an iterative understanding of how the program worked, test our interpretations, and seek alternative explanations (Additional file 3 ). Interviews were recorded and transcribed with permission. Fifty-one interviews were conducted in English, recorded, and transcribed. Seventeen interviews were conducted in Spanish by a native speaker, recorded, transcribed, and translated into English by a certified translator.

Data collection and analysis were done iteratively using a process of retroduction [ 35 , 36 , 38 ] (Fig.  1 ).

Iterative analysis process

As is common in this literature [ 39 ], BIGRS’ theory of change (TOC) was used as the initial program theory (IPT). Following guidance on realist evaluation analysis [ 38 ], we iteratively identified ICMOs and compared them to the IPT and broader literature to develop the program theory. This included an initial thematic coding of the data, a second round of theory refinement coding where themes were split into ICMOs, triangulation of findings from the documents and interviews, and comparison of the findings with existing theory to deepen our understanding of plausible mechanisms. Additional file 3 describes this process in more detail. We conducted this analysis in NVivo12.

Regular discussions were held across the research team to define and iterate on the codebook, discuss emergent themes, and review ICMO configurations. Memos were developed in Microsoft Word and documented ICMO iterations. Draft findings were shared with a subset of the participants for feedback and validation before program theory finalization.

In 2014, BIGRS initiated a common TA program in Accra, Bogotá, and Mumbai. TA interventions, individuals providing and receiving TA, the city context, and the national-level road safety context influenced implementation. Table  4 outlines interventions and outcomes. Interventions are grouped under two outcomes: (1) improved road safety capabilities (via capacity building and data) and (2) increased the uptake of evidence-informed road safety interventions (via infrastructure, enforcement, and policy support).

We present one example per case that demonstrates how different interventions worked together to achieve different outcomes in case study cities. Interventions (i), mechanisms (m), contexts (c), and outcomes (o) are denoted in the text. Reference to KII data is provided as M# for Mumbai, B# for Bogotá, A# for Accra, and G# for KIs working across multiple cities. 

Transforming junctions on Mumbai’s congested streets

When BIGRS began, Mumbai’s road safety officials used high-level figures on traffic fatalities supplemented with national or state-level statistics to guide road safety decision making (c) . The city-specific data required to target road safety interventions was buried in paper-based police records of variable quality (c) . A government official in Mumbai describes:

" That is [a] very big [problem] because we are not like other countries, we are not getting the data correctly.” – M17

In response, BIGRS’ TA first sought to improve surveillance data. An embedded surveillance coordinator partnered with the police to catalog and analyze city surveillance data and package it in new annual city road safety reports (i) [ 40 ]. Infrastructure assessments (i) further demonstrated how specific road junctions contributed to injuries and mortality (o). BIGRS staff described the change in data availability:

“[Before] there were no reports at all […]. Now I have […] a 40-page report that talks about who the road users are, […] a list of high-risk junctions and corridors […], a map that details the hotspots where crashes are occurring […] which vehicle is causing maximum crashes, […] the time of the day, the month of the year, the day of the week.” – M8

New data demonstrated that half of traffic-related crash victims were pedestrians, which was further reported by local city media [ 41 , 42 ]. Providing granular, city-specific data shifted the focus (m) of government towards pedestrian safety (M8, M12, M17). The same BIGRS staff member described:

“The government didn't know that so many pedestrians were dying in crashes. These reports help bring that to light. And when that came to light, they started taking a more serious approach.” – M8

The emphasis on pedestrians was echoed by city government officials, who agreed that the data was illuminating (M12, M17). But the city also required solutions for this perception shift to lead to concrete action. The same government official describes the challenges (c) :

“We lack the best instrument in the old system to make the road elevated, or a road underpass is very difficult because the traffic on that high main road. […] That is very critical because it we are really facing problems.” – M17

BIGRS provided the safe system approach as a solution– but how would it work in Mumbai? This question was a central concern in all cities, especially in densely populated Mumbai, where participants described the street as a ‘contested space’ (M9, M2, M21) (c) . Implementing the safe system approach in Mumbai required a complex adaptation process (G5, M2, M9, G6, M7) (i) . A BIGRS infrastructure partner described:

“We’re constantly trying to balance Global Best Practices versus what can be done in an Indian city while pushing boundaries to be able to think outside the box. […] it's helpful to show International Best Practices, but also at the same time, balancing it out with what’s actually possible in Indian cities.” – M9

Implement-ability was top of mind for city government officials taking risks by trying a new approach (c) . BIGRS staff had to recognize those risks and work collaboratively with city government counterparts to understand how international approaches could work within local realities (B15, M13) (c) . A BIGRS staff member described this:

“[When you introduce international examples], there are a lot of questions and pushback saying, ‘how could this be done [here]? That was also instructive to us. How do you deal with such situations?” – M13

Short-term demonstration projects – for example, temporarily changing traffic flow using cones and other local, low-cost materials (i) – allowed city government counterparts to see the safe system approach in action on their street in a low-risk context, demonstrating that a new approach was possible (B15, M11, G7, M9, B7, M17). A city official describes:

“People are generally not aware of the things [happening outside India], [but] the problems are same. […] That can be taken only if you can show them the models […] because firsthand information from those people is much more important.” – M17

Seeing the possibility of change was perceived to shift the focus of road safety towards vulnerable road users and especially pedestrians (M18B, B15, M11, G7, M9, M15, M18, M17, M2, G2, M9, M7, M14, M8) (m) . It also created a ‘how-to’ moment, enabling city government counterparts to internalize both the concept and implementation feasibility (m). A city official describes what he learned:

“[I learned] new technical things, that might be there's been a certain technical change in junction design or in the road design. […] we were not able to do that thing nicely already […] We were able to grab that opportunity properly.” – M18

Once city government counterparts understood the potential of the safe system approach (o) , BIGRS TA worked with city officials to use the data and select specific junctions for re-design (i) . Data was critical because it helped target infrastructure improvement to junctions with an outsized number of crashes. A BIGRS staff described:

“Now [Mumbai city government are] not just randomly doing the interventions. They're very focused on where crashes are occurring, who the victims are, who the perpetrators are, and how to ensure that these crashes don't occur at all.” – M8

Once the mechanism for change was triggered, transforming junctions started with a pilot (i). Pilots ensured that the safe systems approach was feasible and appropriately adapted to the context, that its impact on different road users was understood and planned for, and that the re-design successfully reduced crashes [ 43 , 44 , 45 ]. Pilots also allowed the city government to understand public sentiment about the changes (c) [ 43 ] . If the public was supportive, this reduced the risk to city officials trying a new approach (m) . The city government counterpart’s confidence in new approaches grew (m). This was further reinforced by data showing that infrastructure redesign positively impacted traffic flow [ 43 ]. A pilot’s success was described as leading to exponential growth in implementation (M21, M1) [ 44 , 46 , 47 ]. A BIGRS partner described:

“If you see our work, it has exponentially grown in impact. […] From that one [pilot] corridor, we […] build a relationship and trust […] and so we got a chance to do design in the intersection. Then you try it with temporary sort of barricades and then it became a big thing. And then one thing just kept leading to another to another.” – M21

As implementation took off, BIGRS engaged local media to spread awareness about the junction transformations (i) (M11). After seeing firsthand what could be accomplished, the city government also committed to improving high-risk intersections in the city. However, despite growing momentum and support from both city government and city engineers to transform individual junctions, the bottom-up pilot approach presented practical scale and sustainability challenges despite this government commitment (M9, M13, documents) (c). A BIGRs partner described:

“It's a challenge at times, when the city does not have the funds allocated in that year. If you do manage a successful pilot and the city takes on doing it, then it's great because they can be scaled up. But in many cases […] pilots are sort of left as just that.” – M9

BIGRS partners described city government approvals as challenges preventing scale. In contrast, city government participants urged respect for government processes and timelines, which they saw as paramount to success (c). In managing these processes, city officials also took on significant work to enable each infrastructure re-design (M12, M18) – a contribution that often went unacknowledged (c).

Comparison of Mumbai’s infrastructure experience with other BIGRS cities

The Mumbai infrastructure example is illustrative of common dynamics. In Bogotá, capacity building was similarly perceived as successful when it used hands-on components specifically relevant to the participants (B15, G2, B9, B7, B14), and when facilitators used a coaching model that emphasized the participant’s experience (B15, B5, B9, G2).

Accra’s and Bogotá’s infrastructure TA were also targeted at bottom-up approaches (G6, A9, A5, A6, A8, M9) and guided by city-specific data (i ), but with limited scale. In Accra, BIGRS focused on low- or no-cost interventions (e.g, changing signal times for pedestrian crosswalks, widening pedestrian medians (i) ) (A5, A6, A8) because the city did not control the infrastructure budget and could not budget for new interventions. BIGRS also worked with the city to re-design the infamously dangerous Lapaz intersection to improve pedestrian safety, which was funded directly by BIGRS via a small grants program (i) [ 48 ]. In Bogotá, tactical urbanism demonstrated speed-calming measures, and feedback from road users was gathered (i) (B2). However, despite promising pilots, the lack of BIGRS’ ability to influence upstream changes to road procurement tenders and design guidelines limited the scale of infrastructure outcomes in each city.

Enforcing road safety legislation in Accra

In Accra, road safety legislation existed but needed to be enforced (c). In the words of a national road safety agency staff , “there’s no real commitment in solving some of these things.” (A17). BIGRS’ enforcement interventions started with relationship building (i). A BIGRS partner describes:

“How important it is to have this relationship with the high-level police officers. Because we cannot just go to a city or road police agency and say that this is what we want to do.” – A22

Trainings on the safe systems approach (i) and evidence-based enforcement operations were enabled by leadership support from the Superintendent of Police (A4) and the Mayor of Accra who championed road safety and several BIGRS initiatives (A4, A6, A3, A5, A8, A1, program documents) [ 48 , 49 ] (c) .

However, translating training into implementation quickly stalled because the police force required equipment and certification for implementing enforcement operations (c) (A1, A4, A6, program documents) . BIGRS’ partners then donated new drink-driving and speed enforcement equipment under the condition that the city utilized the equipment to conduct enforcement pilots (i) . These donations were accompanied by training and certification processes (A1, A4, A25, program documents) (c) .

While the lack of equipment could be directly addressed by BIGRS, the disconnect between city-level enforcement efforts and Ghana’s centralized policing structure could not be so easily overcome. City police did not have the authority to conduct enforcement operations (c) , so in exchange for the donated equipment, the police formed a dedicated tri-partite pilot task force with the authority to use the donated equipment in a series of roadside speed and drink-driving enforcement operations  (i) .

New training and improved accuracy of the equipment were perceived to reduce conflict between police and citizens during enforcement and improved transparency in the enforcement operations (A1, A4, A25, program documents), reducing the perception of risk of public blowback (m). A high-ranking police officer describes the perceived increase in acceptability from the public:

“they don’t complain, they go to the court […] because you’ve told us that the device arefor enforcement operations (A4, A25) limiting further […] the very latest speed device, speed detection devices [equipment] because we’ve told the whole world about it.” – A25

The collective intervention – piloting the enforcement approach, supported by training and in tandem with appropriate equipment – was also received positively by the police. A senior police officer described a shift in focus towards ensuring road safety (m):

“What I’ve realised is, what a positive impact on our capability to be able to ensure road safety. [..]. With the devices, we can go to the route when they see us, all cars, cars you know approaching the robot, reduce their speed and that has really resulted in a lot of improvement.” – A25

However, while the pilot taskforce did conduct enforcement operations, a series of upstream barriers prevented the taskforce from scaling up. Most practically, the police force still lacked dedicated vehicles for enforcement operations (A4, A25) limiting further implementation. More broadly, the social and political context (A1, A4) (c) remained unconducive to enforcement. A BIGRS staff describes:

“During [the enforcement pilot], we did a special round of data collection for speed, and the data showed that there was a reduction in speed. However, the moment could not be sustained. Some of the feedback they got from the police was that [the] police could not boldly or fearlessly enforce.” – A1

Another challenge was that the required authority to change enforcement practices was vested in national agencies instead of the city government, limiting the ability of city police to institutionalize new enforcement operations (A4, program documents) (c). Finally, the transfer of police was described as a challenge to sustaining enforcement interventions (A4, A2, program documents) (c). A BIGRS partner described:

“We can work with person, everything agreed, and then just before we roll out, he's been transferred or there's a rotation, and we have to change everything.” — A22

Comparison of Accra’s enforcement experience with other BIGRS cities

Across the enforcement TA provided in the three cities, building trust with senior police officers was repeatedly emphasized (A4, B4, M16, G7, G10). Using former senior police officers from other countries was seen as key to building that neccessary trust (B4, M16, G7, G10) (c).

Like Bogotá, Accra’s enforcement interventions took place within broader city road safety prioritization (c), and BIGRS donations ensured police had the right equipment (i) , leading to increased enforcement (B8, G12, A4, A1, A22, A25) (o) . However, Bogotá’s enforcement was described as widespread and sustained (B8, G12), while in Accra, enforcement operations remained limited (o). The authority of the police to conduct enforcement was the key difference (c).

In contrast, India was moving towards an automated speed enforcement model, which contrasted with the model proposed by BIGRS (c) . Although automated and roadside enforcement co-exist (and they did in Bogotá), BIGRS’ roadside enforcement model did not align to the broader policy agenda in Mumbai and was not implemented.

Reducing city-wide speed limits in Bogotá – an example of policy change

Before BIGRS, improved mobility had been the focus of several consecutive city administrations (c) . A BIGRS partner described the favorable baseline environment:

“Bogotá has been concerned about road safety for a long time. [Bogotá] already had a Road Safety Directorate; […] there was already a direction with a super great team. It was easy to work in Bogotá because institutionally, they were already armed.” – B13

During Phase II, a new Secretary of Mobility with a public health background further elevated road safety in city administration (c)  which was perceived as critical to the city's subsequent policy change (B1, B11, B12, B13, B14, B15, B16). A city official explained:

“ It is about setting priorities. So, we [the secretariat], from the first day, said the priority is road safety, and we will do everything possible to make it so.” – Bogotá 2

Alongside a change in government, BIGRS also hired new embedded staff, some who were former members of city government, all who were local to Bogotá, and all who were passionately committed to improving road safety (c) (B12). However, support for road safety did not immediately translate to speed. Instead, city officials were interested in reducing drunk driving and were explicitly resistant to tackling speed ( c) . This was due both to concerns that reducing speed would increase traffic and also a perceived lack of concern from the population over speed (G12, G5, B8, B12). A BIGRS staff recalled:

“Even when communicating to [the] Mayor, he had the issue of road safety in his heart the main thing he communicated and did not want to do. ‘Do not slow down on arterial roads’” – B12

However, BIGRS’ analyses of city data (i) identified that speed was a serious concern on arterial roads at night (B8, B12, G5, G8). A BIGRS partner in Bogotá described:

“The first thing I did was share with the Police the data that clearly showed that most of the deaths occurred at night or early in the morning when most roads were empty.” – B8

This was further demonstrated by a modeling study (i) showing both the relationship between speed and the crash rate and that the change in speed limits would not impact average travel times. This study was important evidence, which was only possible because the city’s existing speed detection infrastructure provided the modeling data (B12) (c).

The presentation of this novel information to city officials was perceived to shift the focus of city officials by demonstrating that speeding was prevalent at night when roads were empty, and that reducing speed wouldn’t worsen traffic (m) . City officials used this data to select five arterial road corridors with high speeds, crashes, and deaths to pilot a reduced speed limit of 50 km per hour (kmph) (o and i) .

The speed reduction pilot required close collaboration between the Secretariat of Mobility and the police to conduct nighttime enforcement (c). However, the police lacked necessary nighttime radar equipment (c) (B8, G12), a gap subsequently filled by BIGRS’ donations (i) . TA was provided for the police to use the equipment and to conduct safe nighttime operations (i), increasing enforcement campaigns in the pilot speed management corridors (o). A BIGRS staff described:

“ [It] was clear when you make enforcement operations visible, like speed enforcement down that avenue. In a matter of months, we already saw a reduction [in speed].” – G12

The new roadside enforcement was complemented by automated speed detection cameras ( c) ; however, the public was skeptical of the speed cameras' threatening the pilot’s success (c). Public messaging campaigns were therefore developed using city data to demonstrate the rationale behind the speed reduction and enforcement ( i ). A BIGRS staff described:

“Legitimacy has to do with road users' acceptance of this type of control. […] What decisions were made? Make visible the places where photodetection cameras are installed. They were published on the website of the Ministry of Mobility, and there was a strong media drive to make these cameras visible and associate the cameras with the issue of life-saving cameras.” –B8

BIGRS also provided monitoring and evaluation support (i) which quickly demonstrating the pilot’s effectiveness (o). A BIGRS staff described:

“In a matter of months, I already saw a reduction [in deaths]. That gave the Secretary of Mobility the confidence, trust like, ‘OK, like this is working, we are reducing deaths where we are not messing up traffic. Let's do it.’” – G12

A city official recalled the importance of the pilots:

“Yes, yes, yes, that was very well done. The expressive power of those corridors, of the first ones” – B14

Because of the positive pilot results, the city increased the number of corridors with lowered speed limits (o). The results of the pilot were also shared with the public, reinforcing the message that the speed reduction corridors were lifesaving interventions (G12, G5) (i) and further reducing the perception of risk in lowering the speed limit by building public support (m). As the pilot gained increased support, city counterparts used the data to develop a technical document justifying the lowered speed limits to Bogotá’s city council. A BIGRS staff described:

“To be able to argue before the City Council, it was necessary to argue with objective judgment elements […] Why did they decide to slow down? Not because it occurred to us. No, the speed was lowered because this technical document allows us to support making that decision.” – B8

Aided by the pilot’s success and with the support built through public messaging campaigns, the city council maintained the 50 km/h speed limit on the pilot corridors (o). However, the city council initially did not have the authority to change city-wide speed limits permanently (c), preventing scale-up until a window of opportunity opened in 2020. During the 2020 COVID-19 pandemic, a state of emergency was declared, giving temporary executive authority to the Mayor (c) . Although the Secretary of Mobility (the champion of the pilot) had changed, their successor became a new champion. They successfully argued that the speed limit reduction was preventing traffic crashes, thereby reducing non-COVID-19 health emergencies and freeing up healthcare capacity during the pandemic. This allowed the Mayor to extend the speed reductions city-wide in alignment with the WHO’s advised 50 km/h (o).

Reflecting on Bogotá’s experience with BIGRS, a city official described how BIGRS’ comprehensive TA approach was important in supporting the city’s road safety vision:

“We wanted to build how this systemic vision of approaching the problem. And then Bloomberg supported us with communications, technical, infrastructure, traffic calming, and enforcement issues.” – B2

City officials and BIGRS staff alike credited city leadership for continuously supporting road safety throughout several administrations and for giving political support to technical staff who brought changes to the city (c). One government official commented:

“ Everyone, I think, without exception, has supported this work. I believe that the first requirement to choose a city is that there is willingness. What has been in Bogotá, really, is the political will of the leaders to carry it out. Without it, you do nothing .” – B14

Comparison of Bogotá’s policy experience with other BIGRS cities

The scale of change in Bogotá’s road safety programming stands apart from the other case studies. Second to this was Accra; the city government formed a new road safety council and developed the city's first Pedestrian Action Plan (o). Like Bogotá, BIGRS in Accra leveraged city prioritization for road safety and provided city-specific evidence (i), which focused city stakeholders’ efforts on the importance of pedestrian safety (A8, A1, A5, A3, program documents). Also, like Bogotá, the Mayor was a champion who lent convening power to the development of Accra’s action plan (4, A6, A3, A5, A8, A1, program documents) (c) . The Accra Pedestrian Action Plan was further perceived to improve coordination of different road stakeholders towards a common goal (A8, A1, A5, A3, A6, program documents).

In Mumbai, in contrast, BIGRS staff and partners described a lack of an individual champion with the authority to advance road safety policy and planning at the city level as a key challenge (M10, M11, M12, M13, M14, M15, M21).

Revised program theory

The revised program theory for BIGRS should be considered an initial attempt to synthesize across both positive cases (where outcomes were observed) and negative cases (where outcomes were limited by specific factors) to distill a set of higher-level statements about how BIGRS works at the city level and the contexts that enable or constrain its success.

The first program theory is improved road safety capabilities, focused on capacity and data use interventions described by BIGRS staff and partners as precursors to implementation in each case study city.

Program theory for improved road safety capabilities:

Providing TA to increase capacity and data use (i), if delivered via trusted and credible TA providers who provide hands-on coaching support tailored to city needs and with counterparts interested in engaging with road safety, can strengthen road safety capabilities (o) because it shifts the focus of city officials towards evidence-informed approaches and creates a how-to moment to improve road safety through the safe system approach (m). This outcome is enabled by city prioritization of road safety (c) and can be disrupted if city government officials change (c) .

The second program theory is increasing the uptake of evidence-informed implementation of road safety interventions. In this theory, capacity building and data now comprise the necessary context that supports the interventions, and BIGRS and city officials are characterized as working together to implement.

Program theory for increasing the uptake of evidence-informed implementation of road safety interventions:

If trusted and credible TA providers, working with and through city champions (c), undertake a successful pilot (i), guided by city-specific data that targets interventions (c) , and with facilitation of city implementation via dedicated equipment, training and other supportive resources (i) , then this can increase the uptake of evidence-informed road safety interventions (o). This occurs because a pilot builds confidence that the safe systems approach is feasible in a specific road context (m) , and it reduces the perception of risk in adopting a new approach (m) by allowing city officials to gauge public sentiment. The scale and sustainability of the outcome(s) are determined by the city’s existing prioritization of road safety, the authority of the individuals and road safety agencies targeted in the intervention, and existing socio-cultural norms (c) . It can be disrupted if city government officials change (c) .

BIGRS’ interventions sought to accelerate cities’ adoption of the safe system approach. What united city officials were two questions – will it work here, and how? To answer those questions, TA needed to go beyond recommending that a safe system approach would work, to demonstrating how it could work, to prove that it worked (without provoking negative reactions from the public).

How did TA work?

TA provider credibility and ability to navigate the city context were important. This was demonstrated by embedded staff who continuously connected the evidence-base and resources of international partners with the tacit knowledge and goals of city agencies. By playing a dual ‘insider-outsider’ role, embedded staff worked to create a favorable context for interventions and made interventions a better fit for the context. This describes the role of boundary-spanners who bridge insider and outsider roles to facilitate the adoption of an intervention [ 12 ].

How TA was provided was also essential. TA providers needed to understand the context and work effectively within it, not against it. Capacity-building activities needed to follow a coaching model, amplifying the existing knowledge, needs, and priorities of decision-makers. Interventions needed to be immediately relevant to the context, or TA providers risked losing credibility. BIGRS embedded staff and partners based full-time in the city again had the advantage here. This finding aligns with calls for TA to be context-sensitive [ 50 , 51 ] and aligns with the characteristics of successful change agents [ 12 ].

Why did TA work (or not work)?

The mechanism ‘shifting the focus’ was about data. Aligning with diffusion of innovation theory, data framed a ‘felt need’ for change [ 52 ] in all cities to different degrees. Bogotá was an early adopter; new data was presented within the context of political commitment to road safety, and pre-existing automated enforcement infrastructure enabled BIGRS to develop data-driven machine learning models to predict the results of the speed enforcement pilot. In Mumbai, in comparision, most of BIGRS’ Phase II activities focused on building city data capabilities to catalyze this shift in focus. ‘Shifting the focus’ was further enhanced by city officials’ ability to establish fora for governing the use of data to support policy decisions, consistent with international norms [ 53 ].

But ‘shifting the focus’ was also directly facilitated by BIGRS, making it the most uncertain mechanism. An alternative conceptualization is that BIGRS ‘shifted the focus’ by dedicating resources to specific interventions, informed by its data, which the city endorsed.

The second mechanism, creating a ‘how-to moment’, comes from diffusion of innovation’s knowledge phase [ 52 ]. Adopters must understand how an innovation works, especially if the innovation is complex [ 52 ]. Pilots allowed officials to see change in action, built confidence, and reduced the risk of stakeholder discontent from changing the road environment [ 12 , 52 ]. BIGRS also had an advantage; infrastructure re-design and enforcement are trial-able approaches with quickly observable outcomes which supports innovation adoption [ 10 , 12 ].

Under what conditions did TA work (or not work)?

Moving from the first program theory outcome (‘strengthened road safety capabilities’) to the second (‘increasing evidence-informed interventions’) required more than triggering individual-level mechanisms. To change implementation, individual-level mechanisms had to translate into institutional actions by city officials– e.g., approving pilots, allocating resources, and implementing interventions. It was here that context was critical.

City champions were key to enabling change. Champions are important in diffusion of innovation theory [ 12 ] and were critical here. However, following structure-agent theory, city champions could only change areas within their control [ 54 ] and their agency varied. Comparing Bogotá and Accra is instructive. Bogotá had considerable latitude to change road safety practices, while Accra’s pilot task force failed to scale due to limited institutional and normative authority to enforce legislation. Officials in road safety agencies lamented this alongside BIGRS staff, suggesting that the interventions were compatible with the context [ 52 ] but that the city's agency was constrained.

Structural, or outer, contexts therefore determined the feasibility of converting individual and city level mechanisms into outcomes. Diffusion of innovation theory considers that innovation may not be ‘compatible’ with the context or that the system may not be ‘ready’ for change, which was important in these cases. But more important, however, was how the innovation was introduced, who introduced it, the city's priorities, and city’s authority to adopt the innovation. This points to a critical consideration – if the dissemination approach of how the innovation is introduced is incompatible with structural context, adoption will be slow or unsuccessful (even if the innovation itself fits the context).

Boundary spanning – crossing boundaries to negotiate interactions and translate knowledge from different settings [ 55 ] – is one way to bridge the gap between proposed solutions and local contexts. A 2017 multi-county nutrition project found that boundary spanning was feasible and useful to navigate context-specific challenges [ 56 ]. Our study suggests that boundary spanning – if those doing the boundary spanning are deeply embedded within the local context – could be a useful model for delivering TA. Engaging boundary spanners from the beginning to work with city government officials to design TA programs around local problems and priorities, rather than providing both with a model from elsewhere to adapt, is a practical way to design more context-sensitive TA and surface local innovations [ 13 ].

Strengths and limitations

The goal of this study was to learn from implementation experience and develop a program theory. We did not quantitatively measure outcomes, a limitation. To improve trustworthiness, we triangulated findings across cities and data sources. However, outcomes were mainly validated with informants due to a lack of access to documents across BIGRS partners, creating some uncertainty. Another limitation was the overrepresentation of BIGRS staff and partners in our sample as compared to government officials and other city stakeholders. The reasons for this were both practical – e.g., scheduling interviews over Zoom, governance differences across cities – and representative of broader findings – government official turnover limited available informants. Finally, several authors (but not the first author) were involved in BIGRS’ implementation, which required continual bracketing when analyzing the data.

Our multiple case study design was a strength, enabling ICMO comparison across cities, reducing uncertainty, and increasing confidence. Iterative data collection and validation of the program theory with participants further reduced uncertainty because we could discuss uncertainties with participants and dig deeper. We also verified our interpretations with documents.

We identified broadly applicable insights into the role of TA in strengthening evidence-informed road safety in LMICs and distilled these into a program theory, contributing to knowledge on multisectoral TA programs in global health. Our study is the first we know of to empirically analyze the role of TA in influencing road safety in LMICs. BIGRS’ program theory emphasizes the interaction of trust, credibility, champions and their authority, governance structures, political prioritization, and the implement-ability of evidence in creating the conditions for road safety change. Designing context-specific TA appropriate for structural contexts is critical. If decision makers prioritize road safety, TA can accompany local leaders in adapting international approaches to local realities. In this way, we see cross-country multisectoral projects as important opportunities to improve population health.

Availability of data and materials

Data generated and analyzed during this study are included in this article. Key informants were assured that the raw transcripts would not be shared.

Institute for Health Metrics and Evaluation. GBD Compare [Internet]. University of Washington. 2019 [cited 2023 Feb 10]. Available from: https://vizhub.healthdata.org/gbd-compare/#

World Health Organisation. Global status report on Road safety 2018 [internet]. Geneva: World Health Organization; 2018. Available from: http://apps.who.int/bookorders

Google Scholar  

Hyder AA, Hoe C, Hijar M, Peden M. The political and social contexts of global road safety: challenges for the next decade. Lancet. 2022;400:127–36.

Article   PubMed   Google Scholar  

Demystifying the safe system approach [internet]. Vision zero Network. 2023 [cited 2023 Feb 11]. Available from: https://visionzeronetwork.org/resources/demystifying-the-safe-system-approach/

Haghani M, Behnood A, Dixit V, Oviedo-Trespalacios O. Road safety research in the context of low- and middle-income countries: macro-scale literature analyses, trends, knowledge gaps and challenges. Saf Sci. 2022;146:105513.

Article   Google Scholar  

Shuey R, Mooren L, King M. Road safety lessons to learn from low and middle-income countries. Journal of Road Safety. 2020;31:69–78.

Peden MM, Puvanachandra P. Looking back on 10 years of global road safety. Int Health. 2019;11:327–30.

Soames Job RF, Wambulwa WM. Features of low-income and middle-income countries making Road safety more challenging. Journal of Road Safety. 2020;31:79–84.

West GR, Clapp SP, Averill EMD, Cates W Jr. Defining and assessing evidence for the effectiveness of technical assistance in furthering global health. Glob Public Health. 2012;7:915–30.

Article   PubMed   PubMed Central   Google Scholar  

Dearing JW. Applying diffusion of innovation theory to intervention development. Res Soc Work Pract. 2009;19:503–18.

Dearing JW, Cox JG. Diffusion of innovations theory, principles, and practice. Health Aff. 2018;37:183–90.

Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004;82:581–629.

Olivier de Sardan J-P, Diarra A, Moha M. Travelling models and the challenge of pragmatic contexts and practical norms: the case of maternal health. Health Res Policy Syst. 2017;15:60.

UN Road Safety Fund. Open day knowledge kit. New York; 2023.

International transport forum. The safe system approach in action. Paris; 2022.

Book   Google Scholar  

Turner B, Job S, Mitra S. Guide for Road safety interventions: evidence of what works and what does not work. Washington; 2021.

Sleet D, Baldwin G. Lost in translation: translating injury research into effective interventions. J Australas Coll Road Saf. 2010;

LaJeunesse S, Heiny S, Evenson KR, Fiedler LM, Cooper JF. Diffusing innovative road safety practice: a social network approach to identifying opinion leading U.S. cities. Traffic Inj Prev. 2018;19:832–7.

Knapp K, Walker D, Wilson E. Challenges and strategies for local Road safety training and technology transfer. Transportation Research Record: Journal of the Transportation Research Board. 2003;1819:187–90.

Koon AD, Lopez-Hernandez A, Hoe C, Vecino-Ortiz AI, Cunto FJC, de Castro-Neto MM, et al. Multisectoral action coalitions for road safety in Brazil: an organizational social network analysis in São Paulo and Fortaleza. Traffic Inj Prev. 2022;23:67–72.

UN Road Safety Fund [Internet]. United Nations Road Safety Fund. [cited 2024 Mar 1]. Available from: https://roadsafetyfund.un.org/

Global Road Safety Facility [Internet]. World Bank Group. 2023 [cited 2024 Mar 1]. Available from: https://www.roadsafetyfacility.org/

EU international cooperation in road safety [Internet]. European Commission . [cited 2024 Mar 1]. Available from: https://road-safety.transport.ec.europa.eu/what-we-do/eu-international-cooperation-road-safety_en

United Nations Institute for Training and Research. ROAD SAFETY INITIATIVE [Internet]. [cited 2024 Mar 1]. Available from: https://unitar.org/sustainable-development-goals/people/our-portfolio/road-safety-initiative

Initiative for global Road safety [internet]. Bloomberg Philanthropies 2022 [cited 2022 Dec 11]. Available from: https://www.bloomberg.org/public-health/improving-road-safety/initiative-for-global-road-safety/

Wong G, Westhorp G, Manzano A, Greenhalgh J, Jagosh J, Greenhalgh T. RAMESES II reporting standards for realist evaluations. BMC Med. 2016;14:96. 

Pawson R, Tilley N. Realistic evaluation. Newbury Park: Sage Publications, Inc; 1997.

Yin RK. The case study method as a tool for doing evaluation. Curr Sociol. 1992;40:121–37. 

Yin RK. Case study research - design and methods. 4th ed. Thousand Oaks: SAGE Publications; 2009.

Mukumbang FC, Marchal B, Van Belle S, van Wyk B. Unearthing how, why, for whom and under what health system conditions the antiretroviral treatment adherence club intervention in South Africa works: a realist theory refining approach. BMC Health Serv Res. 2018;18:343.

Greenhalgh T, Pawson R, Wong G, Westhorp G, Greenhalgh J, Manzano A, et al. What realists mean by context; or, why nothing works everywhere or for everyone. 2017.

World Bank Country and Lending Groups [Internet]. World Bank. [cited 2021 Jan 31]. Available from: https://datahelpdesk.worldbank.org/knowledgebase/articles/906519-world-bank-country-and-lending-groups

Population density by city, 2014 [Internet]. Our World in Data. [cited 2022 Dec 26]. Available from: https://ourworldindata.org/grapher/population-density-by-city

Motor vehicles per 1000 inhabitants vs GDP per capita, 2014 [Internet]. Our World in Data . [cited 2022 Dec 26]. Available from: https://ourworldindata.org/grapher/road-vehicles-per-1000-inhabitants-vs-gdp-per-capita?tab=table

Greenhalgh T, Pawson R, Wong G, Westhorp G, Greenhalgh J, Manzano A, et al. Retroduction in realist evaluation. Oxford; 2017.

Manzano A. The craft of interviewing in realist evaluation. Evaluation. 2016;22:342–60.

Saunders B, Sim J, Kingstone T, Baker S, Waterfield J, Bartlam B, et al. Saturation in qualitative research: exploring its conceptualization and operationalization. Qual Quant. 2018;52:1893–907.

Gilmore B, McAuliffe E, Power J, Vallières F. Data analysis and synthesis within a realist evaluation: toward more transparent methodological approaches. International journal of qualitative. Methods. 2019;18.

Mirzoev T, Etiaba E, Ebenso B, Uzochukwu B, Ensor T, Onwujekwe O, et al. Tracing theories in realist evaluations of large-scale health programmes in low- and middle-income countries: experience from Nigeria. Health Policy Plan. 2020;35:1244–53.

Mumbai police traffic control branch, Bloomberg philanthropies initiative for global Road safety. Mumbai Road safety report 2018 key findings [internet]. Mumbai police traffic control branch. Mumbai; 2018 Apr. Available from: https://archive.org/details/mumbairoadsafetyreport2018keyfindings

Mumbai Live Team. Report shows half of road accident casualties in Mumbai in 2018 were pedestrians [Internet]. Mumbai Live. [cited 2023 Feb 15]. Available from: https://www.mumbailive.com/en/transport/mumbai-road-safety-annual-report-2018-shows-half-of-road-accident-casualties-in-mumbai-in-2018-were-pedestrians-40204

Press Trust of India. Mumbai road safety report suggests 22% decline in accident deaths [internet]. India News 2019 [cited 2023 Feb 15]. Available from: https://www.republicworld.com/india-news/general-news/mumbai-road-safety-report-suggests-22-percent-decline-in-accident-deaths.html

Bhatt A, Mascarenhas B, Ashar D. Redesigning One of Mumbai’s Most Dangerous Intersections in 3 Simple Steps [Internet]. TheCityFix. 2019 [cited 2023 Feb 15]. Available from: https://thecityfix.com/blog/redesigning-one-mumbais-dangerous-intersections-3-simple-steps-amit-bhatt-binoy-mascarenhas-dhawal-ashar/

Natu N. LBS Road, 13 other Mumbai junctions set for pedestrian-friendly redesign [internet]. Times of India 2018 [cited 2023 Feb 15]. Available from: https://timesofindia.indiatimes.com/city/mumbai/lbs-road-13-other-mumbai-junctions-set-for-pedestrian-friendly-redesign/articleshow/64673212.cms

Natu N. Mumbai: Times Square experiment at CSMT junction begins [internet]. Times of India. 2019 [cited 2023 Feb 15]. Available from: https://timesofindia.indiatimes.com/city/mumbai/mumbai-times-square-experiment-at-csmt-junction-begins/articleshow/71715425.cms

Minhas G. Mumbai civic body invites urban designers to improve five streets - [Internet]. Governance Now. 2019 [cited 2023 Feb 15]. Available from: https://www.governancenow.com/news/regular-story/mumbai-civic-body-invites-urban-designers-to-improve-five-streets

Singh D. Thirteen-km-stretch of Lal bahadur Shastri Road to be widened, redesigned [internet]. The Indian Express 2018 [cited 2023 Feb 15]. Available from: https://indianexpress.com/article/cities/mumbai/thirteen-km-stretch-of-lal-bahadur-shastri-road-to-be-widened-redesigned-5241735/

AMA-BIGRS to begin road safety enhancement works at Lapaz [internet]. Accra metropolitan Assembly . 2018 [cited 2023 Mar 10]. Available from: https://ama.gov.gh/news-details.php?n=OTkzczkwMnFvMzc4MjI2OTQzNDIxNjJvNW4yM28xbnNxNjE5cDZvbw==

Agbenorsi J, Kwasin J. Police to check speeding on Accra roads [internet]. Graphic Online. 2019 [cited 2023 Feb 16]. Available from: https://www.graphic.com.gh/news/general-news/police-to-check-speeding-on-accra-roads.html

Kanagat N, Chauffour J, Ilunga JF, Yuma Ramazani S, Ovuoraye Ajiwohwodoma JJP, Ibrahim Anas-Kolo S, et al. Country perspectives on improving technical assistance in the health sector. Gates Open Res. 2021;5:141.

Scott VC, Jillani Z, Malpert A, Kolodny-Goetz J, Wandersman A. A scoping review of the evaluation and effectiveness of technical assistance. Implement Sci Commun. 2022;3:70.

Sahin I. Detailed review of Rogers’ diffusion of innovations theory and educational technology-related studies based on Rogers’ theory. Turk Online J Educ Technol. 2006;5:1303–6521.

Hawkins B, Parkhurst J. The ‘good governance’ of evidence in health policy. Evidence & Policy. 2016;12:575–92.

Sewell WH Jr. A theory of structure: duality, agency, and transformation. Am J Sociol. 1992;98:1–29.

Long JC, Cunningham FC, Braithwaite J. Bridges, brokers and boundary spanners in collaborative networks: a systematic review. BMC Health Serv Res. 2013;13:158.

Pelletier D, Gervais S, Hafeez-ur-Rehman H, Sanou D, Tumwine J. Boundary-spanning actors in complex adaptive governance systems: the case of multisectoral nutrition. Int J Health Plann Manag. 2018;33:e293–319.

Download references

Acknowledgements

The authors wish to acknowledge Dr. Jeremy Shiffman, Dr. Svea Closser, and Dr. Nukhba Zia of the Johns Hopkins University Bloomberg School of Public Health who provided comments on an earlier version of this manuscript. The authors also thank Sylviane Ratte, Director, Road Safety Program and Sara Whitehead, Consultant, Public Health and Preventive Medicine, Road Safety Program, at Vital Strategies who provided valuable comments on this research and support in contacting key informants. The authors also wish to thank Alma H. Ramírez of Teasa Translate for providing translation and transcription services for the study. Finally, the authors wish to thank the study participants who generously provided their time and valuable insights.

This project was supported by Bloomberg Philanthropies through the Bloomberg Philanthropies Initiative for Global Road Safety (Grant No. 111882). The funders were not involved in this study or development of this manuscript.

Author information

Authors and affiliations.

Johns Hopkins International Injury Research Unit, Health Systems Program, Department of International Health, Johns Hopkins Bloomberg School of Public Health, 615 N. Wolfe Street Suite E8527, Baltimore, MD, 21205, USA

Rachel Neill, Angélica López Hernández, Adam D. Koon & Abdulgafoor M. Bachani

You can also search for this author in PubMed   Google Scholar

Contributions

RN, ALH, AK, and AB designed this study. RN and ALH collected and analyzed the data. RN wrote the first draft of the manuscript. RN, ALH, AK, and AB provided critical revisions to the manuscript. All authors approved the final version for publication.

Corresponding author

Correspondence to Rachel Neill .

Ethics declarations

Ethics approval and consent to participate.

This study was exempted as non-human subject research by the Johns Hopkins University Bloomberg School of Public Health Institutional Review Board (No: IRB00013713). We received oral informed consent from all participants.

Consent for publication

Not applicable.

Competing interests

The authors declare they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., supplementary material 2., supplementary material 3., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Neill, R., Hernández, A.L., Koon, A.D. et al. Translating global evidence into local implementation through technical assistance: a realist evaluation of the Bloomberg philanthropies initiative for global Road safety. Global Health 20 , 42 (2024). https://doi.org/10.1186/s12992-024-01041-z

Download citation

Received : 20 December 2023

Accepted : 22 April 2024

Published : 10 May 2024

DOI : https://doi.org/10.1186/s12992-024-01041-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Road safety
• Low- and middle-income countries
• Technical assistance
• International development

Globalization and Health

ISSN: 1744-8603

• Submission enquiries: [email protected]
• General enquiries: [email protected]

This website may not work correctly because your browser is out of date. Please update your browser .

Week 32: Better use of case studies in evaluation

Case studies are often used in evaluations – but not always in ways that use their real potential.

Recently I had an opportunity to spend some time with the evaluation unit of UNOIOS  (United Nations Office of Internal Oversight and Inspection) and some of their UN evaluation colleagues exploring ways to better use  case studies in evaluation. Here are five lessons I took away from our time together.

1.    Be clear about what you mean by a case study

Case study is a research design that involves an intensive study of one or more cases rather than an extensive study of many, and which involve multiple sources of evidence – often a combination of quantitative and qualitative data.  

Be clear about what the case is – is it a person, a site, a project, an event, a procedure, a country, or something else? And what is it a case of? A case of successful implementation - or a case that illustrates the barriers to successful implementation? A typical day? A small project, as compared to a large project?

2. Be clear about why you are doing a case study – and then choose the type of case study that matches this.

There are different types of case studies.  Choose the right one for your purpose.  This list draws on a guide " Case Studies in Evaluation " produced by the United State Government Accounting Office, which identified six  different types of case studies – and adds one more (comparative case study):

Illustrative: This is descriptive in character and intended to add realism and in-depth examples to other information about a program or policy. They are  especially useful in evaluations intended to be used by people without direct experience of a program or a situation​.  These are often used to complement quantitative data by providing examples of the overall findings.  These can range from brief narratives to  detailed, vivid, and evocative narratives that provide a vicarious experience and allow readers to understand the connections and meaning​s.

Exploratory: This is also descriptive but is aimed at generating hypotheses for later investigation rather than simply providing illustration.  This type of case study is done before planning a component of the evaluation which will involve extensive data collection (such as a survey)​.

Critical instance: This examines a single instance of unique interest, or serves as a critical test of an assertion about a program, problem or strategy.

Program implementation: This investigates operations, often at several sites, and often with reference to a set of norms or standards about implementation processes.

Program effects: This examines the causal links between the program and observed effects (outputs, outcomes or impacts, depending on the timing of the evaluation) and usually involves multisite, multimethod evaluations.  It involves detailed and strategic data collection to identify and test different theories about what has produced the observed impacts​.

Cumulative :  This brings together findings from many case studies to answer evaluative questions. 

Comparative case studies:  These are not only multiple case studies but ones which are designed to use the comparisons between the cases to build and test hypotheses.

3. Match sampling,  data collection, analysis and reporting to the type of case

In evaluation, it is very unlikely that people will be interested in the case itself without wanting to know how to use those findings to think about a larger population.  Case studies usually use some form of purposeful sampling – random sampling  is rarely appropriate (unless this is the only form of sampling that will be credible to the evaluation users).  

Carefully select the type of purposeful sampling so that appropriate inference or translation of findings can be made.   What I often see in evaluations is inappropriate choice of sampling type which then does not match the type of inference needed.  For example it would not be appropriate to sample extreme cases (such as a very successful site) and then draw conclusions as if the sample were typical cases. I often see case studies where the cases have been chosen in terms of a maximum variety sample drawn across a number of dimensions (for example, choosing countries which show a range of levels of development, region and some relevant contextual factors such as institutional arrangements in the country) -  but then the evaluation is not clear about how to use this information to say something about the larger group.

Being clear about the type of case, and the type of inference that will be made, can make it clear what sort of sample is needed.  For example, an illustrative case study might be done of a case which is identified as " typical " along some dimensions, in order to show what an average case is like.  Or outlier sampling might be used to show what the program looks like when it works particularly well or badly. Or maximum variation sampling might be used to show the range of what it looks like in different situations. 

An exploratory case study might use theory-based sampling , identifying important sub-groups according to the theory of change and sampling from each.  This could be used to develop theories of change for each case and compare them to see how they differ across different cases, or to develop an overall theory of change for the whole program or for types of projects that can be used to guide the next stage of data collection.

4. Link case studies thoughtfully to other elements of an evaluation or a monitoring and evaluation system

Think carefully about when the case studies should be done and how they can be linked. For example, exploratory case studies can be useful to do before a survey; explanatory case studies are likely to be useful after a survey.

5. Create opportunities for iteration

If possible, don’t commit the entire evaluation budget at the beginning but set some aside to follow up emerging findings and test hypotheses by doing additional work such as:

• More data analysis of existing data from cases
• More data collection and analysis from existing cases
• Adding more cases 

You can find more resources about using case studies in evaluation on the Case Study approach page on the BetterEvaluation site.

Do you have other good resources or examples to share?  Do you have questions about using case studies in evaluation?

Related content

'week 32: better use of case studies in evaluation' is referenced in:.

• Impact evaluation

Back to top

© 2022 BetterEvaluation. All right reserved.

• Open access
• Published: 14 May 2024

Developing a survey to measure nursing students’ knowledge, attitudes and beliefs, influences, and willingness to be involved in Medical Assistance in Dying (MAiD): a mixed method modified e-Delphi study

• Jocelyn Schroeder 1 ,
• Barbara Pesut 1 , 2 ,
• Lise Olsen 2 ,
• Nelly D. Oelke 2 &
• Helen Sharp 2  

BMC Nursing volume  23 , Article number:  326 ( 2024 ) Cite this article

31 Accesses

Metrics details

Medical Assistance in Dying (MAiD) was legalized in Canada in 2016. Canada’s legislation is the first to permit Nurse Practitioners (NP) to serve as independent MAiD assessors and providers. Registered Nurses’ (RN) also have important roles in MAiD that include MAiD care coordination; client and family teaching and support, MAiD procedural quality; healthcare provider and public education; and bereavement care for family. Nurses have a right under the law to conscientious objection to participating in MAiD. Therefore, it is essential to prepare nurses in their entry-level education for the practice implications and moral complexities inherent in this practice. Knowing what nursing students think about MAiD is a critical first step. Therefore, the purpose of this study was to develop a survey to measure nursing students’ knowledge, attitudes and beliefs, influences, and willingness to be involved in MAiD in the Canadian context.

The design was a mixed-method, modified e-Delphi method that entailed item generation from the literature, item refinement through a 2 round survey of an expert faculty panel, and item validation through a cognitive focus group interview with nursing students. The settings were a University located in an urban area and a College located in a rural area in Western Canada.

During phase 1, a 56-item survey was developed from existing literature that included demographic items and items designed to measure experience with death and dying (including MAiD), education and preparation, attitudes and beliefs, influences on those beliefs, and anticipated future involvement. During phase 2, an expert faculty panel reviewed, modified, and prioritized the items yielding 51 items. During phase 3, a sample of nursing students further evaluated and modified the language in the survey to aid readability and comprehension. The final survey consists of 45 items including 4 case studies.

Systematic evaluation of knowledge-to-date coupled with stakeholder perspectives supports robust survey design. This study yielded a survey to assess nursing students’ attitudes toward MAiD in a Canadian context.

The survey is appropriate for use in education and research to measure knowledge and attitudes about MAiD among nurse trainees and can be a helpful step in preparing nursing students for entry-level practice.

Peer Review reports

Medical Assistance in Dying (MAiD) is permitted under an amendment to Canada’s Criminal Code which was passed in 2016 [ 1 ]. MAiD is defined in the legislation as both self-administered and clinician-administered medication for the purpose of causing death. In the 2016 Bill C-14 legislation one of the eligibility criteria was that an applicant for MAiD must have a reasonably foreseeable natural death although this term was not defined. It was left to the clinical judgement of MAiD assessors and providers to determine the time frame that constitutes reasonably foreseeable [ 2 ]. However, in 2021 under Bill C-7, the eligibility criteria for MAiD were changed to allow individuals with irreversible medical conditions, declining health, and suffering, but whose natural death was not reasonably foreseeable, to receive MAiD [ 3 ]. This population of MAiD applicants are referred to as Track 2 MAiD (those whose natural death is foreseeable are referred to as Track 1). Track 2 applicants are subject to additional safeguards under the 2021 C-7 legislation.

Three additional proposed changes to the legislation have been extensively studied by Canadian Expert Panels (Council of Canadian Academics [CCA]) [ 4 , 5 , 6 ] First, under the legislation that defines Track 2, individuals with mental disease as their sole underlying medical condition may apply for MAiD, but implementation of this practice is embargoed until March 2027 [ 4 ]. Second, there is consideration of allowing MAiD to be implemented through advanced consent. This would make it possible for persons living with dementia to receive MAID after they have lost the capacity to consent to the procedure [ 5 ]. Third, there is consideration of extending MAiD to mature minors. A mature minor is defined as “a person under the age of majority…and who has the capacity to understand and appreciate the nature and consequences of a decision” ([ 6 ] p. 5). In summary, since the legalization of MAiD in 2016 the eligibility criteria and safeguards have evolved significantly with consequent implications for nurses and nursing care. Further, the number of Canadians who access MAiD shows steady increases since 2016 [ 7 ] and it is expected that these increases will continue in the foreseeable future.

Nurses have been integral to MAiD care in the Canadian context. While other countries such as Belgium and the Netherlands also permit euthanasia, Canada is the first country to allow Nurse Practitioners (Registered Nurses with additional preparation typically achieved at the graduate level) to act independently as assessors and providers of MAiD [ 1 ]. Although the role of Registered Nurses (RNs) in MAiD is not defined in federal legislation, it has been addressed at the provincial/territorial-level with variability in scope of practice by region [ 8 , 9 ]. For example, there are differences with respect to the obligation of the nurse to provide information to patients about MAiD, and to the degree that nurses are expected to ensure that patient eligibility criteria and safeguards are met prior to their participation [ 10 ]. Studies conducted in the Canadian context indicate that RNs perform essential roles in MAiD care coordination; client and family teaching and support; MAiD procedural quality; healthcare provider and public education; and bereavement care for family [ 9 , 11 ]. Nurse practitioners and RNs are integral to a robust MAiD care system in Canada and hence need to be well-prepared for their role [ 12 ].

Previous studies have found that end of life care, and MAiD specifically, raise complex moral and ethical issues for nurses [ 13 , 14 , 15 , 16 ]. The knowledge, attitudes, and beliefs of nurses are important across practice settings because nurses have consistent, ongoing, and direct contact with patients who experience chronic or life-limiting health conditions. Canadian studies exploring nurses’ moral and ethical decision-making in relation to MAiD reveal that although some nurses are clear in their support for, or opposition to, MAiD, others are unclear on what they believe to be good and right [ 14 ]. Empirical findings suggest that nurses go through a period of moral sense-making that is often informed by their family, peers, and initial experiences with MAID [ 17 , 18 ]. Canadian legislation and policy specifies that nurses are not required to participate in MAiD and may recuse themselves as conscientious objectors with appropriate steps to ensure ongoing and safe care of patients [ 1 , 19 ]. However, with so many nurses having to reflect on and make sense of their moral position, it is essential that they are given adequate time and preparation to make an informed and thoughtful decision before they participate in a MAID death [ 20 , 21 ].

It is well established that nursing students receive inconsistent exposure to end of life care issues [ 22 ] and little or no training related to MAiD [ 23 ]. Without such education and reflection time in pre-entry nursing preparation, nurses are at significant risk for moral harm. An important first step in providing this preparation is to be able to assess the knowledge, values, and beliefs of nursing students regarding MAID and end of life care. As demand for MAiD increases along with the complexities of MAiD, it is critical to understand the knowledge, attitudes, and likelihood of engagement with MAiD among nursing students as a baseline upon which to build curriculum and as a means to track these variables over time.

Aim, design, and setting

The aim of this study was to develop a survey to measure nursing students’ knowledge, attitudes and beliefs, influences, and willingness to be involved in MAiD in the Canadian context. We sought to explore both their willingness to be involved in the registered nursing role and in the nurse practitioner role should they chose to prepare themselves to that level of education. The design was a mixed-method, modified e-Delphi method that entailed item generation, item refinement through an expert faculty panel [ 24 , 25 , 26 ], and initial item validation through a cognitive focus group interview with nursing students [ 27 ]. The settings were a University located in an urban area and a College located in a rural area in Western Canada.

Participants

A panel of 10 faculty from the two nursing education programs were recruited for Phase 2 of the e-Delphi. To be included, faculty were required to have a minimum of three years of experience in nurse education, be employed as nursing faculty, and self-identify as having experience with MAiD. A convenience sample of 5 fourth-year nursing students were recruited to participate in Phase 3. Students had to be in good standing in the nursing program and be willing to share their experiences of the survey in an online group interview format.

The modified e-Delphi was conducted in 3 phases: Phase 1 entailed item generation through literature and existing survey review. Phase 2 entailed item refinement through a faculty expert panel review with focus on content validity, prioritization, and revision of item wording [ 25 ]. Phase 3 entailed an assessment of face validity through focus group-based cognitive interview with nursing students.

Phase I. Item generation through literature review

The goal of phase 1 was to develop a bank of survey items that would represent the variables of interest and which could be provided to expert faculty in Phase 2. Initial survey items were generated through a literature review of similar surveys designed to assess knowledge and attitudes toward MAiD/euthanasia in healthcare providers; Canadian empirical studies on nurses’ roles and/or experiences with MAiD; and legislative and expert panel documents that outlined proposed changes to the legislative eligibility criteria and safeguards. The literature review was conducted in three online databases: CINAHL, PsycINFO, and Medline. Key words for the search included nurses , nursing students , medical students , NPs, MAiD , euthanasia , assisted death , and end-of-life care . Only articles written in English were reviewed. The legalization and legislation of MAiD is new in many countries; therefore, studies that were greater than twenty years old were excluded, no further exclusion criteria set for country.

Items from surveys designed to measure similar variables in other health care providers and geographic contexts were placed in a table and similar items were collated and revised into a single item. Then key variables were identified from the empirical literature on nurses and MAiD in Canada and checked against the items derived from the surveys to ensure that each of the key variables were represented. For example, conscientious objection has figured prominently in the Canadian literature, but there were few items that assessed knowledge of conscientious objection in other surveys and so items were added [ 15 , 21 , 28 , 29 ]. Finally, four case studies were added to the survey to address the anticipated changes to the Canadian legislation. The case studies were based upon the inclusion of mature minors, advanced consent, and mental disorder as the sole underlying medical condition. The intention was to assess nurses’ beliefs and comfort with these potential legislative changes.

Phase 2. Item refinement through expert panel review

The goal of phase 2 was to refine and prioritize the proposed survey items identified in phase 1 using a modified e-Delphi approach to achieve consensus among an expert panel [ 26 ]. Items from phase 1 were presented to an expert faculty panel using a Qualtrics (Provo, UT) online survey. Panel members were asked to review each item to determine if it should be: included, excluded or adapted for the survey. When adapted was selected faculty experts were asked to provide rationale and suggestions for adaptation through the use of an open text box. Items that reached a level of 75% consensus for either inclusion or adaptation were retained [ 25 , 26 ]. New items were categorized and added, and a revised survey was presented to the panel of experts in round 2. Panel members were again asked to review items, including new items, to determine if it should be: included, excluded, or adapted for the survey. Round 2 of the modified e-Delphi approach also included an item prioritization activity, where participants were then asked to rate the importance of each item, based on a 5-point Likert scale (low to high importance), which De Vaus [ 30 ] states is helpful for increasing the reliability of responses. Items that reached a 75% consensus on inclusion were then considered in relation to the importance it was given by the expert panel. Quantitative data were managed using SPSS (IBM Corp).

Phase 3. Face validity through cognitive interviews with nursing students

The goal of phase 3 was to obtain initial face validity of the proposed survey using a sample of nursing student informants. More specifically, student participants were asked to discuss how items were interpreted, to identify confusing wording or other problematic construction of items, and to provide feedback about the survey as a whole including readability and organization [ 31 , 32 , 33 ]. The focus group was held online and audio recorded. A semi-structured interview guide was developed for this study that focused on clarity, meaning, order and wording of questions; emotions evoked by the questions; and overall survey cohesion and length was used to obtain data (see Supplementary Material 2  for the interview guide). A prompt to “think aloud” was used to limit interviewer-imposed bias and encourage participants to describe their thoughts and response to a given item as they reviewed survey items [ 27 ]. Where needed, verbal probes such as “could you expand on that” were used to encourage participants to expand on their responses [ 27 ]. Student participants’ feedback was collated verbatim and presented to the research team where potential survey modifications were negotiated and finalized among team members. Conventional content analysis [ 34 ] of focus group data was conducted to identify key themes that emerged through discussion with students. Themes were derived from the data by grouping common responses and then using those common responses to modify survey items.

Ten nursing faculty participated in the expert panel. Eight of the 10 faculty self-identified as female. No faculty panel members reported conscientious objector status and ninety percent reported general agreement with MAiD with one respondent who indicated their view as “unsure.” Six of the 10 faculty experts had 16 years of experience or more working as a nurse educator.

Five nursing students participated in the cognitive interview focus group. The duration of the focus group was 2.5 h. All participants identified that they were born in Canada, self-identified as female (one preferred not to say) and reported having received some instruction about MAiD as part of their nursing curriculum. See Tables  1 and 2 for the demographic descriptors of the study sample. Study results will be reported in accordance with the study phases. See Fig.  1 for an overview of the results from each phase.

Fig. 1  Overview of survey development findings

Phase 1: survey item generation

Review of the literature identified that no existing survey was available for use with nursing students in the Canadian context. However, an analysis of themes across qualitative and quantitative studies of physicians, medical students, nurses, and nursing students provided sufficient data to develop a preliminary set of items suitable for adaptation to a population of nursing students.

Four major themes and factors that influence knowledge, attitudes, and beliefs about MAiD were evident from the literature: (i) endogenous or individual factors such as age, gender, personally held values, religion, religiosity, and/or spirituality [ 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 ], (ii) experience with death and dying in personal and/or professional life [ 35 , 40 , 41 , 43 , 44 , 45 ], (iii) training including curricular instruction about clinical role, scope of practice, or the law [ 23 , 36 , 39 ], and (iv) exogenous or social factors such as the influence of key leaders, colleagues, friends and/or family, professional and licensure organizations, support within professional settings, and/or engagement in MAiD in an interdisciplinary team context [ 9 , 35 , 46 ].

Studies of nursing students also suggest overlap across these categories. For example, value for patient autonomy [ 23 ] and the moral complexity of decision-making [ 37 ] are important factors that contribute to attitudes about MAiD and may stem from a blend of personally held values coupled with curricular content, professional training and norms, and clinical exposure. For example, students report that participation in end of life care allows for personal growth, shifts in perception, and opportunities to build therapeutic relationships with their clients [ 44 , 47 , 48 ].

Preliminary items generated from the literature resulted in 56 questions from 11 published sources (See Table  3 ). These items were constructed across four main categories: (i) socio-demographic questions; (ii) end of life care questions; (iii) knowledge about MAiD; or (iv) comfort and willingness to participate in MAiD. Knowledge questions were refined to reflect current MAiD legislation, policies, and regulatory frameworks. Falconer [ 39 ] and Freeman [ 45 ] studies were foundational sources for item selection. Additionally, four case studies were written to reflect the most recent anticipated changes to MAiD legislation and all used the same open-ended core questions to address respondents’ perspectives about the patient’s right to make the decision, comfort in assisting a physician or NP to administer MAiD in that scenario, and hypothesized comfort about serving as a primary provider if qualified as an NP in future. Response options for the survey were also constructed during this stage and included: open text, categorical, yes/no , and Likert scales.

Phase 2: faculty expert panel review

Of the 56 items presented to the faculty panel, 54 questions reached 75% consensus. However, based upon the qualitative responses 9 items were removed largely because they were felt to be repetitive. Items that generated the most controversy were related to measuring religion and spirituality in the Canadian context, defining end of life care when there is no agreed upon time frames (e.g., last days, months, or years), and predicting willingness to be involved in a future events – thus predicting their future selves. Phase 2, round 1 resulted in an initial set of 47 items which were then presented back to the faculty panel in round 2.

Of the 47 initial questions presented to the panel in round 2, 45 reached a level of consensus of 75% or greater, and 34 of these questions reached a level of 100% consensus [ 27 ] of which all participants chose to include without any adaptations) For each question, level of importance was determined based on a 5-point Likert scale (1 = very unimportant, 2 = somewhat unimportant, 3 = neutral, 4 = somewhat important, and 5 = very important). Figure  2 provides an overview of the level of importance assigned to each item.

Ranking level of importance for survey items

After round 2, a careful analysis of participant comments and level of importance was completed by the research team. While the main method of survey item development came from participants’ response to the first round of Delphi consensus ratings, level of importance was used to assist in the decision of whether to keep or modify questions that created controversy, or that rated lower in the include/exclude/adapt portion of the Delphi. Survey items that rated low in level of importance included questions about future roles, sex and gender, and religion/spirituality. After deliberation by the research committee, these questions were retained in the survey based upon the importance of these variables in the scientific literature.

Of the 47 questions remaining from Phase 2, round 2, four were revised. In addition, the two questions that did not meet the 75% cut off level for consensus were reviewed by the research team. The first question reviewed was What is your comfort level with providing a MAiD death in the future if you were a qualified NP ? Based on a review of participant comments, it was decided to retain this question for the cognitive interviews with students in the final phase of testing. The second question asked about impacts on respondents’ views of MAiD and was changed from one item with 4 subcategories into 4 separate items, resulting in a final total of 51 items for phase 3. The revised survey was then brought forward to the cognitive interviews with student participants in Phase 3. (see Supplementary Material 1 for a complete description of item modification during round 2).

Phase 3. Outcomes of cognitive interview focus group

Of the 51 items reviewed by student participants, 29 were identified as clear with little or no discussion. Participant comments for the remaining 22 questions were noted and verified against the audio recording. Following content analysis of the comments, four key themes emerged through the student discussion: unclear or ambiguous wording; difficult to answer questions; need for additional response options; and emotional response evoked by questions. An example of unclear or ambiguous wording was a request for clarity in the use of the word “sufficient” in the context of assessing an item that read “My nursing education has provided sufficient content about the nursing role in MAiD.” “Sufficient” was viewed as subjective and “laden with…complexity that distracted me from the question.” The group recommended rewording the item to read “My nursing education has provided enough content for me to care for a patient considering or requesting MAiD.”

An example of having difficulty answering questions related to limited knowledge related to terms used in the legislation such as such as safeguards , mature minor , eligibility criteria , and conscientious objection. Students were unclear about what these words meant relative to the legislation and indicated that this lack of clarity would hamper appropriate responses to the survey. To ensure that respondents are able to answer relevant questions, student participants recommended that the final survey include explanation of key terms such as mature minor and conscientious objection and an overview of current legislation.

Response options were also a point of discussion. Participants noted a lack of distinction between response options of unsure and unable to say . Additionally, scaling of attitudes was noted as important since perspectives about MAiD are dynamic and not dichotomous “agree or disagree” responses. Although the faculty expert panel recommended the integration of the demographic variables of religious and/or spiritual remain as a single item, the student group stated a preference to have religion and spirituality appear as separate items. The student focus group also took issue with separate items for the variables of sex and gender, specifically that non-binary respondents might feel othered or “outed” particularly when asked to identify their sex. These variables had been created based upon best practices in health research but students did not feel they were appropriate in this context [ 49 ]. Finally, students agreed with the faculty expert panel in terms of the complexity of projecting their future involvement as a Nurse Practitioner. One participant stated: “I certainly had to like, whoa, whoa, whoa. Now let me finish this degree first, please.” Another stated, “I'm still imagining myself, my future career as an RN.”

Finally, student participants acknowledged the array of emotions that some of the items produced for them. For example, one student described positive feelings when interacting with the survey. “Brought me a little bit of feeling of joy. Like it reminded me that this is the last piece of independence that people grab on to.” Another participant, described the freedom that the idea of an advance request gave her. “The advance request gives the most comfort for me, just with early onset Alzheimer’s and knowing what it can do.” But other participants described less positive feelings. For example, the mature minor case study yielded a comment: “This whole scenario just made my heart hurt with the idea of a child requesting that.”

Based on the data gathered from the cognitive interview focus group of nursing students, revisions were made to 11 closed-ended questions (see Table  4 ) and 3 items were excluded. In the four case studies, the open-ended question related to a respondents’ hypothesized actions in a future role as NP were removed. The final survey consists of 45 items including 4 case studies (see Supplementary Material 3 ).

The aim of this study was to develop and validate a survey that can be used to track the growth of knowledge about MAiD among nursing students over time, inform training programs about curricular needs, and evaluate attitudes and willingness to participate in MAiD at time-points during training or across nursing programs over time.

The faculty expert panel and student participants in the cognitive interview focus group identified a need to establish core knowledge of the terminology and legislative rules related to MAiD. For example, within the cognitive interview group of student participants, several acknowledged lack of clear understanding of specific terms such as “conscientious objector” and “safeguards.” Participants acknowledged discomfort with the uncertainty of not knowing and their inclination to look up these terms to assist with answering the questions. This survey can be administered to nursing or pre-nursing students at any phase of their training within a program or across training programs. However, in doing so it is important to acknowledge that their baseline knowledge of MAiD will vary. A response option of “not sure” is important and provides a means for respondents to convey uncertainty. If this survey is used to inform curricular needs, respondents should be given explicit instructions not to conduct online searches to inform their responses, but rather to provide an honest appraisal of their current knowledge and these instructions are included in the survey (see Supplementary Material 3 ).

Some provincial regulatory bodies have established core competencies for entry-level nurses that include MAiD. For example, the BC College of Nurses and Midwives (BCCNM) requires “knowledge about ethical, legal, and regulatory implications of medical assistance in dying (MAiD) when providing nursing care.” (10 p. 6) However, across Canada curricular content and coverage related to end of life care and MAiD is variable [ 23 ]. Given the dynamic nature of the legislation that includes portions of the law that are embargoed until 2024, it is important to ensure that respondents are guided by current and accurate information. As the law changes, nursing curricula, and public attitudes continue to evolve, inclusion of core knowledge and content is essential and relevant for investigators to be able to interpret the portions of the survey focused on attitudes and beliefs about MAiD. Content knowledge portions of the survey may need to be modified over time as legislation and training change and to meet the specific purposes of the investigator.

Given the sensitive nature of the topic, it is strongly recommended that surveys be conducted anonymously and that students be provided with an opportunity to discuss their responses to the survey. A majority of feedback from both the expert panel of faculty and from student participants related to the wording and inclusion of demographic variables, in particular religion, religiosity, gender identity, and sex assigned at birth. These and other demographic variables have the potential to be highly identifying in small samples. In any instance in which the survey could be expected to yield demographic group sizes less than 5, users should eliminate the demographic variables from the survey. For example, the profession of nursing is highly dominated by females with over 90% of nurses who identify as female [ 50 ]. Thus, a survey within a single class of students or even across classes in a single institution is likely to yield a small number of male respondents and/or respondents who report a difference between sex assigned at birth and gender identity. When variables that serve to identify respondents are included, respondents are less likely to complete or submit the survey, to obscure their responses so as not to be identifiable, or to be influenced by social desirability bias in their responses rather than to convey their attitudes accurately [ 51 ]. Further, small samples do not allow for conclusive analyses or interpretation of apparent group differences. Although these variables are often included in surveys, such demographics should be included only when anonymity can be sustained. In small and/or known samples, highly identifying variables should be omitted.

There are several limitations associated with the development of this survey. The expert panel was comprised of faculty who teach nursing students and are knowledgeable about MAiD and curricular content, however none identified as a conscientious objector to MAiD. Ideally, our expert panel would have included one or more conscientious objectors to MAiD to provide a broader perspective. Review by practitioners who participate in MAiD, those who are neutral or undecided, and practitioners who are conscientious objectors would ensure broad applicability of the survey. This study included one student cognitive interview focus group with 5 self-selected participants. All student participants had held discussions about end of life care with at least one patient, 4 of 5 participants had worked with a patient who requested MAiD, and one had been present for a MAiD death. It is not clear that these participants are representative of nursing students demographically or by experience with end of life care. It is possible that the students who elected to participate hold perspectives and reflections on patient care and MAiD that differ from students with little or no exposure to end of life care and/or MAiD. However, previous studies find that most nursing students have been involved with end of life care including meaningful discussions about patients’ preferences and care needs during their education [ 40 , 44 , 47 , 48 , 52 ]. Data collection with additional student focus groups with students early in their training and drawn from other training contexts would contribute to further validation of survey items.

Future studies should incorporate pilot testing with small sample of nursing students followed by a larger cross-program sample to allow evaluation of the psychometric properties of specific items and further refinement of the survey tool. Consistent with literature about the importance of leadership in the context of MAiD [ 12 , 53 , 54 ], a study of faculty knowledge, beliefs, and attitudes toward MAiD would provide context for understanding student perspectives within and across programs. Additional research is also needed to understand the timing and content coverage of MAiD across Canadian nurse training programs’ curricula.

The implementation of MAiD is complex and requires understanding of the perspectives of multiple stakeholders. Within the field of nursing this includes clinical providers, educators, and students who will deliver clinical care. A survey to assess nursing students’ attitudes toward and willingness to participate in MAiD in the Canadian context is timely, due to the legislation enacted in 2016 and subsequent modifications to the law in 2021 with portions of the law to be enacted in 2027. Further development of this survey could be undertaken to allow for use in settings with practicing nurses or to allow longitudinal follow up with students as they enter practice. As the Canadian landscape changes, ongoing assessment of the perspectives and needs of health professionals and students in the health professions is needed to inform policy makers, leaders in practice, curricular needs, and to monitor changes in attitudes and practice patterns over time.

Availability of data and materials

The datasets used and/or analysed during the current study are not publicly available due to small sample sizes, but are available from the corresponding author on reasonable request.

Abbreviations

British Columbia College of Nurses and Midwives

Medical assistance in dying

Nurse practitioner

Registered nurse

University of British Columbia Okanagan

Nicol J, Tiedemann M. Legislative Summary: Bill C-14: An Act to amend the Criminal Code and to make related amendments to other Acts (medical assistance in dying). Available from: https://lop.parl.ca/staticfiles/PublicWebsite/Home/ResearchPublications/LegislativeSummaries/PDF/42-1/c14-e.pdf .

Downie J, Scallion K. Foreseeably unclear. The meaning of the “reasonably foreseeable” criterion for access to medical assistance in dying in Canada. Dalhousie Law J. 2018;41(1):23–57.

Nicol J, Tiedeman M. Legislative summary of Bill C-7: an act to amend the criminal code (medical assistance in dying). Ottawa: Government of Canada; 2021.

Google Scholar  

Council of Canadian Academies. The state of knowledge on medical assistance in dying where a mental disorder is the sole underlying medical condition. Ottawa; 2018. Available from: https://cca-reports.ca/wp-content/uploads/2018/12/The-State-of-Knowledge-on-Medical-Assistance-in-Dying-Where-a-Mental-Disorder-is-the-Sole-Underlying-Medical-Condition.pdf .

Council of Canadian Academies. The state of knowledge on advance requests for medical assistance in dying. Ottawa; 2018. Available from: https://cca-reports.ca/wp-content/uploads/2019/02/The-State-of-Knowledge-on-Advance-Requests-for-Medical-Assistance-in-Dying.pdf .

Council of Canadian Academies. The state of knowledge on medical assistance in dying for mature minors. Ottawa; 2018. Available from: https://cca-reports.ca/wp-content/uploads/2018/12/The-State-of-Knowledge-on-Medical-Assistance-in-Dying-for-Mature-Minors.pdf .

Health Canada. Third annual report on medical assistance in dying in Canada 2021. Ottawa; 2022. [cited 2023 Oct 23]. Available from: https://www.canada.ca/en/health-canada/services/medical-assistance-dying/annual-report-2021.html .

Banner D, Schiller CJ, Freeman S. Medical assistance in dying: a political issue for nurses and nursing in Canada. Nurs Philos. 2019;20(4): e12281.

Article   PubMed   Google Scholar  

Pesut B, Thorne S, Stager ML, Schiller CJ, Penney C, Hoffman C, et al. Medical assistance in dying: a review of Canadian nursing regulatory documents. Policy Polit Nurs Pract. 2019;20(3):113–30.

Article   PubMed   PubMed Central   Google Scholar  

College of Registered Nurses of British Columbia. Scope of practice for registered nurses [Internet]. Vancouver; 2018. Available from: https://www.bccnm.ca/Documents/standards_practice/rn/RN_ScopeofPractice.pdf .

Pesut B, Thorne S, Schiller C, Greig M, Roussel J, Tishelman C. Constructing good nursing practice for medical assistance in dying in Canada: an interpretive descriptive study. Global Qual Nurs Res. 2020;7:2333393620938686. https://doi.org/10.1177/2333393620938686 .

Article   Google Scholar  

Pesut B, Thorne S, Schiller CJ, Greig M, Roussel J. The rocks and hard places of MAiD: a qualitative study of nursing practice in the context of legislated assisted death. BMC Nurs. 2020;19:12. https://doi.org/10.1186/s12912-020-0404-5 .

Pesut B, Greig M, Thorne S, Burgess M, Storch JL, Tishelman C, et al. Nursing and euthanasia: a narrative review of the nursing ethics literature. Nurs Ethics. 2020;27(1):152–67.

Pesut B, Thorne S, Storch J, Chambaere K, Greig M, Burgess M. Riding an elephant: a qualitative study of nurses’ moral journeys in the context of Medical Assistance in Dying (MAiD). Journal Clin Nurs. 2020;29(19–20):3870–81.

Lamb C, Babenko-Mould Y, Evans M, Wong CA, Kirkwood KW. Conscientious objection and nurses: results of an interpretive phenomenological study. Nurs Ethics. 2018;26(5):1337–49.

Wright DK, Chan LS, Fishman JR, Macdonald ME. “Reflection and soul searching:” Negotiating nursing identity at the fault lines of palliative care and medical assistance in dying. Social Sci & Med. 2021;289: 114366.

Beuthin R, Bruce A, Scaia M. Medical assistance in dying (MAiD): Canadian nurses’ experiences. Nurs Forum. 2018;54(4):511–20.

Bruce A, Beuthin R. Medically assisted dying in Canada: "Beautiful Death" is transforming nurses' experiences of suffering. The Canadian J Nurs Res | Revue Canadienne de Recherche en Sci Infirmieres. 2020;52(4):268–77. https://doi.org/10.1177/0844562119856234 .

Canadian Nurses Association. Code of ethics for registered nurses. Ottawa; 2017. Available from: https://www.cna-aiic.ca/en/nursing/regulated-nursing-in-canada/nursing-ethics .

Canadian Nurses Association. National nursing framework on Medical Assistance in Dying in Canada. Ottawa: 2017. Available from: https://www.virtualhospice.ca/Assets/cna-national-nursing-framework-on-maidEng_20170216155827.pdf .

Pesut B, Thorne S, Greig M. Shades of gray: conscientious objection in medical assistance in dying. Nursing Inq. 2020;27(1): e12308.

Durojaiye A, Ryan R, Doody O. Student nurse education and preparation for palliative care: a scoping review. PLoS ONE. 2023. https://doi.org/10.1371/journal.pone.0286678 .

McMechan C, Bruce A, Beuthin R. Canadian nursing students’ experiences with medical assistance in dying | Les expériences d’étudiantes en sciences infirmières au regard de l’aide médicale à mourir. Qual Adv Nurs Educ - Avancées en Formation Infirmière. 2019;5(1). https://doi.org/10.17483/2368-6669.1179 .

Adler M, Ziglio E. Gazing into the oracle. The Delphi method and its application to social policy and public health. London: Jessica Kingsley Publishers; 1996

Keeney S, Hasson F, McKenna H. Consulting the oracle: ten lessons from using the Delphi technique in nursing research. J Adv Nurs. 2006;53(2):205–12.

Keeney S, Hasson F, McKenna H. The Delphi technique in nursing and health research. 1st ed. City: Wiley; 2011.

Willis GB. Cognitive interviewing: a tool for improving questionnaire design. 1st ed. Thousand Oaks, Calif: Sage; 2005. ISBN: 9780761928041

Lamb C, Evans M, Babenko-Mould Y, Wong CA, Kirkwood EW. Conscience, conscientious objection, and nursing: a concept analysis. Nurs Ethics. 2017;26(1):37–49.

Lamb C, Evans M, Babenko-Mould Y, Wong CA, Kirkwood K. Nurses’ use of conscientious objection and the implications of conscience. J Adv Nurs. 2018;75(3):594–602.

de Vaus D. Surveys in social research. 6th ed. Abingdon, Oxon: Routledge; 2014.

Boateng GO, Neilands TB, Frongillo EA, Melgar-Quiñonez HR, Young SL. Best practices for developing and validating scales for health, social, and behavioral research: A primer. Front Public Health. 2018;6:149. https://doi.org/10.3389/fpubh.2018.00149 .

Puchta C, Potter J. Focus group practice. 1st ed. London: Sage; 2004.

Book   Google Scholar  

Streiner DL, Norman GR, Cairney J. Health measurement scales: a practical guide to their development and use. 5th ed. Oxford: Oxford University Press; 2015.

Hsieh H-F, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005;15(9):1277–88.

Adesina O, DeBellis A, Zannettino L. Third-year Australian nursing students’ attitudes, experiences, knowledge, and education concerning end-of-life care. Int J of Palliative Nurs. 2014;20(8):395–401.

Bator EX, Philpott B, Costa AP. This moral coil: a cross-sectional survey of Canadian medical student attitudes toward medical assistance in dying. BMC Med Ethics. 2017;18(1):58.

Beuthin R, Bruce A, Scaia M. Medical assistance in dying (MAiD): Canadian nurses’ experiences. Nurs Forum. 2018;53(4):511–20.

Brown J, Goodridge D, Thorpe L, Crizzle A. What is right for me, is not necessarily right for you: the endogenous factors influencing nonparticipation in medical assistance in dying. Qual Health Res. 2021;31(10):1786–1800.

Falconer J, Couture F, Demir KK, Lang M, Shefman Z, Woo M. Perceptions and intentions toward medical assistance in dying among Canadian medical students. BMC Med Ethics. 2019;20(1):22.

Green G, Reicher S, Herman M, Raspaolo A, Spero T, Blau A. Attitudes toward euthanasia—dual view: Nursing students and nurses. Death Stud. 2022;46(1):124–31.

Hosseinzadeh K, Rafiei H. Nursing student attitudes toward euthanasia: a cross-sectional study. Nurs Ethics. 2019;26(2):496–503.

Ozcelik H, Tekir O, Samancioglu S, Fadiloglu C, Ozkara E. Nursing students’ approaches toward euthanasia. Omega (Westport). 2014;69(1):93–103.

Canning SE, Drew C. Canadian nursing students’ understanding, and comfort levels related to medical assistance in dying. Qual Adv Nurs Educ - Avancées en Formation Infirmière. 2022;8(2). https://doi.org/10.17483/2368-6669.1326 .

Edo-Gual M, Tomás-Sábado J, Bardallo-Porras D, Monforte-Royo C. The impact of death and dying on nursing students: an explanatory model. J Clin Nurs. 2014;23(23–24):3501–12.

Freeman LA, Pfaff KA, Kopchek L, Liebman J. Investigating palliative care nurse attitudes towards medical assistance in dying: an exploratory cross-sectional study. J Adv Nurs. 2020;76(2):535–45.

Brown J, Goodridge D, Thorpe L, Crizzle A. “I am okay with it, but I am not going to do it:” the exogenous factors influencing non-participation in medical assistance in dying. Qual Health Res. 2021;31(12):2274–89.

Dimoula M, Kotronoulas G, Katsaragakis S, Christou M, Sgourou S, Patiraki E. Undergraduate nursing students’ knowledge about palliative care and attitudes towards end-of-life care: A three-cohort, cross-sectional survey. Nurs Educ Today. 2019;74:7–14.

Matchim Y, Raetong P. Thai nursing students’ experiences of caring for patients at the end of life: a phenomenological study. Int J Palliative Nurs. 2018;24(5):220–9.

Canadian Institute for Health Research. Sex and gender in health research [Internet]. Ottawa: CIHR; 2021 [cited 2023 Oct 23]. Available from: https://cihr-irsc.gc.ca/e/50833.html .

Canadian Nurses’ Association. Nursing statistics. Ottawa: CNA; 2023 [cited 2023 Oct 23]. Available from: https://www.cna-aiic.ca/en/nursing/regulated-nursing-in-canada/nursing-statistics .

Krumpal I. Determinants of social desirability bias in sensitive surveys: a literature review. Qual Quant. 2013;47(4):2025–47. https://doi.org/10.1007/s11135-011-9640-9 .

Ferri P, Di Lorenzo R, Stifani S, Morotti E, Vagnini M, Jiménez Herrera MF, et al. Nursing student attitudes toward dying patient care: a European multicenter cross-sectional study. Acta Bio Medica Atenei Parmensis. 2021;92(S2): e2021018.

PubMed   PubMed Central   Google Scholar  

Beuthin R, Bruce A. Medical assistance in dying (MAiD): Ten things leaders need to know. Nurs Leadership. 2018;31(4):74–81.

Thiele T, Dunsford J. Nurse leaders’ role in medical assistance in dying: a relational ethics approach. Nurs Ethics. 2019;26(4):993–9.

Download references

Acknowledgements

We would like to acknowledge the faculty and students who generously contributed their time to this work.

JS received a student traineeship through the Principal Research Chairs program at the University of British Columbia Okanagan.

Author information

Authors and affiliations.

School of Health and Human Services, Selkirk College, Castlegar, BC, Canada

Jocelyn Schroeder & Barbara Pesut

School of Nursing, University of British Columbia Okanagan, Kelowna, BC, Canada

Barbara Pesut, Lise Olsen, Nelly D. Oelke & Helen Sharp

You can also search for this author in PubMed   Google Scholar

Contributions

JS made substantial contributions to the conception of the work; data acquisition, analysis, and interpretation; and drafting and substantively revising the work. JS has approved the submitted version and agreed to be personally accountable for the author's own contributions and to ensure that questions related to the accuracy or integrity of any part of the work, even ones in which the author was not personally involved, are appropriately investigated, resolved, and the resolution documented in the literature. BP made substantial contributions to the conception of the work; data acquisition, analysis, and interpretation; and drafting and substantively revising the work. BP has approved the submitted version and agreed to be personally accountable for the author's own contributions and to ensure that questions related to the accuracy or integrity of any part of the work, even ones in which the author was not personally involved, are appropriately investigated, resolved, and the resolution documented in the literature. LO made substantial contributions to the conception of the work; data acquisition, analysis, and interpretation; and substantively revising the work. LO has approved the submitted version and agreed to be personally accountable for the author's own contributions and to ensure that questions related to the accuracy or integrity of any part of the work, even ones in which the author was not personally involved, are appropriately investigated, resolved, and the resolution documented in the literature. NDO made substantial contributions to the conception of the work; data acquisition, analysis, and interpretation; and substantively revising the work. NDO has approved the submitted version and agreed to be personally accountable for the author's own contributions and to ensure that questions related to the accuracy or integrity of any part of the work, even ones in which the author was not personally involved, are appropriately investigated, resolved, and the resolution documented in the literature. HS made substantial contributions to drafting and substantively revising the work. HS has approved the submitted version and agreed to be personally accountable for the author's own contributions and to ensure that questions related to the accuracy or integrity of any part of the work, even ones in which the author was not personally involved, are appropriately investigated, resolved, and the resolution documented in the literature.

Authors’ information

JS conducted this study as part of their graduate requirements in the School of Nursing, University of British Columbia Okanagan.

Corresponding author

Correspondence to Barbara Pesut .

Ethics declarations

Ethics approval and consent to participate.

The research was approved by the Selkirk College Research Ethics Board (REB) ID # 2021–011 and the University of British Columbia Behavioral Research Ethics Board ID # H21-01181.

All participants provided written and informed consent through approved consent processes. Research was conducted in accordance with the Declaration of Helsinki.

Consent for publication

Not applicable.

Competing interests

The authors declare they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., supplementary material 2., supplementary material 3., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Schroeder, J., Pesut, B., Olsen, L. et al. Developing a survey to measure nursing students’ knowledge, attitudes and beliefs, influences, and willingness to be involved in Medical Assistance in Dying (MAiD): a mixed method modified e-Delphi study. BMC Nurs 23 , 326 (2024). https://doi.org/10.1186/s12912-024-01984-z

Download citation

Received : 24 October 2023

Accepted : 28 April 2024

Published : 14 May 2024

DOI : https://doi.org/10.1186/s12912-024-01984-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Medical assistance in dying (MAiD)
• End of life care
• Student nurses
• Nursing education

BMC Nursing

ISSN: 1472-6955

• General enquiries: [email protected]

Evaluation of online job portals for HR recruitment selection using AHP in two wheeler automotive industry: a case study

• ORIGINAL ARTICLE
• Published: 12 May 2024

Cite this article

• S. M. Vadivel   ORCID: orcid.org/0000-0002-5287-3693 1 &
• Rohan Sunny   ORCID: orcid.org/0009-0002-2347-3081 2  

38 Accesses

Explore all metrics

Automotive companies are booming worldwide in the economy. In order to sustain in the highly competitive world, every organization tries to create itself a trademark in the market. In our research, we looked at how two wheelers automotive company's selection enhances an organizational performance, which ensures the company's future growth. In today's fast-paced, globally integrated world, human resources are one of the most important production variables. It is critical to preserve and improve economic competitiveness by properly selecting and developing these resources. The main aim of this study is to identify the best online job portal website for recruitment at Two Wheeler Company and to suggest an HR strategy which resonates company’s values and culture. In this study, we have selected 6 criteria and 6 online popular job portals for recruitment with a sample of 15 candidates have been selected. Findings reveal that, AHP method has significant results on the selection of best employer, which helps HR Manager to finalize the decision making process/strategies. Towards the managerial implications section, the researcher aims to design an functional and effective HR strategy that can grasp, engage and retain the top talent in the organization.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Similar content being viewed by others

Reskilling and Upskilling the Future-ready Workforce for Industry 4.0 and Beyond

Exploring Human Resource Management Digital Transformation in the Digital Age

Work engagement and employee satisfaction in the practice of sustainable human resource management – based on the study of Polish employees

Availability of data and material.

'Not applicable' in this section.

Abbreviations

Analytic hierarchy process

Artificial intelligence

Analysis of variance

Chief Human Resources Officer

Consistency index

Curriculum vitae

Consistency ratio

Decision making

Faculty Development Programme

Hierarchical linear modelling

Human resources

Research and Development

Randomized index

Structural equation modelling

Search engine optimization

Triple bottom line

Technique for order preference by similarity

Maximum Eigen value

The normalized value of ith criterion for the jth alternative

The normalized value of jth criterion for the ith alternative

The number of alternatives for a certain MCDM problem

The number of criteria for a certain MCDM problem

Avinash Kapse S, Vishal Patil S, Nikhil PV (2021) E-Recruitment. Int J Eng Adv Technol (IJEAT) 1(4):82–86

Google Scholar  

Chahar B, Jain SR, Hatwal V (2021) Mediating role of employee motivation for training, commitment, retention, and performance in higher education institutions. Probl Perspect Manag 19(3):95

Chauhan P (2019) Impact of training and development programs on motivation of employees in “A” graded commercial bank of Nepal. Int J Res Anal Rev 6(3):850–857

Elsafty A, Oraby M (2022) The impact of training on employee retention: an empirical research on the Private Sector in Egypt. Int J Bus Manage 17(5):58–74

Article   Google Scholar  

Habibie M, Mustika I (2020) The effect of training on work motivation and its impact on employee performance (Case Study at BPJS Ketenagakerjaan Headquarters). Int J Innovat Sci Res Technol 5(7):51–57

Haryono S, Supardi S, Udin U (2020) The effect of training and job promotion on work motivation and its implications on job performance: evidence from Indonesia. Manage Sci Lett 10(9):2107–2112

Horodyski P (2023a) Recruiter’s perception of artificial intelligence (AI)-based tools in recruitment. Comp Human Behav Reports 10:100298

Horodyski P (2023b) Applicants’ perception of artificial intelligence in the recruitment process. Comp Human Behav Reports 11:100303

Jack Walker H, Feild HS, Giles WF, Bernerth JB, Short JC (2011) ‘so what do you think of the organization? a contextual priming explanation for recruitment web site characteristics as antecedents of job seekers’ organizational image perceptions. Organ Behav Human Decis Process 2(2011):165–178

Khan N (2018) Does training & employee commitment predict employee retention. In: International Conference on Management and Information Systems (Vol. 21, pp. 120–124)

Lee I (2005) The evolution of E-Recruiting. A content analysis of Fortune 100 Career Websites. J Electronic Commerce Organ 3(3):57–68

Lievens F, Harris MM (2003) Research on Internet recruiting and testing: current status and future directions. Int Rev Ind Organ Psychol 16:131–165

Lin CY, Huang CK (2021) Employee turnover intentions and job performance from a planned change: the effects of an organizational learning culture and job satisfaction. Int J Manpow 42(3):409–423

Martins D, Diaconescu LM (2014) Expatriates recruitment and selection for long-term international assignments in Portuguese companies. Tékhne 12:48–57

RoyChowdhury T, Srimannarayana M (2013) Applicants’ perceptions on online recruitment procedures. Manage Labour Stud 38(3):185–199

Ryu G, Moon SG (2019) The effect of actual workplace learning on job satisfaction and organizational commitment: The moderating role of intrinsic learning motive. J Workplace Learn 31(8):481–497

Saaty TL (1990) How to make a decision: the analytic hierarchy process. Eur J Oper Res 48(1):9–26

Sengazhani Murugesan V, Sequeira AH, Shetty DS, Jauhar SK (2020) Enhancement of mail operational performance of India post facility layout using AHP. Int J Syst Assur Eng Manage 11(2):261–273

Sigalingging H, Pakpahan ME (2021) The Effect of training and work environment on employee performance with motivation as an intervening variable At PT Intraco Agroindustry. South East Asia Journal of Contemporary Business. Econ Law 24(6):130–139.

Sharawat K, Dubey SK (2018) An approach to vendor selection on usability basis by AHP and fuzzy topsis method. In: Soft computing: theories and applications: proceedings of SoCTA 2016, vol 2. Springer, Singapore, pp 595–604

Steil AV, de Cuffa D, Iwaya GH, Pacheco RCDS (2020) Perceived learning opportunities, behavioral intentions and employee retention in technology organizations. J Work Learn 32(2):147–159

Sugiarti E (2022) The influence of training, work environment and career development on work motivation that has an impact on employee performance at PT. Suryamas Elsindo Primatama In West Jakarta. Int J Artif Intell Res 6(1.2).

Sumrit D (2020) Supplier selection for vendor-managed inventory in healthcare using fuzzy multi-criteria decision-making approach. Decis Sci Lett 9(2):233–256

Thompson LF, Braddy PW, Wuensch KL (2008) E-recruitment and the benefits of organizational web appeal. Comp Human Behav 24(5):2384–2398

Turan FK, Scala NM, Sacre MB, Needy KL (2009) An Analytic Network Process (ANP) approach to the project portfolio management for Organizational Sustainability. In: Proceedings of the Industrial Engineering Research Conference. Institute of Industrial Engineers.

Wadhawan S, Sinha S (2018) Factors Influencing Young Job Seekers Perception towards Job Portals. AIMS Int J Manage 12(3).

Weerarathna RS, Somawardana WSD (2021) Impact on training and employee motivation in an electricity company. Future Work, 497.

Zusman R, Landis R (2002) Applicant preferences for Web-based versus traditional job postings. Comput Hum Behav 18(3):285–329

Download references

Acknowledgements

The authors would like to express their gratitude to two wheeler Automotive Industries in Chennai, Tamil Nadu, India, for their invaluable assistance and cooperation. We greatly acknowledge Ms. Ruchi Mishra, Research scholar from NIT Karnataka, for editing this manuscript in better form.

There is no funding provided in this research.

Author information

Authors and affiliations.

Operations Management Division, Vellore Institute of Technology Chennai, Vandalur-Kelambakkam Road, Chennai, 600127, India

S. M. Vadivel

Vellore Institute of Technology Chennai, Vandalur-Kelambakkam Road, Chennai, 600127, India

Rohan Sunny

You can also search for this author in PubMed   Google Scholar

Contributions

S M Vadivel: Methodology, Writing—review & editing, Supervision. Rohan Sunny: Data Curation, Writing—original draft preparation.

Corresponding author

Correspondence to S. M. Vadivel .

Ethics declarations

Conflict of interests.

The authors declare that they have no competing interests.

Ethics approval and consent to participate

This manuscript has a research study involves human participants (Interview Candidates) for studying job portal evaluations in Indian two wheeler company running in Chennai, Tamil Nadu.

Consent for publication

‘ Not applicable’ in this section.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Vadivel, S.M., Sunny, R. Evaluation of online job portals for HR recruitment selection using AHP in two wheeler automotive industry: a case study. Int J Syst Assur Eng Manag (2024). https://doi.org/10.1007/s13198-024-02358-z

Download citation

Received : 04 December 2023

Revised : 07 April 2024

Accepted : 25 April 2024

Published : 12 May 2024

DOI : https://doi.org/10.1007/s13198-024-02358-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Analytical hierarchy process (AHP)
• Online job portals
• Automobile two wheeler industries
• Artificial intelligence (AI)
• HR recruitment
• Find a journal
• Publish with us
• Track your research