research on quality management

From total quality management to Quality 4.0: A systematic literature review and future research agenda

• Review Article
• Open access
• Published: 13 March 2023
• Volume 10 , pages 191–205, ( 2023 )

Cite this article

You have full access to this open access article

• Hu-Chen Liu 1 ,
• Ran Liu 1 ,
• Xiuzhu Gu 2 &
• Miying Yang 3  

2976 Accesses

9 Citations

Explore all metrics

Quality 4.0 is an emerging concept that has been increasingly appreciated because of the intensification of competition, continually changing customer requirements and technological evolution. It deals with aligning quality management practices with the emergent capabilities of Industry 4.0 to improve cost, time, and efficiency and increase product quality. This article aims to comprehensively review extant studies related to Quality 4.0 to uncover current research trends, distil key research topics, and identify areas for future research. Thus, 46 journal articles extracted from the Scopus database from 2017 to 2022 were collected and reviewed. A descriptive analysis was first performed according to the year-wise publication, sources of publication, and research methods. Then, the selected articles were analyzed and classified according to four research themes: Quality 4.0 concept, Quality 4.0 implementation, quality management in Quality 4.0, and Quality 4.0 model and application. By extracting the literature review findings, we identify the Quality 4.0 definitions and features, develop the quality curve theory, and highlight future research opportunities. This study supports practitioners, managers, and academicians in effectively recognizing and applying Quality 4.0 to enhance customer satisfaction, achieve innovation enterprise efficiency, and increase organizational competitiveness in the era of Industry 4.0.

Article PDF

Download to read the full article text

Similar content being viewed by others

Customer experience: fundamental premises and implications for research

Larissa Becker & Elina Jaakkola

The customer value proposition: evolution, development, and application in marketing

Adrian Payne, Pennie Frow & Andreas Eggert

A Review on Quality of Service and SERVQUAL Model

Avoid common mistakes on your manuscript.

Ali K, Johl S K (2022). Soft and hard TQM practices: Future research agenda for Industry 4.0. Total Quality Management & Business Excellence, 33(13–14): 1625–1655

Article   Google Scholar  

Alzahrani B, Bahaitham H, Andejany M, Elshennawy A (2021). How ready is higher education for Quality 4.0 transformation according to the LNS research framework? Sustainability, 13(9): 5169

Antony J, McDermott O, Sony M (2022a). Quality 4.0 conceptualisation and theoretical understanding: A global exploratory qualitative study. TQM Journal, 34(5): 1169–1188

Antony J, Sony M, Furterer S, McDermott O, Pepper M (2022b). Quality 4.0 and its impact on organizational performance: An integrative viewpoint. TQM Journal, 34(6): 2069–2084

Antony J, Sony M, McDermott O, Jayaraman R, Flynn D (2023). An exploration of organizational readiness factors for Quality 4.0: An intercontinental study and future research directions. International Journal of Quality & Reliability Management, 40(2): 582–606

Asif M (2020). Are QM models aligned with Industry 4.0? A perspective on current practices. Journal of Cleaner Production, 258: 120820

Balouei Jamkhaneh H, Shahin A, Parkouhi S V, Shahin R (2022). The new concept of quality in the digital era: A human resource empowerment perspective. TQM Journal, 34(1): 125–144

Baran E, Korkusuz Polat T (2022). Classification of Industry 4.0 for total quality management: A review. Sustainability, 14(6): 3329

Broday E E (2022). The evolution of quality: From inspection to Quality 4.0. International Journal of Quality and Service Sciences, 14(3): 368–382

Bui L T C, Carvalho M, Pham H T, Nguyen T T B, Duong A T B, Truong Quang H (2022). Supply chain quality management 4.0: Conceptual and maturity frameworks. International Journal of Quality & Reliability Management, in press, doi: https://doi.org/10.1108/IJQRM-07-2021-0251

Chiarini A (2020). Industry 4.0, quality management and TQM world: A systematic literature review and a proposed agenda for further research. TQM Journal, 32(4): 603–616

Chiarini A, Kumar M (2022). What is Quality 4.0? An exploratory sequential mixed methods study of Italian manufacturing companies. International Journal of Production Research, 60(16): 4890–4910

Christou I T, Kefalakis N, Soldatos J K, Despotopoulou A M (2022). End-to-end industrial IoT platform for Quality 4.0 applications. Computers in Industry, 137: 103591

Clancy R, Bruton K, O’Sullivan D T J, Cloonan A J (2022). The HyDAPI framework: A versatile tool integrating Lean Six Sigma and digitalisation for improved quality management in Industry 4.0. International Journal of Lean Six Sigma, in press, doi: https://doi.org/10.1108/IJLSS-12-2021-0214

Di Vaio A, Hassan R, Alavoine C (2022). Data intelligence and analytics: A bibliometric analysis of human-artificial intelligence in public sector decision-making effectiveness. Technological Forecasting and Social Change, 174: 121201

Dias A M, Carvalho A M, Sampaio P (2022). Quality 4.0: Literature review analysis, definition and impacts of the digital transformation process on quality. International Journal of Quality & Reliability Management, 39(6): 1312–1335

Elibal K, Özceylan E (2022). Comparing Industry 4.0 maturity models in the perspective of TQM principles using Fuzzy MCDM methods. Technological Forecasting and Social Change, 175: 121379

Emblemsvåg J (2020). On Quality 4.0 in project-based industries. TQM Journal, 32(4): 725–739

Escobar C A, McGovern M E, Morales-Menendez R (2021). Quality 4.0: A review of big data challenges in manufacturing. Journal of Intelligent Manufacturing, 32(8): 2319–2334

Fonseca L, Amaral A, Oliveira J (2021). Quality 4.0: The EFQM 2020 model and Industry 4.0 relationships and implications. Sustainability, 13(6): 3107

Gebhardt M, Kopyto M, Birkel H, Hartmann E (2022). Industry 4.0 technologies as enablers of collaboration in circular supply chains: A systematic literature review. International Journal of Production Research, 60(23): 6967–6995

Glogovac M, Ruso J, Maricic M (2022). ISO 9004 maturity model for quality in Industry 4.0. Total Quality Management & Business Excellence, 33(5–6): 529–547

Hyun Park S, Seon Shin W, Hyun Park Y, Lee Y (2017). Building a new culture for quality management in the era of the Fourth Industrial Revolution. Total Quality Management & Business Excellence, 28(9–10): 934–945

Jacob D (2017). What is Quality 4.0? Online Blog

Kannan K S P N, Garad A (2021). Competencies of quality professionals in the era of Industry 4.0: A case study of electronics manufacturer from Malaysia. International Journal of Quality & Reliability Management, 38(3): 839–871

Maganga D P, Taifa I W R (2022). Quality 4.0 transition framework for Tanzanian manufacturing industries. TQM Journal, in press, doi: https://doi.org/10.1108/TQM-01-2022-0036

Maganga D P, Taifa I W R (2023). Quality 4.0 conceptualisation: An emerging quality management concept for manufacturing industries. TQM Journal, 35(2): 389–413

Nenadál J, Vykydal D, Halfarová P, Tylečková E (2022). Quality 4.0 maturity assessment in light of the current situation in the Czech Republic. Sustainability, 14(12): 7519

Prashar A (2022). Quality management in Industry 4.0 environment: A morphological analysis and research agenda. International Journal of Quality & Reliability Management, in press, doi: 10.1108/IJQRM-10-2021-0348

Ramezani J, Jassbi J (2020). Quality 4.0 in action: Smart hybrid fault diagnosis system in plaster production. Processes, 8(6): 634

Ranjith Kumar R, Ganesh L S, Rajendran C (2022). Quality 4.0: A review of and framework for quality management in the digital era. International Journal of Quality & Reliability Management, 39(6): 1385–1411

Sader S, Husti I, Daróczi M (2019). Industry 4.0 as a key enabler toward successful implementation of total quality management practices. Periodica Polytechnica Social and Management Sciences, 27(2): 131–140

Sader S, Husti I, Daroczi M (2022). A review of Quality 4.0: Definitions, features, technologies, applications, and challenges. Total Quality Management & Business Excellence, 33(9–10): 1164–1182

Saihi A, Awad M, Ben-Daya M (2023). Quality 4.0: Leveraging Industry 4.0 technologies to improve quality management practices, a systematic review. International Journal of Quality & Reliability Management, 40(2): 628–650

Santos G, Sá J C, Félix M J, Barreto L, Carvalho F, Doiro M, Zgodavová K, Stefanović M (2021). New needed quality management skills for quality managers 4.0. Sustainability, 13(11): 6149

Sariyer G, Mangla S K, Kazancoglu Y, Ocal Tasar C, Luthra S (2021). Data analytics for quality management in Industry 4.0 from an MSME perspective. Annals of Operations Research, in press, doi: https://doi.org/10.1007/s10479-021-04215-9

Silva C S, Borges A F, Magano J (2022). Quality Control 4.0: A way to improve the quality performance and engage shop floor operators. International Journal of Quality & Reliability Management, 39(6): 1471–1487

Singh J, Ahuja I P S, Singh H, Singh A (2022). Development and implementation of Autonomous Quality Management System (AQMS) in an automotive manufacturing using Quality 4.0 concept: A case study. Computers & Industrial Engineering, 168: 108121

Sony M, Antony J, Douglas J A (2020). Essential ingredients for the implementation of Quality 4.0: A narrative review of literature and future directions for research. TQM Journal, 32(4): 779–793

Sony M, Antony J, Douglas J A, McDermott O (2021). Motivations, barriers and readiness factors for Quality 4.0 implementation: An exploratory study. TQM Journal, 33(6): 1502–1515

Souza F F, Corsi A, Pagani R N, Balbinotti G, Kovaleski J L (2022). Total quality management 4.0: Adapting quality management to Industry 4.0. TQM Journal, 34(4): 749–769

Sozinova A A, Saveleva N K (2022). Marketing quality management in Industry 4.0 in transborder markets. International Journal of Qualitative Research, 16(3): 955–968

Sureshchandar G S (2022). Quality 4.0: Understanding the criticality of the dimensions using the analytic hierarchy process (AHP) technique. International Journal of Quality & Reliability Management, 39(6): 1336–1367

Sureshchandar G S (2023). Quality 4.0: A measurement model using the confirmatory factor analysis (CFA) approach. International Journal of Quality & Reliability Management, 40(1): 280–303

Tambare P, Meshram C, Lee C C, Ramteke R J, Imoize A L (2022). Performance measurement system and quality management in data-driven Industry 4.0: A review. Sensors, 22(1): 224

Tan T, Mills G, Papadonikolaki E, Liu Z (2021). Combining multi-criteria decision making (MCDM) methods with building information modelling (BIM): A review. Automation in Construction, 121: 103451

Thekkoote R (2022). Enabler toward successful implementation of Quality 4.0 in digital transformation era: A comprehensive review and future research agenda. International Journal of Quality & Reliability Management, 39(6): 1368–1384

Yadav N, Shankar R, Singh S P (2021). Critical success factors for Lean Six Sigma in Quality 4.0. International Journal of Quality and Service Sciences, 13(1): 123–156

Zheng T, Ardolino M, Bacchetti A, Perona M (2021). The applications of Industry 4.0 technologies in manufacturing context: A systematic literature review. International Journal of Production Research, 59(6): 1922–1954

Zonnenshain A, Kenett R S (2020). Quality 4.0: The challenging future of quality engineering. Quality Engineering, 32(4): 614–626

Download references

Author information

Authors and affiliations.

School of Economics and Management, Tongji University, Shanghai, 200092, China

Hu-Chen Liu & Ran Liu

Department of Industrial Engineering and Economics, Tokyo Institute of Technology, Tokyo, 152-8552, Japan

Group of Sustainability, School of Management, Cranfield University, Cranfield, MK43 0AL, UK

Miying Yang

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Ran Liu .

Additional information

This work was partially supported by the major project of National Social Science Fund of China (Grant No. 21ZDA024).

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Liu, HC., Liu, R., Gu, X. et al. From total quality management to Quality 4.0: A systematic literature review and future research agenda. Front. Eng. Manag. 10 , 191–205 (2023). https://doi.org/10.1007/s42524-022-0243-z

Download citation

Received : 31 July 2022

Accepted : 01 November 2022

Published : 13 March 2023

Issue Date : June 2023

DOI : https://doi.org/10.1007/s42524-022-0243-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• quality management
• Quality 4.0
• Industry 4.0
• literature review
• predictive quality

Advertisement

• Find a journal
• Publish with us
• Track your research

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 09 July 2019

Management at the service of research: ReOmicS, a quality management system for omics sciences

• Antonella Lanati 1   na1 ,
• Marinella Marzano 2   na1 ,
• Caterina Manzari 2 ,
• Bruno Fosso 2 ,
• Graziano Pesole 2 , 3 &
• Francesca De Leo   ORCID: orcid.org/0000-0003-0421-7699 2  

Palgrave Communications volume  5 , Article number:  75 ( 2019 ) Cite this article

7508 Accesses

2 Citations

3 Altmetric

Metrics details

• Business and management
• Science, technology and society

Management and research represent a binomial almost unknown, whose potentialities and requirements have not yet been fully exploited even if, recently, the scientific and social communities have felt the burden of producing results and data requiring at the same time reproducibility, reliability, safety and efficacy of the discoveries, as well as a profitable use of resources. A Quality Management System (QMS) could represent a valid tool for these purposes, improving the quality of the research. The research community could ask whether and how it is possible to apply this approach in a research laboratory without hindering their creativity, and what the possible benefits might be. On the other hand, an international standard for a quality management system appropriate for a research laboratory is yet to come. The choice, the design and the application of a QMS, inspired by the Good Laboratory Practices, in a research laboratory specialized on “omics” sciences, is fully described in this paper. Its application has already shown good outcomes as testified by specific metric of efficiency and effectiveness. The approach is innovative as there is no obvious requirement for research laboratories to develop and define quality objectives. The paper highlights how the QMS approach enhances the relationship with public and private sectors by increasing customer confidence and loyalty, as well as improving the overall performance of the laboratory in terms of throughput and value of research. These results encourage proposing it as a QMS model providing a new and scalable operational strategy to be applied in a research environment with the same target and even in a generic research laboratory.

Similar content being viewed by others

Considerations for implementing electronic laboratory notebooks in an academic research environment

Stuart G. Higgins, Akemi A. Nogiwa-Valdez & Molly M. Stevens

Reproducible, scalable, and shareable analysis pipelines with bioinformatics workflow managers

Laura Wratten, Andreas Wilm & Jonathan Göke

SciSciNet: A large-scale open data lake for the science of science research

Zihang Lin, Yian Yin, … Dashun Wang

Introduction

Next Generation Sequencing (NGS) technologies have dramatically changed the field of genomics and are routinely applied to a variety of functional genomics investigations including, but not restricted to, whole genome sequencing, global identification of genomic rearrangements, epigenetic modifications, single nucleotide polymorphism (SNP) discovery, transcriptome profiling and metagenomics. In recent years, using these technologies thousands of genomes assembled from short DNA sequence readings of humans, plants, animals and microbes have been collected and explored, enabling scientists to develop a deeper understanding and gaining new insights into the molecular mechanisms related to different diseases, including many types of cancer, allergies, or other disorders (Wiese et al., 2018 ). Furthermore, the genomics has profoundly influenced the pharmaceutical industry and reshaped the processes allowing to discover, investigate, and develop new drugs. No less important is the research carried out using these technologies in the environmental field for industrial and biotechnological purposes (Tiwari et al., 2018 ). Indeed, NGS is a complex process that, on the one hand, requires the preparation of sequencing libraries that respond to the specific standard requirements of the platforms, and on the other hand generates unprecedented volumes of data to be analyzed (Stephens et al., 2015 ). Moreover, an NGS analysis usually involves collaboration between several departments, laboratories and data analysis groups, characterized by different scientific backgrounds and, above all, applying different experimental approaches. With the growing need of managing information, it has become challenging to keep track of data, processes and outcomes of research over long periods of time and across the collaborating units. Nowadays Big Data generation and its management create extraordinary challenges for storage, transfer, analysis, interpretation and last but not least security of information. Regarding this last aspect, the scientific community has recently expressed the necessity to manage NGS data according to the principles of Findability, Accessibility, Interoperability, and Re-usability of digital assets (FAIR data) (Wilkinson et al., 2016 ; Corpas et al., 2018 ). The expectation is to produce digital resources with more rigorous management and stewardship that can be used by the entire scientific community. Good data management is not a goal in itself, but rather it is the key conduit leading to discovery and innovation, through data integration and reuse by the scientific community after the publication process. Good data management, ensuring reliability and usability, needs a holistic approach tracking the process of data and metadata generation and all the different organizational aspect that, on the one hand may affect it, and on the other hand can keep it under control. This can be achieved by means of a management system focused on the quality of the results.

Furthermore, in recent years, in the context of the scientific research, we are witnessing a new phenomenon, defined as “reproducibility crisis” by Baker (Baker, 2016 ) and Dirnagl et al. (Dirnagl et al., 2018 ) characterized by the reduction of the reliability, reproducibility, traceability and predictability of research results. These problems can not only compromise the robustness and rigor of research (Dirnagl et al., 2018 ) but have also a significant impact from an economic point of view, reducing the profitability of research funds (Lanati, 2018 ). The standardization and simplification of experimental workflows, such as those applied for “omics” applications, is becoming a need both for academic and private research laboratories. As described by Endrullat et al. ( 2016 ), standards act as basic guidelines to ensure comparability and exchange of experimental data conducive to the acceleration of the innovation process, aiding improvement of transferability, transparency and reproducibility of results. Furthermore, the advantages deriving from the standardization of processes could reduce costs and increase services (Endrullat et al., 2016 ; Cargill, 2011 ). A Quality Management System (QMS) can support the correct management of the NGS research environment, providing directions for data and operations management. A suitable quality system ensures safety, reliability and reproducibility of the non-clinical tests on chemicals intended for use on humans, animals and the environment. A QMS supports the generation of high quality scientific data and associated services, it is also helpful in improving the economic and social impacts of research. Quality research management reinforces scientific communities and improves the attractiveness and effectiveness of the service. In a QMS activities are properly planned and documented, operations are regulated by means of standard operating procedures, and the correct behavior, compliant with internal and external standards, is guaranteed by regular inspections. A QMS can help in giving proper attention to sensitive data and in correctly managing them, setting internal standard, organizing rules and forms and maintaining due control. Good Research Practices, as a quality management standard dedicated to the research environment, are at present not yet organized in an international reference text and consist of different prescriptive documents that are drafted and/or personalized by each research institution interested in aspects of quality management. Researchers can only refer to the WHO Handbook of Quality in Biomedical Research (WHO, 2006 ) as a guiding text to comply with generic quality principles. However, several references for designing a QMS can be found among international standards suitable for the management of a research laboratory: ISO 9001:2015, the most general quality management standard; ISO 17025, derived from the ISO9001 and dedicated to test and calibration laboratories; and the Good Laboratory Practice, mandatory international reference for development and testing of drugs and other substances intended for human and animal use (Lanati, 2018 ).

In this paper, we describe the choice, the design and the application of a QMS, called ReOmicS (Research Environment management system for Omics Sciences), at the Molecular Biodiversity Laboratory (MoBiLab), a NGS research infrastructure located in Bari (Italy) at the Institute of Biomembranes, Bioenergetics and Molecular Biotechnologies of CNR (CNR-IBIOM). MoBiLab is a research environment, fully equipped with operative platforms based on the most innovative NGS technologies and powerful resources for data storage and computational analysis, whose mission is to contribute to innovation with original studies. Moreover, CNR-IBIOM is involved in the construction of the national nodes of ELIXIR and LifeWatch Research Infrastructures (included in the ESFRI Roadmap), as well as in a substantial empowering of its infrastructural components for omics data production and analysis consistently scaling up the available instruments and facilities. The described experience could represent a scalable model to be applied to MoBiLab and to other research laboratories in order to ensure the highest levels of reliability, reproducibility and traceability of the results, a process that is also expected to foster their potential exploitation.

Material and methods

Considering the managerial aspects of the study, methods described in this section are tools used in quality and organizational management, occasionally modified to be adopted for the specific use of a research laboratory, such as MoBiLab.

Decision grid

The decision grid (or matrix) is a tool that supports a decision among many options. Once the aim of the decision is clearly defined, the criteria used to characterize each solution must be identified. Each criterion is given a weight (1 = lowest to 5 = highest) based on its importance in the final decision. The selected options among which the choice has to be made is then assessed with respect to their suitability to each criterion (1 = lowest to 5 = highest). The sum of the weighted assessments gives the final score for each proposal. Referring to Table 1 , the head of the table defines the aim of the decision. The options are listed in the columns. Criteria are listed in the rows and weighted in importance in column W. Each option is given a specific assessment (column A) with respect to the relative criterion, and an overall score, which is the product of the importance of the criterion and the given assessment. The final score for the proposal is given by the sum of all scores for each criterion.

SWOT analysis

The SWOT analysis represents, within a rationale, the influence exerted by some key factors on a goal in order to identify actions that reinforce the positive factors and counteract the influence of negative factors. The key factors in the analysis, whose initials give the name to the technique, are:

Strength: a resource that can be used to best achieve the goal;

Weakness: an obstacle to achieving the goal;

Opportunity: a favorable situation in the external context that favors the achievement of the objective;

Threat: an external, unfavorable situation in the external context that potentially hinders the achievement of the objective.

The analysis combines internal factors (strengths and weaknesses) and external factors (threats and opportunities), as well as positive aspects (strengths, opportunities) and negative aspects (weaknesses, threats). In this way SWOT Analysis allows defining strategies aimed at capitalizing on strengths, eliminating weaknesses, exploiting opportunities and mitigating threats.

Risk assessment

A risk assessment was performed on the main analytical process, according to the requirements of an ISO High Level Structure (HLS) and the ISO 9001:2015 standard. For each experimental step (Fig. 1 , first column), some pitfalls were identified (second column); each pitfall was assessed with respect to its Severity S (from 1 = low to 3 = high) i.e., how serious would be the consequence of an error on the final result, and Probability P (from 1 = low to 3 = high), i.e., how frequently a specific mistake has recently occurred (column 3 and 4, respectively). The Risk R in column 5 was then calculated as S × P for each pitfall identified. Risk values range from 9 (greatest) to 1 (lowest). Operations with risk R greater or equal to 4 are judged worthy of specific interventions to prevent errors, as recorded in column 6 “Solution”. Interventions are prioritized according to the level of risk. Referring to the legend of Fig. 1 , colors indicate the need for improvement actions: red for urgent, orange for medium and yellow for minor need, while green indicates no need for action.

Risk assessment of the primary process at MoBiLab before the introduction of the ReOmicS. Pitfalls are placed in descending order according to R. Colors indicate the need for improvement actions: red for urgent, orange for medium and yellow for slight need, while green indicates no need for action

The SIPOC diagram, first outlined by Juran (Defeo and Juran, 2010 ) calling it TRIPOL, was then employed in the Six Sigma approach for analyzing a process. It is named SIPOC from the acronym of Supplier, Input, Process, Output, and Customer: the key elements of a process. A flowchart of the process is usually inserted in the third column “process” and for each step: input supplier, input needed by the operation, output of the operation and recipient of the output (customer) are listed.

The SIPOC-like flowchart is structured on the following categories:

source: the origin of the input

input: raw data, metadata, materials or samples needed by the study activities

process: steps of analysis and controls, logically linked

supervisor/person in charge: the supervisor and/or the person in charge to carry out each task of the previous column

output: the result/product of each task

procedure: the SOP describing the specific task

Due to the limited availability of data for the period preceding the introduction of the QMS, the metrics system has been necessarily simplified to two indicator of efficiency and three indicators of effectiveness:

Efficiency indicator: analysis throughput. The values of four parameters (total number of processed samples, the average number of samples, the number of sequencing runs and the sequencing platform output), related both to all the MoBiLab applications ( a ) and only to Metagenomics ( b ), referring to the 3-year periods (2013–15 and 2016–2018), are shown

efficiency:

outcome of the updated risk assessment, compared with the initial one performed in designing the QMS. With respect to the initial Risk Assessment, three new columns have been added: “New-Probability” with the updated probability of the pitfall(s) considered, “New risk Assessment” with the updated value for the parameter R = SxP, and “Audit” recording reasons and considerations regarding the improvement. A paired one-sided Wilcoxon test was performed, to verify whether the risk estimation prior to the QMS adoption was significantly higher than the next.

evaluation of the analysis throughput in terms of the total sequencing run number, the total number of processed samples, the average number of samples per sequencing run and the run output (Gb). For each parameter, the values related to the two three-year periods were collected. The analyses were carried out on both the data derived from all the MoBiLab applications (Genomics, Transcriptomics and Metagenomics) (Fig. 2a ) and those produced only from Metagenomics (Fig. 2b ).

effectiveness:

number of publications and the related impact factor (base: 3-year period): we considered the peer-reviewed publications of three researchers 100% involved in MoBiLab research projects (Source: JCR-ISI Web of Knowledge; https://login.webofknowledge.com ). In case of co-authorship the journal was counted once, and the number of publications within the period has been calculated considering the average number of the papers per year. The scientific areas of the journals are Biochemistry, Genetics and Molecular Biology Medicine Agricultural and Biological Sciences, Immunology and Microbiology, Multidisciplinary Environmental Science, Neuroscience, Computer science, Mathematics.

scientific attractiveness , i.e., number of active external collaborations in MoBiLab publications (base: 3-year period) (Source: PubMed-National Library of Medicine; https://www.nlm.nih.gov/bsd/pubmed.html ): author’s affiliation to the papers published in the first 3-year period 2013–15 (before the introduction of the QMS) are compared with author’s affiliation referring to the 3-year period 2016–2018, after the progressive introduction of the QMS.

satisfaction survey: two separate surveys were prepared using the online tool SurveyMonkey ( https://it.surveymonkey.com ). The first (B9M5SNL), dedicated to all customers/collaborators about perceived quality, was sent by mail to 54 MoBiLab collaborators. The second (BDDF65F) was sent, in addition to the first, only to those customers/collaborators (25/54) who worked with MoBiLab in both three-year periods before and after the introduction of the QMS. A two week deadline was given. The analysis results were provided by the tool and further analyzed and elaborated by the team (Supplementary Material (SM) 1 and 2 ).

The indicators of efficiency measure the ability of the MoBiLab to increase productivity and reduce costs, while the effectiveness indicators show the quality and importance of the analysis of results.

Results and discussion

Choosing the qms standard.

To choose the best reference standard for the characteristics of MoBiLab, we compared three international standards: ISO 9001:2015, ISO 17025, and GLPs by means of a decision grid. The criteria for this choice were identified as:

compatible with regulation environment

suitable for customer’s requirements

oriented to R&D

focused on analytical process

low management costs

easy to fit in Laboratory activities

not linked to external third parties

suitable for expansion

The results are illustrated in Table 1 . Evaluating the criteria for choice, we considered that the “customers” of the MoBiLab research services are laboratories already working under the principles of GLPs and they could benefit from a rigorous and standardized work environment for the production of their data, as well as from a common management language and references. All criteria are listed in the first column of Table 1 . The GLP obtained the best assessment weighted on the importance of each criterion, mainly for their suitability for customer’s requirements, the lower cost, the independence from third party evaluations and the opportunity for development.

GLPs are mandatory in OECD countries for preclinical tests, but should be also considered as a reference for laboratory management systems, that can be referred to as “GLP-like” quality systems, although outside GLPs main scope. As textually described by Kauffmann et al. ( 2017 ) the application of GLPs principles to “omics” studies based on NGS, in a regulatory context, would serve the following goals (i) to promote the consistent quality and validity of data used for determining the safety of chemical products—a primary objective of the GLP principles (OECD, 1998 ); (ii) to promote transparent process descriptions and thus support the traceability of study results; and (iii) to facilitate the exchange of information and enhance the regulatory impact of ‘ omics data, when successfully used for hazard and risk assessment purposes. The use of GLP standard system helps the management of the human genomic data sharing respecting the privacy of the data, the reuse of the data under the current legislation and any further integration.

Design of ReOmicS

A director, three researchers, a technician and a quality consultant have been the working group that has met fortnightly via videoconference. The team assessed the choice of the GLPs as main reference for ReOmicS by means of a SWOT analysis, whose results are illustrated in Fig. 3 . Strengths and opportunities validated the choice of following GLPs as main reference against minor weaknesses and threats. In any case, as illustrated in what follows, some actions have been planned and taken in order to tackle some weaknesses and threats in a near future.

SWOT analysis diagram. Strength, weaknesses, opportunities and threats have been evaluated for the choice of Good Laboratories Practices (GLPs) as QMS reference

The team then started the design of ReOmicS from the risk analysis of the main NGS process, based on a standard Risk Assessment. With this analysis, the team aimed at identifying the laboratory’s weaknesses in order to address them with specific interventions when developing the QMS.

Errors and problems experienced during the last 3 years were collected and attributed to the relevant process steps. Each problem was then assessed with respect to severity S and probability P. The risk R = S × P associated with each pitfall determined the need and the priority of a specific operating procedure. Figure 1 summarizes these results.

Major problems to be addressed when defining Standard Operating Procedures (SOPs) were identified in the fields of communication with the customer; traceability of samples; warehousing, archive, metadata and database management; planning; organizational structure; and controls and checks. No needs for specific instructions were envisaged for problems with R less than 4. Major needs and related priority were taken into account in the drafting phase of the SOPs.

As a first step, in order to define the internal context, the MoBiLab organizational structure was designed according to the GLP requirement taking into account the dimension and the constraints of the research institution: the major roles, such as Director, Study Director, Laboratory Manager, Principal Investigator, Archivist et al, were identified and assigned to laboratory staff (Fig. 4 ). Once defined the roles and responsibilities, the team analyzed the laboratory internal processes, identifying primary and support processes (see Table 2 ). The team outlined the main (primary) process by means of a Supplier-Input-Process-Output-Customer (SIPOC)-like flowchart, which includes the person in charge of the activity and the related documented information (SOPs, and records) (see Fig. 5 ).

Organizational Chart. Under the supervision of the MoBiLab director, resources are divided into two main groups: (i) managerial and technical support and (ii) research and experimentation

SIPOC-like flowchart of the Study management. Columns collect inputs and related providers, main flow-chart with tasks, person in charge/supervisor for every task, output and prescriptive document (SOP) for each task. The flowchart is divided into two main sections: Execution/experimentation and Analysis/results

This SIPOC-like chart acted as the backbone of the SOP of the management of the Study: moreover, most steps of the study process, together with the results of the risk assessment, led to the identification of technical/scientific procedures , these were then accompanied by attachments describing technical details as required. After having identified the SOPs required for the operational processes , researchers were provided with a template and with the instructions to draft them. In parallel, management procedures were defined by the whole team, drafted, and supported by flowcharts and other quality tools whenever needed (e.g., SIPOC). SOPs were ranked by priority, driven by the risk assessment results; few SOPs required by GLPs were not considered since were not needed for our specific research activities (e.g., management of test systems). The management support processes were described by SOPs and were, as far as possible, compliant with GLP requirements.

Following the results of the SWOT analysis (where this aspect was judged as a weakness) and in order to ensure the highest level of reliability, reproducibility and traceability of the results, the team also planned to develop and optimize a LIMS (Laboratory Information Management System) platform for managing all the laboratory activities through a suite of integrated modules, in collaboration with an Italian ICT company. The platform will be structured starting from the SIPOC-like flowchart for the management of the study and the modules will be developed and customized in agreement with the SOPs.

The structure of the SOP list (Table 2 ) is directly related to the allocation of responsibilities in the laboratory and conforms to the organization of the management of the studies (Fig. 5 ): the primary and the scientific SOPs are the responsibility of the researchers and technicians operating in the MoBiLab under the supervision of a Principal Investigator, nominated by the director of the study. At the same time, management and technical support SOPs govern staff indirectly involved in the project, caring for an environment suitable for the studies. The split into two different areas, research and support, is clearly represented in the organization chart (Fig. 4 ).

Indeed, the MoBiLab belongs to a public research institution whose mission is to achieve scientific outputs in national and international funded project. Staff organization is related to the skills required and tasks assigned in the study program. For this reason, in the primary SOP the study corresponds to the project and the director of the study refers to the scientist responsible of the project. Only in a few cases the MoBiLab is working as a service provider, producing genomic and data analysis directly commissioned by external customers. For all these reasons, the scientific SOPs can evolve by integrating new requirements highlighted by customers or scientific partners.

Following all these considerations, the ReOmicS was structured as illustrated in Table 3 . Each SOP is structured according to a general template with the following sections:

-definitions, terms and acronyms

-references

-activities and responsibilities

-materials and equipment

-safety rules

-history of revisions (change register)

Results of the application of ReOmicS

The system has been progressively introduced, starting in 2016 from the scientific SOPs. At present the application of ReOmicS is underway with few limitations, due to the fact that processes and their management are challenging to pursue without IT support. This is also the reason that a LIMS is being developed. For example, audits have been conducted in limited form, mainly focusing on the suitability and effectiveness of the quality management system. Despite these limitations, the MoBiLab, in the 3-years following the introduction of common rules and references, is experiencing a concrete improvement, as testified by the positive trend of the five metrics chosen: risk assessment, analysis throughput, number/quality of publications, external collaborations and satisfaction survey.

The first metric chosen for the assessment of efficiency is the comparison between the outcomes of the risk evaluation before and after the introduction of ReOmicS (Fig. 6 ). The results show a statistically significant improvement ( p   =   0.0005 ) with respect to the initial assessment for the application of technical SOPs, while that of management SOPs is still limited. A better compliance to the QMS is expected in the future, mainly in the areas of study-planning, assignment of tasks and storage control.

Efficiency indicator. Efficiency is evaluated by means of the update of the risk assessment performed in the initial phase of the project, before the application of ReOmicS

A second metric has been chosen to evaluate the efficiency, i. e. the overall performance of MoBiLab, in terms of total number of processed samples, number of samples/sequencing runs, number of sequencing runs and sequencing platform output (GigaBase). The data shown in Fig. 2 compare the 3-year periods, 2013–2015 and 2016–2018. Considering all the MoBiLab applications (Genomic, Transcriptomics and Metagenomics), the analyzed parameters, except the number of sequencing run, show an improvement after the introduction of ReOmicS (Fig. 2a ). At the same time, it is important to underline that, despite the reduction in the number of sequencing runs due to a forced six months interruption of MoBiLab activities for logistical issues, the total number of samples processed and the average number of samples increased together with the platform output. In an NGS analysis, we can speculate that maintaining a high throughput whilst at the same time increasing the number of samples, represents an important laboratory challenge. These results can be ascribed to operator’s competences and training, but also to the improvement of the management of the process and the control of the analysis provided by the QMS. The efficiency of the laboratory was therefore assessed by taking into consideration the amount of data produced, referring mainly to Metagenomics analysis, the most requested application at MoBiLab (approximately 52% of the total amount of analysis performed) (Fig. 2a ). Overall, a positive increase was shown (Fig. 2b ) by all the parameters during the 3-year period 2016–2018.

To assess the influence of the QMS on the effectiveness of the MoBiLaB, three metrics have been chosen. The first one is the number of publications and the related Impact Factor (IF). In Table 4 , data referring to the first 3-year period 2013–15 (before the introduction of ReOmicS) are compared with data from the 3-year period after the progressive introduction of the QMS, 2016–2018. Table 4 also shows the average IF values obtained for each period. The lowest and the highest values of journal IF were excluded from the analysis. The QMS improved also downstream processes as demonstrated by the increased number of papers published in peer review journals. The number of published papers has grew from 13 to 23 in the last 3 years. Indeed, the number of published papers doubled even if the IF increase is not significant (data not shown). The second metric chosen for the assessment of effectiveness of ReOmicS, i.e., number of active external collaborations in MoBiLab publications (based on a 3-year period), shows the attitude of the laboratory to be a national and world leading scientific NGS laboratory and to be an enabling facility in the support of science. Table 4 shows the average number of external authors in the two 3-year periods. The total number of authors for each paper did not significantly change in the two periods considered, nor did the number of authors with an Italian affiliation. On the other hand, the number of general affiliations increased by a third and the number of international collaborations almost tripled.

The third effectiveness metric measures the satisfaction of customers and collaborators who had the opportunity to take advantage of the analysis service of MoBiLab, by means of two separate surveys: the first one dedicated to all customers/collaborators about perceived quality and the second to customers/collaborators who worked with MoBiLab in both three-year periods before and after the introduction of the QMS. The first survey was sent to 54 collaborators and 25 answers were collected. The second survey was sent to 25 collaborators, obtaining 12 answers. Of these last 12, 6 were discarded for inconsistency in answers to single questions, for this reason only a qualitative evaluation can be made. Results of both surveys are illustrated in Fig. 7 and show a good level of satisfaction from customers and collaborators, together with a demonstrable improvement of perceived quality after the introduction of ReOmicS.

Customer satisfaction. A qualitative measure of the customer satisfaction was evaluated analyzing the results obtained from the two surveys: the first one dedicated to all customers/collaborators about perceived quality ( a ) and the second to customers/collaborators who worked with MoBiLab in both three-year periods before and after the introduction of the QMS ( b )

As a final consideration on metrics, it was difficult to gather complete and detailed data regarding the projects developed in the years 2013–2015 for comparison with those pertaining to the years 2016–2018 because, before the ReOmicS introduction, a lot of the information was scattered among different research environments within and outside the MoBiLab. Since this data unavailability was judged unacceptable in maintaining due control on the work of the laboratory and on the improvement process, a more complete set of metrics has been studied which will be integrated in the planned LIMS (Table 5 ).

Main deviations from GLPs

Not all the requirements of GLP can be accomplished in the development of ‘omics’ studies, as clearly shown by Kauffmann et al. ( 2017 ). The limitations involve technical aspects, but in our case have had an impact also on the organizational requirements.

The first requirement that cannot be met is the management of test systems, because in the NGS procedures they are not used. External databases are used as reference and these are validated by the well-known mechanism of peer review. This is in partial disagreement with the GLP direction about data management and validation, but is common practice in genomics.

As far as data storage is concerned, MoBiLab depends on the servers made available by the INFN. The commercial agreement with INFN is stipulated by the IBIOM Institute: so far MoBiLab is not in the position to insert specific GLP requirements. With the planned development of the LIMS, new conditions and agreements more suitable for GLP compliance about data management will be implemented.

As an example of GLP procedure requirements that needed a new definition, the compliance statement required by the GLP is intended not towards the GLP, but to ReOmicS QMS itself.

As an example of organizational GLP requirements that could not be met, the dimension of the research unit and specifically of the laboratory is an issue when trying to identify an independent quality assurance structure. However, within the laboratory, a person has been appointed for the quality assurance tasks described in the dedicated SOP with support from an external quality consultant for methodological matters or concerns. Furthermore, the title of Principal Investigator (PI), which in the GLP is an individual who, for a multi-site study, acts on behalf of the Study Director, is known here as “research project manager”, as in the most common meaning for research laboratories working on funded projects.

Moreover, the role of archivist has not yet been allocated, since the planned introduction of a LIMS will ease the task of archiving and will allow a clearer allocation of responsibilities.

Conclusions and future perspectives

We can demonstrate that the application of a QMS, giving precise references for research management, introducing controls—thus increasing both result reliability and reducing opportunities for error—and promoting the efficiency in planning, conducting, analyzing and reporting on the processes, represents a valid tool for overcoming these difficulties and, at the same time, an opportunity for significant improvement for a research laboratory.

In the experience illustrated, GLPs—among different quality management standards—was judged most suitable for the purpose of the MoBiLab QMS, when implemented in the aspects appropriate to the characteristics of the laboratory. The development of the QMS was performed making use of quality techniques and methods to ensure a lean and rigorous process.

Among the positive outcomes that can be ascribed to the adoption of the QMS, we can stress an evident increase in the efficiency of the laboratory, evaluated by the decrease of the risks and of the errors occurring during analysis workflow. Moreover, staff was more motivated thanks to a better organization of the team and an acknowledgment of their competences. The effectiveness was also improved, increasing the number of collaborators, the customer confidence and the availability of a databases organized for future investigations.

Indeed, the experience of QMS in MoBiLab demonstrates that the performances of the analysis and number of the primary “products” of academic research—publications—increases after QMS introduction, together with the appeal of the laboratory as witnessed by more active international collaborations. Number of publications in high-impact factor journals, number of citations, and number of opportunities of excellent scientific collaborations indicate how the laboratory aims to be a national and world scientific leading infrastructure and an enabling facility supporting science.

Experimental and service data (throughput analysis) prove that the laboratory, thanks to the ReOmicS introduction, is increasing its ability to provide high quality scientific data and associated services.

The results of the surveys give evidence of a positive user satisfaction regarding support and collaboration, not only in reference to the facility, but also to the staff employed in the MoBiLab. Good opinions were also expressed comparing MoBiLab with other similar laboratories. We have planned to run such surveys periodically with different audiences to assess satisfaction, achievements, collaborations and expectations.

As future perspectives, scientific, economic and technological impact could be assessed reporting, for instance, the training of skilled researchers, the development of new methodologies and software for NGS, the growth of network and social interactions or the creation of a new firm (e.g., a spin-off).

As proposed by the OECD ( 2019 ) for the assessment of Research Infrastructures, the metrics chosen to evaluate efficiency and effectiveness of MoBiLab before and after the introduction of the QMS can be presented as possible indicators to demonstrate the impact of an NGS research and analysis laboratory.

The quantifiable impacts captured through quantitative metrics (number of publications, citations…), as well as the non-quantifiable metrics obtained by dedicated surveys, are also helpful when approaching an economy and policy impact analysis of the genomic and bioinformatic research.

Although the ReOmicS QMS did not incorporate all requirements of the GLPs, and its application is still to be completed with respect to a small number of controls and the complete traceability of results, the positive outcomes already obtained are also due to the gradual increase in confidence by laboratory staff with the quality approach and to the early adoption of the standard protocols described by the technical SOPs.

This experience and the results obtained prove that a NGS laboratory, and therefore any other research laboratory, can benefit from the introduction of a quality framework, if properly translated from the generic standards and adapted to the specific requirements of a research environment. The metrics and the indicators showed in this study will be followed up in the implementation of the MoBiLab thanks to the grant received by the Italian Research Minister to empower facilities and equipment of this laboratory.

The standards of the ISO 900 family are often declined in sector-specific versions (e.g., the already mentioned ISO 17025 for testing and calibration laboratories, ISO/TS 16949 for automotive, ISO 13485 for medical devices), adding requirements unique to a particular application. The ReOmicS has this precise purpose: to detail the requirements for the management and control of a generic research laboratory. To this end, as a first choice, we took as a reference the GLPs, whose adequacy for a research laboratory in ‘omics—despite some specific exclusions—is attested by Kauffman et al. ( 2017 ). The directions of the GLP are about management and control of the study and experimentations (the primary process), and of the tools and materials used, as well as the rules for reporting and archiving data, samples and documents (which are only a few of the support processes envisaged by the ISO9001 standard). In a general sense, these aspects are common to all research laboratories, regardless of the field of application. Starting from the structure of a GLP-like QMS, the ReOmicS can be completed with the specific requirements of ISO 9001 on the parts not governed by the GLP, such as context and risk assessment, supplier management and improvement process. In this light, we strongly believe that the ReOmicS can be taken as a reference for any type of research laboratory.

Indeed, the ReOmicS GLP-like quality system is expected to evolve into a complete quality system according to ISO9001 to achieve the specific certification. For this reason, a risk assessment—both strategic and operational has already been performed, and several SOPs have already been arranged to comply with main requirements of ISO 9001.

In the future, quality management would be streamlined by the introduction of the IT tool (LIMS). This development is expected to foster the potential exploitation of the NGS activities of the MoBiLab. The evolution of the ReOmicS is then expected to follow the Lean Production (a.k.a. Toyota Production System), which is a wide-ranging methodology developed in manufacturing to reduce waste and improve product quality (Womack et al., 1991 ) and recently used also in the research environment (Barnhart, 2013 ). The Lean approach will be strictly connected with the LIMS system, leveraging its features to ensure the best control. The GLP-like system and the Lean approach will allow the MoBiLab to improve its efficiency, limiting wastage of time and materials, and reducing opportunities for error, at the same time enhancing the effectiveness traceability and reproducibility of results.

Data availability

All data generated or analyzed during this study are included in this published article.

Baker M (2016) 1500 scientists lift the lid on reproducibility. Nature 533:452–454. https://doi.org/10.1038/533452a

Article   ADS   CAS   Google Scholar  

Barnhart T (2013) Creating a lean R&D system. CRC Press, Boca Raton

Google Scholar  

Cargill CF (2011) Why standardization efforts fail. J Electron Publ. https://doi.org/10.3998/3336451.0014.103

Corpas M, Kovalevskaya NV, McMurray A, Nielsen FGG (2018) A FAIR guide for data providers to maximise sharing of human genomic data. PLOS Comput Biol 14(3):e1005873. https://doi.org/10.1371/journal.pcbi.1005873

Article   ADS   CAS   PubMed   PubMed Central   Google Scholar  

Defeo J, Juran JM (2010) Juran’s quality handbook: the complete guide to performance excellence 6/e McGraw-Hill Professional, US

Dirnagl U, Kurreck C, Castaños-Vélez E, Bernard R (2018) Quality management for academic laboratories: burden or boon? Professional quality management could be very beneficial for academic research but needs to overcome specific caveats. EMBO Rep. https://doi.org/10.15252/embr.201847143

Article   Google Scholar  

Endrullat C, Glökler J, Franke P, Frohme M (2016) Standardization and quality management in next-generation sequencing. Appl Transl Genomics. https://doi.org/10.1016/j.atg.2016.06.001

Kauffmann HM, Kamp H, Fuchs R, Chorley BN, Deferme L et al. (2017) Framework for the quality assurance of’omics technologies considering GLP requirements. Regul Toxicol Pharmacol. https://doi.org/10.1016/j.yrtph.2017.10.007

Lanati A (2018) Quality management in scientific research-challenging irreproducibility of scientific results. Springer, Cham, Switzerland

Book   Google Scholar  

Stephens ZD, Lee SY, Faghri F, Campbell RH, Zhai C et al. (2015) Big data: astronomical or genomical? PLoS Biol. https://doi.org/10.1371/journal.pbio.1002195

Tiwari R, Nain L, Labrou NE, Shukla P (2018) Bioprospecting of functional cellulases from metagenome for second generation biofuel production: a review. Crit Rev Microbiol. https://doi.org/10.1080/1040841X.2017.1337713

UNDP/World Bank/WHO Special Programme for Research and Training in Tropical Diseases & Scientific Working Group on Quality Practices inBasic Biomedical Research (2006) Handbook: quality practices in basic biomedical research/prepared for TDR by the Scientific Working Group on Quality Practices in Basic Biomedical Research. World Health Organization. https://apps.who.int/iris/handle/10665/43512

Wiese L, Schmitt AO, Gultas M (2018) Big data technologies for DNA sequencing. Springer, Basel

Wilkinson MD, Dumontier M, Aalbersberg IJ, Appleton G, Axton M et al. (2016) The FAIR Guiding Principles for scientific data management and stewardship. Sci Data. https://doi.org/10.1038/sdata.2016.18

Womack JP, Jones DT, Roos D (1991) The machine that changed the world. Harper Perennial, New York

OECD (1998) Organisation for economic co-operation and development principles on good laboratory practice (As Revised in 1997). OECD principles of good laboratory practice and compliance monitoring. No. 1. ENV/MC/CHEM(98)17

OECD (2019) Reference framework for assessing the scientific and socio-economic impact of research infrastructures. OECD Science, Technology and Industry Policy Papers, No. 65

Download references

Acknowledgements

This work was supported by the projects “Lifewatch” Roadmap ESFRI and “OMICS4FOOD”, cod. 1JLZKDPOR, Avviso “Innonetwork” A.D. n.124 del 16/10/2017, Puglia FESR-FSE 2014–2020 Azione 1.6. The authors thank Gabrielle Nasca Quadraccia for critical reading and English revision of the paper.

Author information

These authors contributed equally: Antonella Lanati, Marinella Marzano

Authors and Affiliations

Valore Qualità, 27100, Pavia, Italy

Antonella Lanati

Istituto di Biomembrane, Bioenergetica e Biotecnologie Molecolari (IBIOM), CNR, 70126, Bari, Italy

Marinella Marzano, Caterina Manzari, Bruno Fosso, Graziano Pesole & Francesca De Leo

Dipartimento di Bioscienze, Biotecnologie e Biofarmaceutica, Università degli Studi di Bari “Aldo Moro”, 70126, Bari, Italy

Graziano Pesole

You can also search for this author in PubMed   Google Scholar

Corresponding authors

Correspondence to Antonella Lanati or Francesca De Leo .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Lanati, A., Marzano, M., Manzari, C. et al. Management at the service of research: ReOmicS, a quality management system for omics sciences. Palgrave Commun 5 , 75 (2019). https://doi.org/10.1057/s41599-019-0283-0

Download citation

Received : 28 February 2019

Accepted : 12 June 2019

Published : 09 July 2019

DOI : https://doi.org/10.1057/s41599-019-0283-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Research and Quality Improvement: How Can They Work Together?

Affiliations.

• 1 Director, Data Science, Quality Insights, Williamsburg, VA.
• 2 Chair, ANNA Research Committee.
• 3 President, ANNA's Old Dominion Chapter.
• 4 Instructor, Case Western Reserve University, Cleveland, OH.
• 5 Associate Degree Nursing Instructor, Northeast Wisconsin Technical College, Green Bay, WI.
• PMID: 35503694

Research and quality improvement provide a mechanism to support the advancement of knowledge, and to evaluate and learn from experience. The focus of research is to contribute to developing knowledge or gather evidence for theories in a field of study, whereas the focus of quality improvement is to standardize processes and reduce variation to improve outcomes for patients and health care organizations. Both methods of inquiry broaden our knowledge through the generation of new information and the application of findings to practice. This article in the "Exploring the Evidence: Focusing on the Fundamentals" series provides nephrology nurses with basic information related to the role of research and quality improvement projects, as well as some examples of ways in which they have been used together to advance clinical knowledge and improve patient outcomes.

Keywords: kidney disease; nephrology; quality improvement; research.

Copyright© by the American Nephrology Nurses Association.

• Nephrology*
• Quality Improvement*

Grants and funding

• K23 NR019744/NR/NINR NIH HHS/United States

25 years of quality management research – outlines and trends

Purpose The purpose of this paper is to explore and describe how research on quality management (QM) has evolved historically. The study includes the complete digital archive of three academic journals in the field of QM. Thereby, a unique depiction of how the general outlines of the field as well as trends in research topics have evolved through the years is presented. Design/methodology/approach The study applies cluster and probabilistic topic modeling to unstructured data from The International Journal of Quality & Reliability Management, The TQM Journal and Total Quality Management & Business Excellence. In addition, trend analysis using support vector machine is performed. Findings The study identifies six central, perpetual themes of QM research: control, costs, reliability and failure; service quality; TQM – implementation and performance; ISO – certification, standards and systems; Innovation, practices and learning and customers – research and product design. Additionally, historical surges and shifts in research focus are recognized in the study. From these trends, a decrease in interest in TQM and control of quality, costs and processes in favor of service quality, customer satisfaction, Six Sigma, Lean and innovation can be noted during the past decade. The results validate previous findings. Originality/value Of the identified central themes, innovation, practices and learning appears not to have been documented as a fundamental part of QM research in previous studies. Thus, this theme can be regarded as a new perspective on QM research and thereby on QM.

• Related Documents

Locally focused and digitally oriented: examining eco-museums' digitization in a service quality management perspective

PurposeEco-museums safeguard the cultural authenticity and the historical identity of the place in which they operate. Conventional organizational models and management practices are generally employed to achieve this institutional aim. Conversely, innovative solutions – such as digitization – are overlooked. Adopting a service quality management perspective, the article intends to examine the role of managerialization and professionalization in triggering eco-museums' digitization.Design/methodology/approachAn empirical analysis involving 126 eco-museums operating in Italy as of 2018 was designed to investigate the implications of managerialization and professionalization on the eco-museums' propensity to embark on a digitization process. Two different forms of digitization were examined: (1) the presence of eco-museums in the digital environment; and (2) the exploitation of digital tools for service delivery. The mediating role of two “soft” total quality management (TQM) practices, i.e. people centredness and strategic focus on visitors' experience, was contemplated in the empirical analysis.FindingsThe research findings suggest that managerialization and professionalization have ambiguous effects on eco-museums' digitization. Nevertheless, they indirectly contribute to a greater digital presence of eco-museums and to a larger use of digital tools for service delivery through an increased use of soft TQM practices.Research limitations/implicationsManagerialization and professionalization are likely to foster the digital transition of eco-museums, which advances their ability to protect and promote the local cultural heritage. Soft TQM practices intended to achieve people-centredness and to enhance the visitors' experience should be exploited to stimulate the eco-museums' digitization.Originality/valueThe article examines the triggers of eco-museums' digitization, providing some food for thought to scholars and practitioners.

Use of multiple methodologies for developing a customer-oriented model of total quality management in higher education

Purpose – Educational institutes must embrace the principles of total quality management (TQM) if they seek to remain competitive, and survive and succeed in the long run. An educational institution must embrace the principles of quality management and incorporate them into all of their activities. Starting with a theoretical background, the paper outlines the results of a study conducted on both internal and external customers of the educational system, with select engineering and management institutes as foci of study. The study is an attempt toward developing an integrated customer-centric model of quality management in education, through the use of multiple methodologies so as to be able to evaluate service quality; prioritize improvement of service; and guide and develop educational services by incorporating the voice of the customer (VOC). The purpose of this paper is to establish the prioritization for improvement of service design of an educational system through incorporation of the VOC, be it internal or external customers, through the use of multiple methodologies, leading to generalization of results. Design/methodology/approach – The study uses multiple methodologies with various techniques for analysis through the application of the SERVQUAL; quality function deployment (QFD); interpretive structural modeling (ISM); and path analysis. The SERVQUAL was applied to identify the gap and determine the level of service quality. Following this, QFD, ISM and path analysis were used to identify the set of minimum design characteristics/quality components that would meet the requirements the various internal and external customers of the educational system. The QFD was used to identify the set of minimum design characteristics/quality components that meet the requirements of the various internal and external customers of the educational system. The ISM and path analysis were used to identify and prioritize the design characteristics/quality components that would meet the requirements the various internal and external customers of the educational system. Findings – The findings from the various techniques were amalgamated, and proposed as an integrated model of TQM in higher education. The study helped identify with a customer perspective, the quality components which would help design TQM for higher education institutions in India. Research limitations/implications – The paper could be useful to government bodies, funding agencies, policy makers and administrators in developing a system that could lead to satisfaction of both internal and external customers of the educational system. Originality/value – The study includes within its scope the varied customers of the educational system, namely, internal and external customers of the educational system; the internal customers being the faculty and the administrative staff, and the external customers being the students and the industry (as the employer). This is yet to be seen in other research studies. Also, the integration of the multiple tools and their application to the field of higher education in India, has not yet been made available in the literature.

From quality control to TQM, service quality and service sciences: a 30-year review of TQM literature

Purpose This paper aims to review total quality management (TQM) literature in the past three decades to identify the quality related key terms, to analyse their linkage among the identified key terms and their developmental trends. Design/methodology/approach Bibliometric and statistical methods are used to analyse article titles published in the Total Quality and Business Excellence journal during 1990–2019. The current research is based on a search from the ProQuest academic database and the journal’s website, resulting in 2,452 articles collected. The VOSviewer and Microsoft Excel were then used for the analyses. Findings A total of 52 key terms were extracted from the journal’s 2,452 article titles, the top three key terms in terms of occurrences were “quality,” “total quality management” and “service quality.” Five themes were then proposed from clustering the 52 key terms: “frameworks/models,” “essentials/enablers,” “methods/techniques,” “culture/characteristics” and “effects/results.” Trend analyses were also conducted regarding the five themes in an attempt to highlight the patterns of research publications from 1990 to 2019. It is found that the research publications for “essentials/enablers,” “methods and techniques” and “effects/results” have steadily increased during the analysis period, while “frameworks/models” and “culture/characteristic” have slightly decreased. These insights provide implication for the historical evolution of quality from “quality control,” “total quality management” and “service quality,” combining with the development of “service sciences.” Originality/value This paper highlights the concept of quality since its meaning has changed and evolved over time from quality control, TQM to service quality. And it is emerging in the present and future development of service sciences because of both of TQM and service sciences having the same nature of multidisciplinary background and characteristics. Also the authors can conclude that quality and service sciences are in fact two sides of the same coin because both of them having the same duality of “tangible-intangible” and “physical-virtual” faces which are the important topics that TQM should focus on.

Stakeholder-oriented service excellence: the case of Ajman Free Zone Authority of United Arab Emirates

Subject area Business management, quality management, service quality and customer service in public sectors. Study level/applicability This case is most relevant to upper-level undergraduate business students taking quality management, strategy and service management courses. It is also relevant to practitioners working in similar positions. The case is based on primary and secondary data, and all materials mentioned were taken from real work environments. Case overview In contemporary competitive markets, all entities face a growing challenge to retain customers by satisfying them. In this case study of Ajman Free Zone Authority (AFZA), which is a public entity which was started in 1988 with the aim of boosting industrial development in Ajman, it is seen that the entity (AFZA) recognized a competitive advantage by improving service quality. However, AFZA focused on implementing various service quality improvement initiatives for not only customers, but also for other stakeholders as well (e.g. employees, strategic partners, suppliers and society). AFZA sought to understand stakeholders' needs, which led to service excellence. The purpose of this case is to highlight how AFZA differentiated itself by using initiatives that focused on disparate stakeholders to achieve customer satisfaction. The concepts of service quality (SERVQUAL), total quality management (TQM) and continuous improvement offer insights into how to improve organizational performance. It highlights how AFZA used Stakeholder Theory to identify and then collaborate with stakeholders to attain best service quality outcomes. The case study is developed using both secondary and primary sources. Expected learning outcomes After reading and analysing this case study, the student will be able to identify stakeholders in a service-based entity; apply Deming's Cycle or SERVQUAL to suggest improvement programmes; describe relationships among all stakeholders; and describe initiatives that contribute to service excellence. Supplementary materials Teaching Notes are available for educators only. Please contact your library to gain login details or email [email protected] to request teaching notes.

The influence of culture on quality management practices and their effects on perceived service quality by secondary school students

Purpose The purpose of this paper is to investigate how quality culture influences the relationship between total quality management (TQM) and secondary school students’ perceived service quality (PSQ). Design/methodology/approach The authors underpin research to analyse the effect of quality culture and TQM practices on PSQ. The sample included questionnaires completed by 268 teachers and 559 students from 56 secondary schools in the Murcia Region (Spain). The proposed model comprises an exogenous construct (quality culture) and three endogenous ones (two constructs represent the TQM model’s infrastructure practices and core practices, whereas one represents PSQ), and specifies the relations among them. The authors used the variance-based structural equation modeling technique and the partial least squares estimation method to test the hypotheses. Findings Its empirical analysis reveals that the quality culture influences the effectiveness of QM practices by suggesting a significant strong effect on infrastructure and core quality practices. In turn, the analysis reveals that these two QM aspects differently have an impact on PSQ. Finally, the mediation analysis results reveal the indirect significant impact of the quality culture on PSQ through the mediator effect of QM practices. Originality/value The main contribution of this work is to theoretically explain and empirically prove some mechanisms by which education centers can develop and implement a total quality initiative. The findings provide ideas for management teams about how to personalize TQM practices to achieve optimum performance outcomes.

Role Of Total Quality Management In Service Innovations: An Empirical Study Of Pakistans Financial Services Firms

This empirical study examines the relationship between total quality management (TQM) and service innovation as well as the relationship between service innovation and service quality in the Pakistans financial services industry. Most of the past research linked innovation performance with subjective performance of the firm. But, this study empirically evaluates the effect of innovation performance on firms judgmental performance (Service quality).There were 190 respondents from financial service firms in Pakistan. Multiple regression analysis was used to observe the connection between TQM, service innovation practices and service quality. A model is proposed based on theoretical considerations, connecting TQM constructs to the service innovation and to the service quality construct. The theoretical construct explains the connection among TQM practices, service innovation practices and service quality. The tri-dimensional relationship bridges the gap between TQM, service innovation and service quality and shows the importance of TQM in explaining the relationship between service innovation and service quality. This research also integrates the connection among TQM implementation, Service innovation practices and service quality. Data analysis shows that TQM implementation has a positive and significant impact on service innovation as well service quality. It has contributed in confirming that TQM practices deployed in a financial service firm in Pakistan has positive impact on service innovation and service quality.

Critical success factors for total quality management implementation and implications for sustainable academic libraries

PurposeThe purpose of this study is to investigate the critical success factors for total quality management implementation and implications for sustainable academic libraries in Ghana. This study is part of a PhD project that focussed on selected technical university libraries in Ghana.Design/methodology/approachThis study adopted a quantitative approach to collect the data. Samples of 124 participants were involved in this study. PLS-SEM (Smart PLS3) software was used to analyse the data. Convergent, discriminant validity assessment was computed. Eight variables of critical success factors were tested in relation to total quality management implementation at selected academic libraries in Ghana.FindingsThis study established that out of the eight variables tested, six of them (i.e. top management commitment, employee innovation employee training, organisational culture, teamwork and effective communication, quality performance) indicated a significant positive relationship with total quality management implementation apart from strategic planning and human resource management.Research limitations/implicationsThis study was limited to eight variables as the critical success factors mentioned in the previous paragraph. The use of one methodology might be a limitation as the use of multimethod might have given a more comprehensive picture than the case. This study was limited to only technical university libraries in Ghana hence caution must be exercised when applying the results to contextually different academic environments. The results are applicable to academic universities library in Ghana and beyond if they are adjusted to suit the context.Practical implicationsThis study is timely as it may lead to effective total quality management implementation and the sustainability of technical university libraries in Ghana and Africa in general.Originality/valueThe proposed model can be used to enhance the smooth implementation of total quality management in academic libraries in general and Ghana in particular. The framework is termed as Eddie and Pat's Achievement of Quality Performance (EPAfQP) model.

From IMS and six sigma toward TQM: an empirical study from Serbia

Purpose – The purpose of this paper is to present the current status of a quality management practice in Serbia, in terms of the development and application of integrated managements systems (IMS), and research and implementation of Six Sigma and the related techniques. Design/methodology/approach – Two main aspects of total quality management (TQM) have been considered: institutional (organisational or strategic) aspect that corresponds to standardised management systems and their integration, and technical (quality engineering) aspect whose main contributor is Six Sigma. The findings of a comprehensive study on IMS implementation in Serbia have been presented, based on the results of a questionnaire that was sent to 54 organisations during 2013. Also, the significant technical improvements and tangible benefits of a recently conducted Six Sigma project were shown, including the application of the advanced quality engineering techniques within DMAIC method. Findings – Good QM practice in Serbia is improving, both in terms of the organisational (IMS) and technical (Six Sigma) aspect, that server as a good basis for the adoption of TQM in manufacturing companies from various sectors. It could be anticipated that these results will facilitate the adoption of an overall TQM culture in Serbia and leverage its future sustainability. Originality/value – This paper offers key insights into IMS and Six Sigma implementation in Serbia. This could encourage manufacturing organisations in developing countries to adopt IMS and Six Sigma, in order to boost the overall TQM culture and gain a competitive advantage.

Total quality management and sustainability in the public service sector: the mediating effect of service innovation

PurposeThis study aims to critically investigate the structural relationships between total quality management (TQM), service innovation and sustainability performance in the public service sector of the United Arab Emirates (UAE).Design/methodology/approachThe study employed an online survey to collect data from 400 employees working in eight selected UAE public service sector organisations located in Abu Dhabi. The collected data were analysed using structural equation modelling (SEM) to empirically examine whether TQM practices improve service innovation and, subsequently, sustainability performance in the UAE's public service sector.FindingsThe results show that TQM has a significant impact on service innovation and sustainability performance in the UAE's public service sector. Additionally, service innovation partially mediates the relationship between TQM and sustainability performance.Practical implicationsThe public service sector's TQM practices and service innovation in the UAE have a much greater impact on social and environmental sustainability than on economic sustainability performance. Adopting five dimensions of TQM (following the Abu Dhabi Award for Excellence in Government Performance [ADAEP] model) across the UAE's public organisations will enable government departments to deliver innovative services to its beneficiaries.Originality/valueThis study provides a substantial contribution by addressing the gaps in the literature. Very few studies have empirically investigated the possible association between TQM, service innovation and sustainability performance in public sector organisations, particularly in developing countries such as the UAE, where the increasing efforts in TQM practices are still in their emerging stages, mainly targeting innovative service offerings and sustainable performance.

Benchmarking of TQM practices in the Jordanian pharmaceutical industry (a comparative study)

Purpose The purpose of this paper is to establish practical guidelines for benchmarking eight total quality management (TQM) practices vital to pharmaceutical companies’ performance. The paper also proposes the use of an analytic total quality index (TQI) as a benchmarking tool and illustrates the importance and effectiveness of this benchmarking methodology by applying it in two comparative studies of three Jordanian pharmaceutical companies. Design/methodology/approach In order to achieve the above-mentioned purpose, the data were gathered through a questionnaire that was used to evaluate the gap between the ideal and current status of the quality management system and distributed to the quality units from three companies: pharmaceutical manufacturing company, a pharmaceutical manufacturing company working in the same field and a pharmaceutical service providing research services to a pharmaceutical manufacturing companies. And the mean differences between the current and ideal states for the eight critical TQM practices were compared for these two comparative studies using the t-test. Findings Each of the two comparisons reveals statistically significant differences regarding the perceptions of actual and ideal scores for manufacturing and service companies on five out of eight critical factors and, on two out of eight critical factors for manufacturing and manufacturing companies. Practical implications The pharmaceutical companies, regardless of whether they are manufacturing or service company, can adopt benchmarking techniques which were applied in this case study to improve their performance and their product/service quality. Originality/value The consequences of this research can support organization managers and policy makers in effectively benchmarking the identified TQM practices in their organizations using the proposed TQI benchmarking tool.

Export Citation Format

Share document.

• Open access
• Published: 21 May 2015

Quality management and innovation: new insights on a structural contingency framework

• Dara Schniederjans 1 &
• Marc Schniederjans 2  

International Journal of Quality Innovation volume  1 , Article number:  2 ( 2015 ) Cite this article

40k Accesses

14 Citations

2 Altmetric

Metrics details

With increasing market competition, organizations are striving for greater innovation in products and services. Quality management has the potential to invigorate an organization’s product, process and administrative innovation when strategically aligned with internal contingencies. This paper seeks to address the relationship between social and technical quality management with innovation. Moreover, this paper empirically assesses contingency factors including organization size, task and managerial ethics which play roles in moderating the relationship between quality management and innovation. Based on an empirical study we find social quality management practices, not technical quality management practices, are positively associated with innovation. We also find a reciprocal positive relationship between social quality management and technical quality management. In addition our research reveals the positive relationship between quality management and innovation is moderated by the effects of organizational size, task and managerial ethics.

In an increasingly competitive environment, factors such as innovation and quality management can lead to competitive advantage. A recent survey of the Boston Consulting Group found that innovation was among the top three strategic priorities for 71% of companies [ 1 ]. This is in part due to innovation’s being able to provide unique products and processes which create greater value for consumers as well as financial benefits for the organization [ 2 ]. Other research suggests quality management is a known driver of innovation in organizations. Quality management practices have also been associated with operational and financial performance allowing firms to achieve a sustainable competitive advantage ([ 2 , 3 ];).

The importance of innovation and quality management has motivated researchers to identify various driving forces of innovation and to seek new ways of creating it through quality management practices. The current research defining the relationship between quality management and invigorating innovation appears to possess some shortcomings.

First and foremost is the presentation of mixed results. While some studies found a positive association between quality management practices and innovation [ 4 , 5 ] others showed no such connection [ 6 , 7 ]. We contend a reasonable explanation for these mixed results is due to a lack of understanding of potential contingency factors. To survive in a dynamic and competitive environment, organizations need practices that are aligned with their own individual organizations [ 3 , 8 ]. As such this study seeks to address several contingency factors using a theoretical basis of structural contingency theory. These factors include organizational size and task as well as managerial ethical evaluation.

Secondly, while some studies express the importance of managerial leadership as a basis for enhancing the value of quality management in innovation (e.g., [ 2 ]) there is limited development on this construct, especially when it comes to motivational factors like ethical evaluation. Thus we seek to also highlight the importance of managerial ethics (via managerial deontological and teleological evaluation) on improving innovation through quality management.

Third, prior research suggests the need for more studies to analyze the different dimensions of quality management on each other as well as on other variables like innovation [ 9 ]. We therefore distinguish two dimensions (social and technical) of quality management and seek to address their relationship with innovation. We also seek to assess the impact of quality management on various aspects of innovation including product, process and administrative innovation.

In summary this paper seeks to address the following questions: do quality management practices impact innovation, and more specifically, do social quality management practices (i.e., cross-functional cooperation, cross-training, long-term supply chain relationships) and technical quality management practices (i.e., just in time (JIT) manufacturing and design for manufacturing) impact innovation (i.e., radical and incremental product and process innovation and administrative innovation)? Do social quality management practices impact technical quality management practices and vice versa? Do certain contingency factors (i.e., organizational size, task and managerial ethical evaluation) play a role in the relationship between quality management and innovation?

To address these questions our paper is structured as follows: first, we present a brief literature review of quality management, innovation and our contingency factors. Next, we address our conceptual model and explain how structural contingency theory explains the relationships within the model. Then we discuss the results of our survey analysis. Finally, we address research and practical implications in our discussion and conclusion.

Literature review

Innovation is broadly defined in business research literature. In general it is defined as new applications of knowledge, ideas, or methods which generate new capabilities and leverage competitive sustainability [ 2 , 10 , 11 ]. Current research takes into account that there are several types of innovation which require an organization to demonstrate unique and refined responses. As such we follow the work of Kim et al. [ 2 ] which characterizes five unique types of innovation: incremental product, incremental process, radical product, radical process and administrative. Previous research has validated these types of innovation empirically (e.g., [ 12 - 14 ]).

In this paper we delineate three classes of innovation: product, process and administrative. Product innovation refers to the technological innovation of the firm in developing novel products for consumers. Process innovation refers to the technological innovation of firms in their production processes. Both product and process innovation can be either radical or incremental. Radical innovation refers to the adoption of new technology in order to create demand that is not recognizable by markets [ 2 ]. Incremental innovation is minor changes to current technologies when it comes to design, function, price, quantity or other features to meet the needs of current customers [ 2 ]. Administrative innovation in contrast to technological innovation is often enhanced by internal needs for structure and coordination. Instead of a focus on the customer it focuses on internalized structures and systems.

Innovation has a long history of research addressing how to define it (e.g., [ 15 ]) or how to enhance it (e.g., [ 16 ]). In this study we focus on the latter of the two questions addressing the impact of quality management on innovation.

Quality management practice

Quality management like innovation is also a broadly defined topic. Most research, however, agree that the main goal of quality management is to improve and meet stakeholder needs by removing deficiencies including error and rework [ 17 , 18 ]. While a vast majority of studies view quality management practices as a single variable (e.g., [ 18 ]) other more recent studies delineate the various practices into multiple dimensions (e.g., [ 3 ]). In this study we characterize quality management into two different dimensions: social and technical practices.

Social quality management practices refer to quality management practices that are social/behavioral in nature. These practices have a focus toward involvement and commitment of management, employees, and supplier training, learning and cooperation or teamwork [ 3 ]. Previous studies have captured the social essence of quality management by a focus on internal social structures (e.g., cross functional cooperation) as well as external social structures (e.g., long-term supplier relationships). In this study we will focus on both by borrowing the three dimensions of Ketokivi and Schroeder [ 19 ]: quality training, cross-functional cooperation and long-term supply chain relationships. Quality training is the degree to which employees receive training to perform quality management tasks. Cross-functional cooperation refers to the degree to which different departments and individuals coordinate their quality activities and efforts. Long-term supply chain relationships refer to the degree to which the organization encourages development of recurring exchanges, supplier involvement and reliable information sharing with suppliers.

Technical quality management refers to the mechanical methods used by employees of an organization. It is generally defined as practices with a focus on controlling processes and products through tools for the purpose of conforming to and satisfying established requirements [ 3 ]. Previous literature defines technical quality management in a variety of ways including process management [ 20 - 22 ], preventative maintenance [ 23 ] and housekeeping [ 24 ]. In our study we define technical quality management through the lens of JIT and design for manufacturability which encompass each of these dimensions [ 19 ]. Just-in-time (JIT) refers to the degree to which the organization seeks to eliminate waste, minimize inventories through measures such as set up time reduction, frequent resupply and delivery and plant layout [ 19 ]. Design for manufacturability is the degree to which a plant’s products are designed to reduce any complication in manufacturing through practices including simplified design and reduction of parts [ 19 ].

As previously mentioned prior research both confirmed and denied the relationship between quality management and innovation. Given this conflicting evidence we believe it is important to assess contingency factors which may govern this relationship. Structural contingency theory provides an adequate theoretical basis for guidance in distinguishing specific contingency factors that may impact this relationship.

Structural contingency factors

Structural contingency theory posits that an organization must strive to align the contingencies of the firm with factors in the external and internal environment [ 25 ]. Ultimately, the success of an organization depends on whether an organization’s processes and practices fit with both environmental and internal practices [ 26 - 30 ]. Applied to the context of this paper we will examine the impact of quality management practices on innovation performance whose relationship will be determined by the three characteristics of organizational size, organizational task and managerial ethical evaluation.

Organizational size has been operationalized most often as the approximate number of employees of an organization [ 31 , 32 ]. This dimension is one of the two main contingencies that structural contingency theory considers. The other contingency is organizational task. Organizational task can be operationalized through task uncertainty and task interdependence [ 33 ]. Task uncertainty refers to the lack of information about how to perform a specific task [ 34 ]. Task interdependence is the degree to which individuals perceive that they interact with and depend on others in order to carry out their work [ 35 ]. Both organizational size and task have been well documented factors which play roles in establishing adequate fit for performance [ 25 ]. As such we will assess these factors in our model.

An often under-represented contingency, specifically for assessing quality management, is managerial ethical evaluation. While managerial leadership continues to be a well-documented construct and is also referred to as a necessity for establishing a myriad of performance dimensions via quality management, a lack of research provides for confusion toward motivational factors that might impact such a relationship. As such we will assess managerial ethical evaluation. Managerial ethical evaluation can be operationalized via teleological and deontological evaluation. Ethical judgments are determined by both teleological and deontological evaluations. Teleological evaluation is a function of the perceived consequences of each alternative for stakeholders, the probability that each consequence will occur for those stakeholders, and the desirability and importance of each consequence for the stakeholder [ 36 ]. Deontological evaluation is the process whereby an individual compares different alternatives with a set of predetermined deontological norms representing an individual’s personal values or perceived moral obligations [ 36 ]. Previous research supports the strong association between an individual’s ethical evaluations and actions.

Hypotheses development

According to structural contingency theory the goal of an organization and its managers should be fit which indicates consistency between a firm’s processes and practices with contingency factors. Management should respond to this by developing, selecting and implementing quality management strategies to maintain fit with contingencies such as organizational size, task and their own ethical evaluations in order to enhance performance initiatives especially in the innovation domain. Quality managers should also understand various relationships between quality management practices in order to further knowledge of the complex dynamics which lead to innovation. As such we present our conceptual model in Figure  1 .

Research model.

Quality management and innovation

The prior research involving quality management and innovation has shown the positive association between certain quality management practices and innovation [ 4 , 5 , 9 , 37 - 39 ]. Other research has suggested that not all quality management practices relate to performance or innovation [ 22 , 40 ].

Understandably, researchers need to specify the type of quality management practice and how it impacts innovation. Social quality management has the potential to improve product innovation in a myriad of ways. Quality training enhances the skills of an employee to efficiently and effectively improve teamwork, thus reducing errors and enhancing job satisfaction which can impact product innovation [ 2 ]. Cross-functional cooperation enables open communication supporting creative idea suggestions which are essential to product innovation [ 3 ]. Further, promoting greater relationships within a supply chain network can result in greater information sharing about innovative products which enables a buying company to decrease product development time and put more effort toward developing product innovation [ 2 ].

Social quality management practices also have a unique potential to invigorate process quality. Quality training enhances employees’ abilities to work efficiently and effectively further allowing them to recognize how to implement quality techniques and principles in quality management processes [ 2 ]. Cross-functional cooperation motivates employees to be involved in innovative design processes, developing teamwork and enhancing productivity essential to process design [ 41 ]. In addition long-term supply relationships are fundamental for obtaining high quality materials and leveraging unique knowledge and expertise to facilitate process innovation [ 42 ].

While social quality management has been linked to process and product innovation in previous literature, little work has been done regarding its association with administrative innovation. Social quality management has the potential to influence not only product and organizational processes, but also internal structures as well. For example, quality training helps establish teamwork, encourages creative ideas from all levels of employees, and promotes an information sharing climate enhancing internal innovation [ 43 ]. Cross functional cooperation on the other hand promotes employee involvement and teamwork allowing for a diverse group of individuals to collaborate and come up with innovative internal process ideas. Finally, the development of supplier relationship management enables organizations to exchange innovative ideas for new products, improves processes as well as enhances internal operations [ 2 ]. Suppliers can be involved with both product and process design which allows a buying organization to not only reduce time and costs, but also focus on strategic technology development [ 2 , 44 , 45 ]. Based on the previous literature we hypothesize:

H1. Social quality management practice is positively associated with innovation.

Along with social quality management technical quality management has also been assessed in terms of its impact on innovation. One important facet of JIT allows each employee key involvement in quality efforts dealing with quality data, designing products and managing processes [ 2 ]. Previous research suggests immediate and useful feedback from employees is instrumental for speeding a new product to the market which is applicable to product innovation [ 43 , 46 ]. Design for manufacturability can also promote innovation through simplification. While some expect simplification to be an impetus for de-innovation, finding new ways to reduce design problems in a product can be a form of product innovation in and of itself.

Both JIT and design for manufacturability require employees to focus on the continuous improvement of these processes. The focus on continuous improvement requires a level of process innovation in order to constantly strive for greater quality. Martinez-Costa and Martinez-Lorente [ 9 ] found that the use of common quality management technical tools leads to both product and process innovation. Design for manufacturability suggests an efficient design is characterized by fewer standardized components enhancing process management, smaller process variance and less process complexity thereby enhancing process, product and even service innovation [ 44 , 47 ].

Administrative innovation can also be linked to technical quality management. Kim et al. [ 2 ] argues that implementing quality management (like JIT and Design for manufacturability) can not only help produce innovative products and processes, but also helps develop innovation plans internally. Effective management of processes can encourage firms to develop routines formed by a set of best practices thereby establishing a learning base for further innovative activities externally and internally [ 38 ]. Based on the previous research we hypothesize the following:

H2. Technical quality management practice is positively associated with innovation.

Social and technical quality management practices

The research pertaining to the relationships between different quality management practices is still in its infancy. However, given the conflicting findings on the association between quality management and innovation, it is imperative to not only define quality management adequately, but also assess the potential relationships between different practices which may impact this relationship. Previous research suggests that “soft” quality management practices including small group problem solving, employee suggestion and task-related training have a positive impact on “hard” quality management practices including process management and quality information [ 3 ].

The quality training aspect of social quality management provides a foundation for highly motivated employees with sufficient and effective problem solving abilities. This is needed for the adoption and utilization of JIT and design for manufacturability [ 3 ]. Cross functional cooperation also requires collection and dissemination between organizational functions in order to effectively enhance quality performance. It can nurture a corporate culture of autonomy, cooperation and teamwork which provides support for technical quality management. Cross functional cooperation informs operators and engineers about defective parts immediately so that corrective actions can be taken to remedy problems and reduce defects [ 22 , 44 ]. It can also identify non-value-added processes that helps employees when modifying and improving quality processes [ 44 ]. Further, supply chain network decisions should be aligned with decisions regarding simplified product design required for manufacturability [ 48 ]. Based on this previous research we hypothesize the following:

H3. Social quality management practices are positively associated with technical quality management practices.

While we agree with studies like Zeng et al. [ 3 ] who found social quality management practices are used as a basis for creating a climate suitable for technical quality management, there is also potential for a reciprocal relationship.

The continuous use of technical quality management practices can be used as a driver for social quality management especially if there is a focus on continuous improvement which JIT requires [ 49 ]. One of several facets of work improvement programs in JIT management is developing an organization with a focus on constant improvement requiring consistent quality management training as well as cross functional cooperation [ 50 ]. Moreover, JIT requires frequent resupply and delivery of materials, thus necessitating continuous improvement of supplier relationship management [ 19 ].

Design for manufacturability requires organizations to find new ways to simplify the design of their products and reduce part counts [ 19 ]. In order to do so employees need to be cognizant of design simplification strategies and tools, thus requiring a greater need for education and training. Additionally, new designs require greater customer relationship management skills making it mandatory for inter-functional cooperation between operations/engineering and marketing. Organizations may also seek out buyer development from suppliers in order to enhance product design. Thus, organizations seek out new ways to enhance their relationships with suppliers. Based on this previous research we hypothesize the following:

H4. Technical quality management practices are positively related to social quality management practices.

Structural contingency

There appears to be little research on specific contingency factors which may play roles in the often misrepresented relationship between quality management and innovation. Structural contingency theory provides a reasonable theoretical foundation to assess this relationship.

Structural contingency relies heavily on the premise that organizations should align their practices with contingency factors in order to promote greater performance. Two main contingency factors defined in structural contingency theory are organizational size and task [ 51 ]. For purposes of this paper we also include managerial ethical evaluation.

Organizational size

Organizational size, or the approximate number of employees in an organization, is adjacent to the number of personnel resources used to facilitate quality management in enhancing innovation. That is, the more individuals an organization has, the more likely it is to have enough human resources to maintain adequate levels of quality training, participate in cross-functional collaboration and cultivate greater relationships between buyers and suppliers. As evidenced by previous studies these social quality management practices can be used to improve products, processes, and administrative innovation [ 2 , 3 , 41 , 42 ].

Moreover, having a larger workforce also enhances the performance of technical quality management practices on perpetuating greater innovation. For example a larger workforce can offer greater insight in set up time reduction methods as well as maximizing the efficiency of the plant layout, thus potentially amplifying process and administrative innovation [ 2 , 38 ]. Moreover, full departments can be dedicated to maximizing the supply and delivery of materials leading to enhanced product innovation [ 2 ]. Based on this previous research we hypothesize the following:

H5. Organizational size moderates the positive association between social quality management practices and innovation.

H6. Organizational size moderates the positive association between technical quality management practices and innovation.

Organizational task

As mentioned previously organizational task is separated into two dimensions: task uncertainty and task interdependence. These two dimensions oppose each other in terms of the potential to facilitate innovation via quality management.

While task uncertainty is the lack of information about how to perform a specific task [ 34 ], in the context of this paper task uncertainty is defined as an employee’s uncertainty in performing both social and technical quality management practices. This uncertainty can lead to a variety of problems in implementing social quality management thereby limiting its potential to increase all forms of innovation. If an individual has a lack of knowledge brought about by insufficient training, or if quality management trainers have inadequate knowledge of quality management tools, it is likely to hinder the relationship between quality training and innovation. Moreover, a lack of knowledge regarding tools used for cross-functional or supply chain information sharing can also hinder intra- and inter-organizational relationships which are valuable for enhancing product, process and administrative innovation [ 2 , 3 , 41 , 42 , 44 , 45 ].

Task uncertainty can also be a strong hindrance to the relationship between technical quality management and innovation. Both JIT and design for manufacturability require employees at all levels to have adequate and extensive knowledge of tools used for quality management purposes [ 52 ]. As such, task uncertainty can limit the effectiveness of both JIT and design for manufacturability on a variety of performance dimensions including product, process and administrative innovation.

While task interdependence is the degree to which employees perceive their interaction with and dependence on others to carry out work [ 35 ], in the context of this study it refers to the perception of employees regarding interaction and dependence on others (via internal and external stakeholders) to carry out quality management practices. Unlike task uncertainty, task interdependence can be used to facilitate performance in organizations. Social quality management practices including quality training, cross-functional cooperation and developing long-term relationships with suppliers require high levels of task interdependence between quality management employees and internal and external stakeholders [ 53 ]. A commonly held critical success factor of both JIT and design for manufacturability is teamwork [ 54 ]. Task interdependence helps build an environment conducive for teamwork [ 55 ] leading to effective quality management practices employment and consequently enhancing product, process and administrative innovation.

Based on previous research task uncertainty and task interdependence will have conflicting impacts on the relationship between quality management and innovation. While task uncertainty is likely to have a negative effect, task interdependence will likely have a positive impact on the relationship. In order to account for this task uncertainty will be assessed in terms of task certainty. In doing so we hypothesize the following:

H7. Organizational task moderates the positive association between social quality management practices and innovation.

H8. Organizational task moderates the positive association between technical quality management practices and innovation.

Managerial ethics

While structural contingency theory provides an adequate theoretical basis for explaining the impact of organizational size and task on the relationship between quality management and innovation, it is not without limitations. Critics hold that structural contingency theory is deterministic [ 56 ]. That is, an organization reacts to changes in its contingencies, which alter its environment and in turn changes its contingencies [ 51 ]. The theory argues that an organization’s structure is determined solely by its situation [ 51 ]. However, previous research argues that managerial free choice opposes contingency theory [ 57 ]. It is thus important to take into account other potential contingencies related to managerial impact. Specifically, managers with adequate motivation have the potential to lead an organization in practices that increase organizational performance, despite external or structural contingencies [ 58 ]. Ethical evaluation has been shown to impact behavior and organizational performance [ 59 ]. We will add this contingency into the proposed model to help explain the relationship between quality management practices and innovation.

As previously stated managerial ethical evaluation can be operationalized via teleological and deontological evaluation. Teleological evaluation refers to an individual’s perceived desirability and importance of a particular action’s consequences [ 36 ]. In the context of this study it refers to a manager’s perceived desirability and importance of consequences associated with conducting quality management. Very little research has been conducted on ethical evaluation and its ability to promote either quality management or be used as a contingency to promote the performance of quality management. However, an individual’s positive teleological evaluation has been known to contribute to managerial motivation in performing tasks [ 60 ] in which motivation to perform impacts performance [ 58 ]. Thus, when people perceive positive consequences deriving from promoting quality training, cross functional cooperation, and long-term supplier relationships, they are likely to enhance their practice ultimately impacting performance. It will also facilitate the extensiveness of these social quality management practices leading to greater innovation as evidence from previous studies by Ahire and Ravichandran [ 41 ], Lemke et al. [ 42 ], Kim et al. [ 2 ], and Zeng et al. [ 3 ]. Similarly, perceived positive consequences from JIT and design for manufacturability may also enhance managerial motivation ultimately impacting the relationship between technical quality management practices and innovation performance.

While deontological evaluation is a person’s perception of an action based one’s own set of personal values and perceived moral obligations [ 36 ], in the context of this study it is a manager’s perception regarding whether the use of quality management is aligned with personal values or moral obligations. An organization often shapes its strategy around certain moral obligations to its stakeholders [ 61 ]. Managers often believe they have fundamental moral obligations to stakeholders via their managerial roles [ 61 ]. This moral obligation can promote job involvement or performing job functions to one’s utmost potential [ 62 ]. This includes enhancing both social and technical quality management practices. Quality management can greatly impact an organization’s performance and often may have dire consequences if not adequately conducted. This is evidenced by situations including GM’s reluctant recall of a defective ignition switch causing 13 deaths and 54 accidents [ 63 ]. Unfortunate events like this remind firms of top management’s moral obligations to not only consumers, but also to internal and supply chain stakeholders as well. Positive deontological evaluation will help enhance managerial motivation to conduct quality management practices extensively, thus impacting product, process and administrative innovation as evidenced by previous literature [ 4 , 5 , 9 , 37 - 39 ].

It is not within the confines of this article to argue that the adoption of quality management is an ethical decision. However, based on previous literature the impact of ethical evaluation on job involvement or motivation to perform is undeniable [ 59 , 62 ]. Further, motivation enhances managerial leadership which is a critical success factor in facilitating performance including innovation via quality management [ 2 ]. While ethical evaluation may not have a direct impact on quality management or innovation, it has been shown to enhance motivation which perpetuates individual action in facilitating performance. As such we hypothesize the following:

H9. Managerial ethical evaluation moderates the positive association between social quality management practices and innovation.

H10. Managerial ethical evaluation moderates the positive association between technical quality management practices and innovation.

We chose to use survey analysis to empirically test our hypotheses for two reasons. First, survey research has the ability to generalize theoretically developed models to a larger population of interest with a known degree of accuracy [ 64 ]. Since our model is both theoretically supported by previous literature and structural contingency theory, the use of survey analysis to quantitatively examine the empirically questionable relationships is appropriate. Secondly, survey analysis allows us to gather sensitive information about organizational innovation and managerial ethical evaluation through an anonymous means.

Questionnaire development

The development of our survey was carried out in two steps: (1) collecting validated items from previous research and (2) large scale analysis. To ensure content validity a literature review was conducted to define each construct. Each question was assessed by experts in the fields of Operations Management and Business Ethics to ensure the reliability and validity of the scales. A total of 43 Likert-type scale questions were created. The question items are listed in Table  1 .

Our conceptual model requires specific knowledge about an organization’s quality management practices and how they impact innovation. It also requires general knowledge on decision makers’ ethical evaluations. As such we chose to survey respondents at senior management levels. Since we also needed specific information on design for manufacturability and product innovation, we chose to use manufacturing organizations. An online survey organization was employed to collect data from these individuals. A total of 58 responses were collected. Characteristics of the respondents appear in Table  2 . All respondents indicated their knowledge of the organizations’ quality management practices and innovation strategies.

Data analysis

Partial Least Squares analysis (PLS) [ 65 ] was utilized for the analysis. PLS is useful in a context where subject sample sizes tend to be small. [ 66 ]. Also, PLS can model both reflective and formative constructs [ 66 , 67 ]. PLS allows parameters to be estimated independent of sample size [ 68 ]. It is best when used not only with small sample sizes but when assumptions of multivariate normality cannot be made, and when the concerned with the prediction of the dependent variable [ 69 ]. Wold [ 70 ] illustrated the low sample size requirement for PLS by analyzing a data set consisting of 10 observations. Chin and Newsted [ 71 ] also provide evidence using a Monte Carlo simulation where they found PLS path modeling approaches can provide information about the appropriateness of indicators at a sample size as low as 20.

Using Hulland [ 72 ] we analyzed the validity and reliability of the scales by assessing (1) construct/item reliability, (2) convergent validity and (3) discriminant validity. These results are presented in Table  3 . To examine construct reliability we identified Cronbach’s alphas. In order to be considered a reliable construct it is recommended that Cronbach’s alphas exceed 0.7 [ 72 , 73 ]. All Cronbach’s alphas exceeded 0.7 indicating no reliability problems.

To examine convergent validity we assessed composite reliability and average variance extracted (AVE). Previous research suggests acceptable levels of composite reliability be greater than 0.7 and values be greater than 0.5 for AVE [ 74 - 76 ]. All of our composite reliability and AVE values were above 0.7 and 0.5, respectively indicating no problems with convergent validity

In order to assess for discriminant validity we compared the square root of the AVE with construct correlation coefficients and other measures [ 67 - 75 ]. As seen in Table  3 the square root of the AVE is larger with each constructs’ correlation coefficient. Based on these results each of the constructs has acceptable reliability, convergent and discriminant validity.

Since the data were collected from single informants we assessed common method bias using Harman’s [ 77 ] single factor test and a modified marker variable test [ 78 - 80 ]. Assessing the data using Harman’s single factor approach we found no single factor emerged from a factor analysis of all survey items. No one factor accounted for the majority of the variance in the model with one factor explaining only 33% [ 78 ]. In addition to the Harman’s single factor approach we also performed Lindell and Whitney’s [ 81 ] marker variable test. We assessed the correlation between a theoretically unrelated construct (marker variable) and the other constructs. The results from the model indicate that the marker variable did not have any significant influence on the endogenous latent variables. Based on these results we conclude that common method bias does not seem to be a limiting factor in this model.

We also assessed for non-response bias by conducting an analysis of the variance for differences between early and late responders. The differences were non-significant indicating non-response bias was not a problem in this study [ 82 ].

PLS does not provide an index for the validation of a theoretical model [ 67 , 83 ]. In order to assess goodness of fit Tenenhaus et al. [ 83 ] proposes assessing R 2 with a suggested cut-off value of 0.36 [ 84 ]. Our model exceeds the rigid cut-off with an R 2 value of 0.46.

We examined the statistical significance of the parameter estimates using bootstrap with replacement. Our results presented in Table  4 are based on a bootstrapping sample of 500. Hypothesis 3 and Hypothesis 4 were examined with two different models. There were no statistically significant differing results between the two models.

Hypothesis 1 examined the relationship between social quality management and innovation. The results supported that social quality management practices have a positive association with innovation in organizations (β = 0.489, p < 0.01). Hypothesis 2 examined the relationship between technical quality management practice and innovation. The results did not support that technical quality management practices have a positive association with innovation in organizations (β = 0.054, p > 0.10). Hypothesis 3 examined the relationship between social quality management practices and technical quality management practices. The results supported that social quality management practices are positively associated with technical quality management practices (β = 0.489, p < 0.01). Further, the reciprocal relationship between technical quality management and social quality management was also found to be significant (β = 0.490, p < 0.01) providing support for Hypothesis 4.

Hypotheses 5–10 examined the moderating relationships between our structural contingency factors (organizational size, organizational task, and managerial ethics) and the relationships between social and technical quality management with innovation. Our results supported the moderating role organizational size has with the relationship between technical quality management and innovation (β = 0.891, p < 0.01) and the moderating role managerial ethics has between technical quality management and innovation (β = 0.219, p < 0.05). Thus, both Hypotheses 6 and 10 were supported. We found a significant negative moderating relationship organizational task has between social quality management and innovation (β = −0.518, p < 0.10). Further, we found no significant moderating effects of organizational size (β = 0.151, p > 0.10) or managerial ethics (β = −0.834, p > 0.10) with the relationship between social quality management and innovation. We also found no significant moderating relationship of organizational task between technical quality management and innovation (β = −0.247, p > 0.10). Thus, Hypotheses 5, 7, 8 and 9 were not supported.

Discussion/conclusions

Our results provide support for the reciprocal relationship between social quality management practices and technical quality management practices. That is, social quality management practices enhance the use of technical quality management practices, and in turn the use of technical quality management enhances social quality management. In addition we found that social quality management enhances innovation. We also found that organizational size and managerial ethics positively moderate the relationship between technical quality management and innovation.

These results provide theoretical support for the relationship between social quality management and innovation as well as social contingency theory. The relationship between social quality management and innovation has been discussed and empirically validated in previous literature [ 2 , 3 , 41 , 42 ]. Our results further validate this positive association. Our results, however, provide theoretical support detailing the various types of social quality management and an overall consensus that it positively impacts product, process and administrative innovation. This finding further provides empirical support for technical brokering which suggests innovations come about through combining two or more ideas or concepts [ 85 ]. Social quality management increases the likelihood of organizational innovation through rapid sharing and dissemination of ideas within either a single or between multiple organizations. Our findings also provide further empirical support for structural contingency theory. We found both organizational size and organizational task moderate the relationship between practice and performance, but in different ways. This study contributes to social contingency theory research by empirically validating that another contingency factor (managerial ethics) enhances the relationship between technical quality management and innovation.

These results augment Operations Management knowledge as well. In the past it has been assumed and empirically validated that social quality management practices enhance the use of technical quality management [ 2 ]. Our results confirm a reciprocal relationship. After examining two different models it appears technical quality management is similarly positively associated with social quality management. From this it seems organizations can use both social and technical quality management practices to enhance one another. However, not all types of quality management practices appear to increase organizational innovation.

We found no significant relationship between technical quality management and innovation. Moreover, we did not find any significant moderating relationship of organizational size or managerial ethics between social quality management and innovation. We also found a significant negative moderating relationship of organizational task between social quality management and innovation and a slightly negative, but non-significant, moderating relationship of organizational task between technical quality management and innovation.

The finding that technical quality management does not have a significant positive association with innovation has an interesting managerial implication. Based on our results technical quality management seems to be positively associated with social quality management which has a strong association with innovation. This implies that although technical quality management may not have a direct association with innovation, it might indirectly impact innovation by encouraging social quality management. For example, JIT management requires on-time delivery from suppliers. The more on-time deliveries made, the greater is the likelihood of establishing a long-term relationship with suppliers and enhancing close communications, thus potentially impacting innovation through frequent information sharing between suppliers. Understandably, while technical quality management does not have a direct association with innovation, it should not sway operations managers from technical quality management practices, but rather they should understand the dynamics between the two types of quality management in order to increase innovation performance. It is recommended that future research address the association between social and technical quality management with different types of supply chain or operations management performance.

The lack of findings regarding the moderating roles of organizational size, organizational task and managerial ethics contradicts structural contingency theory and ethics theory. It appears organizational size and task did not moderate the relationship between social and technical quality management with innovation. While structural contingency theory provides strong support for these contingency factors, they may not moderate the relationship between all types of practices and performance. For example many small organizations may spend time maintaining constant contact with their suppliers, thereby enhancing opportunities for organizational innovation. In addition high levels of managerial task certainty may actually negatively impact the relationship between both social and technical quality management and innovation. A manager who believes quality management is well understood by employees may reduce the likelihood of an organization implementing new or improved employee reward/training schemes fundamental for innovation. Future research should further explore these contradictory findings by using a larger sample size.

Another surprising finding is the lack of a moderating role that managerial ethics has between social and technical quality management practices with innovation. This finding contradicts previous research in ethics theory which suggests the importance of ethical evaluation in decision making and behavior ([ 59 - 62 ];). Perhaps the reasoning behind this finding lies in the definition of social and technical quality management practices. Quality management is fundamental not only on a managerial level, but also among lower level employees. Employees at all levels must work together with teams to establish long-term relationships with suppliers. A manager’s ethical stance about quality management may have little or no impact on the actual relationship between social and technical quality management practices of lower level employees and firm innovation. On the other hand a manager that is highly motivated to implement quality management practices (via a high moral stance toward high quality) may actually limit an employee’s ability to work with teams to find new ways to innovate because of a need to exercise control. Social quality management unlike technical quality management focuses on employee empowerment which enhances innovation. However, when management controls the actions of employees, employee empowerment is hindered, hindering teamwork and thereby innovation. Future research should further explore this surprising finding. Perhaps surveying all levels of employees would be an interesting undertaking.

Slater SF, Mohr J, Sengupta S (2014) Radical product innovation capability: literature review, synthesis and illustrative research propositions. J Prod Innov Manag 31(3):552–566

Article   Google Scholar  

Kim DY, Kumar V, Kumar U (2012) Relationship between quality management practices and innovation. J Oper Manag 30:295–315

Zeng J, Phan CA, Matsui Y (2015). The impact of hard and soft quality management on quality and innovation performance: An empirical study. International Journal of Production Economics, In press 162:216–226.

Feng J, Prajogo DI, Tan KC, Sohal AS (2006) The impact of TQM practices on performance: a comparative study between Australian and Singaporean organizations. Eur J Innov Manag 9(3):269–278

Abrunhosa A, Moura E, Sa P (2008) Are TQM principles supporting innovation in the Portuguese footwear industry? Technovation 28(4):208–221

Singh PJ, Smith A (2004) Relationship between TQM and innovation: an empirical study. J Manuf Technol Manag 15(5):394–401

Santos-Vijande M, Alvarez-Gonzalez L (2007) Innovativeness and organizational innovation in total quality oriented firms: the moderating role of market turbulence. Technovation 27(9):514–532

Gibson BC, Birkinshaw J (2004) The antecedents, consequences, and mediating role of organizational ambidexterity. Acad Manag J 47(2):209–226

Martinez-Costa M, Martinez-Lorente AR (2008) Does quality management foster or hinder innovation? an empirical study of spanish companies. Total Qual Manag Bus Excell 19(3):209–221

Daft RL (1978) A dual-core model of organizational innovation. Acad Manag J 21(2):193–210

Andersson M, Lindgren R, Henfridsson O (2008) Architectural knowledge in inter-organizational IT innovation. J Strateg Inf Syst 17(1):19–38

Chandy RK, Tellis GJ (1998) Organizing for radical product innovation: the over looked role of willingness to cannibalize. J Mark Res 35(4):474–487

Salavou H, Lioukas S (2003) Radical product innovations in SMEs: the dominance of entrepreneurial orientation. Creat Innov Manag 12(2):94–108

Di Benedetto CA, DeSarbo WS, Song M (2008) Strategic capabilities and radical innovation: an empirical study in three countries. IEEE Trans Eng Manag 55(3):420–433

Johannessen JA, Olsen B, Lumpkin GT (1998) Innovation as newness: what is new, how new, and new to whom? Eur J Innov Manag 4(1):20–31

Hurley RF, Hult GTM (1998) Innovation, market orientation, and organizational learning: an integration and empirical examination. J Mark 62(3):42–54

Juran JM, Godfrey AB (1998) Juran’s quality handbook, 5th edn. McGraw-Hill Professional, New York, NY

Google Scholar  

Kull TJ, Wacker JG (2010) Quality management effectiveness in Asia: the influence of culture. J Oper Manag 28(3):223–239

Ketokivi MA, Schroeder RG (2004) Strategic, structural contingency and institutional explanations in the adoption of innovative manufacturing practices. J Oper Manag 22:63–89

Saraph JV, Benson PG, Schroeder RG (1989) An instrument for measuring the critical factors of quality management. Decis Sci 20(4):810–829

Anderson JC, Rungtusanatham M, Schroeder RG, Devaraj S (1995) A path analytic model of a theory of quality management underlying the Deming management method: preliminary empirical findings. Decis Sci 26(5):637–658

Flynn BB, Schroeder RG, Sakakibara S (1995) The impact of quality management practices on performance and competitive advantage. Decis Sci 26(5):659–691

Arauz R, Matsuo H, Suzuki H (2009) Measuring changes in quality management: an empirical analysis of Japanese manufacturing companies. Total Qual Manag Bus Excell 20(12):1337–1374

Schonberger RJ (2007) Japanese production management: an evolution with mixed success. J Oper Manag 25(2):403–419

Donaldson L (2001) The contingency theory of organizations. Sage, Thousand Oaks, CA

Book   Google Scholar  

Lawrence PR, Lorsch JW (1967) Organization and environment. Harvard University Press, Cambridge, MA

Thompson JD (1967) Organizations in action. McGraw-Hill, New York, NY

Nadler D, Tushman M (1980) A model for diagnosing organizational behavior. Organ Dyn 9:35–51

Scott WR, Cole R (2000) Introduction. In: Cole R, Scott WR (eds) The quality movement and organizational theory. Sage, Thousand Oaks, CA

Flynn BB, Huo B, Zhao X (2010) The impact of supply chain integration on performance: a contingency and configuration approach. J Oper Manag 28:58–71

Pugh DS, Hickson DJ, Hinings CR, Turner C (1969) The contexts of organizational structure. Adm Sci Q 14:91–114

Pugh DS, Hinings CR (1976) Organizational structure: extensions and replications: the aston programme II. Saxon House, Farnborough, England

Graubner M (2006) Task, firm size, and organizational structure in management consulting: an empirical analysis from a contingency perspective. Deutscher Universitats-Verlag, Germany

Stock GN, Tatikonda MV (2008) The joint influence of technology uncertainty and interorganizational interaction on external technology integration success. J Oper Manag 26(1):65–80

Lin TC, Huang CC (2008) Understanding knowledge management system usage antecedents: an integration of social cognitive theory and task technology fit. Information Manag 45(6):410–417

Shang RA, Chen YC, Chen DC (2008) Ethical decisions about sharing music files in the P2P environment. J Bus Ethics 80:349–365

Hoang DT, Igel B, Laosirihongthong T (2006) The impact of total quality management on innovation: findings from a developing country. Int J Qual Reliability Manag 23(8–9):1092–1117

Perdomo-Ortiz J, Gonzalez-Benito J, Galende J (2006) Total quality management as a forerunner of business innovation capability. Technovation 26(10):1170–1185

Prajogo DI, Hong SW (2008) The effect of tQM on performance in R&D environments: a perspective from South Korean firms. Technovation 28(12):855–863

Ravichandran T, Rai A (2000) Quality management in systems development: an organizational system perspective. MIS Q 24(3):381–415

Ahire SL, Ravichandran T (2001) An innovation diffusion model of TQM implementation. IEEE Trans Eng Manag 48(4):445–464

Lemke F, Goffin K, Szwejczewski M (2003) Investigating the meaning of supplier-manufacturer partnerships. Int J Phys Distrib Logist Manag 33(1):12–35

Flynn BB, Schroder RG, Sakakibara S (1994) A framework for quality management research and an associated measurement instrument. J Oper Manag 11(4):339–366

Kaynak H (2003) The relationship between total quality management practices and their effects on firm performance. J Oper Manag 21(4):405–435

Zu X, Fredendall LD, Douglas TJ (2008) The evolving theory of quality management: the role of six sigma. J Oper Manag 26(5):630–650

Miller R (1995) Applying quality practices in R&D. Res Technol Manag 38(2):47–54

Ahire SL, Dreyfus P (2000) The impact of design management and process management on quality: an empirical examination. J Oper Manag 18(5):549–575

Fixson SK (2005) Product architecture assessment: a tool to link product, process, and supply chain design decisions. J Oper Manag 23(3–4):345–369

Bond TC (1999) The role of performance measurement in continuous improvement. Int J Oper Produc Manag 19(12):1318–1334

Lea R, Parker B (1989) The JIT spiral of continuous improvement. IMDS 4:10–13

Miles JA (2012) Management and organization theory. Jossey-Bass, San Francisco, CA

Siemsen E, Roth AV, Balasubramanian S (2008) How motivation, opportunity, and ability to drive knowledge sharing: the constraining-factor model. J Oper Manag 26(3):426–445

Wageman R (2012) The meaning of interdependence. In: Turner M (ed) Groups at work: theory and research. Routledge, New York, NY

Salaheldin SI (2008) Critical success factors for TQM implementation and their impact on performance of SMEs. Int J Product Perform Manag 58(3):215–237

Ramamoorthy N, Flood PC (2004) Individualism/collectivism, perceived task interdependence and teamwork attitudes among Irish blue-collar employees: a test of the main and moderating effects. Human Relations 57(3):347–366

Perrow C (1984) Normal accidents: living with high-risk technologies. Basic Books, New York, NY

Schreyogg G (1980) Contingency and choice in organization theory. Organ Stud 1:305–326

Yukl G (1989) Managerial leadership: a review of theory and research. J Manag 15(2):251–289

Hunt SD, Vitell S (1986) A general theory of marketing ethics. J Macrmarketing 6(Spring):5–16

Hunt SD, Vasquez-Parraga AZ (1993) Organizational consequences, marketing ethics, and salesforce supervision. J Mark Res 30:78–90

Berman SL, Wicks AC, Kotha S, Jones TM (1999) Does stakeholder orientation matter? the relationship between stakeholder management models and firm financial performance. Acad Manag J 42(5):488–506

Kolodinsky RW, Giacalone RA, Jurkiewicz CL (2008) Workplace values and outcomes: exploring personal, organizational, and interactive workplace spirituality. J Bus Ethics 81:465–480

Morran, C. (2015). GM ignition death toll rises again, now at 90. http://consumerist.com/2015/04/27/gm-ignition-death-toll-rises-again-now-at-90/ . Accessed April 30, 2015.

Dale A (2006) Quality issues with survey research. Int J Social Res Methodol 9(2):143–158

Ringle CM. Wende S, Will S (2005). Smart PLS 2.0 M3 Beta. Available at: http://www.smartpls.de (accessed 19 September 2014).

Peng DX, Lai F (2012) Using partial least squares in operations management research: a practical guideline and summary of past research. J Oper Manag 30(6):467–480

Chin WW (1998) The partial least squares approach to structural equation modeling. In: Marcoulides GA (ed) Modern methods for business research. Lawrence Erlbaum Associates Inc., Mahwah, NJ, pp 295–336

Graham JL, Mintu AT, Rodgers W (1994) Explorations of negotiation behaviors in ten foreign cultures using a model developed in the United States. Manag Sci 40(1):72–95

Birkinshaw J, Morrison A, Hulland J (1995) Structural and competitive determinants of a global integration strategy. Strateg Manag J 16(8):637–655

Wold HO (1989) Introduction to the second generation of multivariate analysis. In: Wold HO (ed) Theoretical empiricism: a general rationale for scientific model-building. Paragon House, New York, NY, pp VII–XL

Chin WW, Newsted PR (1999) Structural equation modeling analysis with small samples using partial least squares. In: Hoyle RH (ed) Statistical strategies for small sample research. Sage, Thousand Oaks, CA, pp 307–342

Hulland J (1999) Use of partial least squares (PLS) in strategic management research: a review of four recent studies. Strateg Manag J 20:195–204

Carmines EG, Zeller RA (1979) Reliability and validity assessment. Sage, Beverly Hills, CA

Nunnally JC (1978) Psychometric theory. McGraw Hill, New York

Fornell C, Larcker DF (1981) Evaluating structural equation models with unobservable variables and measurement error. J Mark Res 18:39–50

Bagozzi RP, Yi Y (1988) On the evaluation of structural equation models. J Acad Mark Sci 16:74–94

Harman HH (1967) Modern factor analysis. University of Chicago Press, Chicago, IL

Podsakoff PM, Organ D (1986) Reports in organizational research: problems and prospects. J Manag Stud 27:305–327

Podsakoff PM, Mackenzie SB, Lee JY, Podsakoff NP (2003) Common method biases in behavioral research: a critical review of the literature and recommended remedies. J Appl Psychol 88:879–903

Ronkko M, Ylitalo J (2011). PLS marker variable approach to diagnosing and controlling for method variance. Research Methods and Philosophy, ICIS 2011 Proceedings. Shanghai, China.

Lindell MK, Whitney DJ (2001) Accounting for common method variance in cross-sectional research designs. J Appl Psychol 86:114–121

Armstrong JS, Overton TS (1977) Estimating nonresponse bias in mail surveys. J Mark Res 14:396–402

Tenenhaus M, Vinzi VE, Chatelin YM, Lauro C (2005) PLS path modeling. Comput Stat Data Anal 48:159–205

Perols J, Zimmerman C, Kortmann S (2013) On the relationship between supplier integration and time-to-market. J Oper Manag 31:153–167

Hargadon A (2003) How breakthroughs happen: the surprising truth about how companies innovate. Harvard Business School Press, Boston, MA

Download references

Author information

Authors and affiliations.

Department of Marketing and Supply Chain Management College of Business Administration, University of Rhode Island Kingston, Kingston, RI, 79409, USA

Dara Schniederjans

Department of Management College of Business Administration, University of Nebraska-Lincoln, Lincoln, NE, 68588-0491, USA

Marc Schniederjans

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Dara Schniederjans .

Additional information

Competing interests.

The authors declare that they have no competing interests.

Authors’ contributions

DGS and MJS jointly developed the research study, performed the analysis, and drafted the manuscript. Both authors have read and approved the final manuscript.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( https://creativecommons.org/licenses/by/4.0 ), which permits use, duplication, adaptation, distribution, and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Cite this article.

Schniederjans, D., Schniederjans, M. Quality management and innovation: new insights on a structural contingency framework. Int J Qual Innov 1 , 2 (2015). https://doi.org/10.1186/s40887-015-0004-8

Download citation

Received : 14 October 2014

Accepted : 19 December 2014

Published : 21 May 2015

DOI : https://doi.org/10.1186/s40887-015-0004-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Social quality management
• Technical quality management
• Empirical research

Increasing the value of quality management systems

International Journal of Quality and Service Sciences

ISSN : 1756-669X

Article publication date: 27 July 2021

Issue publication date: 14 September 2021

Over one million organisations have a quality management system (QMS) certified to the ISO 9001 standard; however, the system requires a lot of resources and its value has been questioned. This critique also leads to a questioning of the strategic relevance of quality management. The purpose of this paper is to explore how different types of uses of QMS correlate with management perceptions of quality management in terms of respect, cost and strategic importance.

Design/methodology/approach

The paper is based on a mixed method data collection strategy, quantitative data being collected from a survey in 8 organisations ( n = 108) and qualitative data being collected from 12 interviews with quality managers in 12 different organisations.

The paper shows that a compliance-oriented QMS usage will more likely lead to a view of quality management as costly and of little respect, than a business or improvement-oriented QMS usage. Moreover, it nuances the view on compliance-oriented usage, showing that it is mainly documentation that negatively influences how management views quality management, whereas standardisation that is part of the compliance-oriented use is perceived as more value-adding.

Originality/value

This paper suggests three types of QMS use, namely, business management, improvement, and compliance-oriented use, and that a wise selection of how to use the QMS will affect the respect, strategic importance and cost that management associates with quality management.

• Quality management system
• Quality Management
• Quality audit

Gremyr, I. , Lenning, J. , Elg, M. and Martin, J. (2021), "Increasing the value of quality management systems", International Journal of Quality and Service Sciences , Vol. 13 No. 3, pp. 381-394. https://doi.org/10.1108/IJQSS-10-2020-0170

Emerald Publishing Limited

Copyright © 2021, Ida Gremyr, Jan Lenning, Mattias Elg and Jason Martin.

Published by Emerald Publishing Limited. This article is published under the Creative Commons Attribution (CC BY 4.0) licence. Anyone may reproduce, distribute, translate andcreate derivative works of this article (for both commercial and non-commercial purposes), subject to full attribution to the original publication and authors. The full terms of this licence may be seen at http://creativecommons.org/licences/by/4.0/legalcode

Introduction

Today, more than one million companies and organisations globally are certified in accordance with ISO 9001 ( ISO – International Organization for Standardization, 2018 Survey). In organisations’ quality management work, a substantial amount of time and focus is given to the quality management systems (QMS) ( Elg et al. , 2011 ). Thus, it is important that QMS adds value to the organisations ( Lenning and Gremyr, 2017 ). The interest in QMS has further grown by its potential to support sustainability efforts through integrated management systems, or by improving environmental management systems based on lessons learned from QMS ( Siva et al. , 2016 ). This potential has, however, not yet been fully exploited, and it is suggested that increased formalization and bureaucracy, induced by a certified QMS, is a reason stated for cases in which quality management hinders rather than support implementation of sustainability efforts ( Allur et al. , 2018 ; Barouch and Kleinhans, 2015 ). Even with a focus on QMS per se , that is, not as a support for an environmental management system, QMS has been subject to critique for hindering creativity, being detached from actual practice and providing limited support for quality improvement ( Poksinska et al. , 2006 ), having negative effects on process compliance ( Gray et al. , 2015 ; Karapetrovic et al. , 2010 ) and can limit focus to production and management systems instead of supporting sustainable development and green innovation ( Li et al. , 2018 ).

At the same time, evidence suggests that QMS provides a critical and established structure with potential to create value ( Rönnbäck et al. , 2009 ), contribute to product quality and operational performance ( Iyer et al. , 2013 ; Kafetzopoulos et al. , 2015b ), increase net asset value ( Ochieng et al. , 2015 ) and support continuous improvement ( Lenning and Gremyr, 2017 ). To ensure that the QMS contributes to as much value as possible, it is vital to have support from management and an appreciation of quality management work ( Beer, 2003 ; Dubey et al. , 2018 ; Joiner, 2007 ; Kaynak, 2003 ; Kafetzopoulos et al. , 2015a ; Lakhal et al. , 2006 ), and that management shows and communicates their awareness of the purpose of the QMS ( Zelnik et al. , 2012 ).

This paper aims to contribute to the existing body of research on QMS by describing different ways of using a QMS (drawing on Maguad, 2006 ); detailing and nuancing the understanding of why QMS might be perceived as non-value-adding ( Lenning and Gremyr, 2017 ; Poksinska et al. , 2006 ); and extending research evaluating the impact of QMS beyond a focus on financial performance ( Aba et al. , 2015 ; Cândido et al. , 2016 ). For practitioners, this paper aims to support a broadened understanding of how different usage of a QMS impact managements’ perception of quality management, which in turn possibly impact their willingness to invest resources in QMS.

Drawing on the various ways of operationalizing quality management proposed by ( Maguad, 2006 ), this study investigates three types of QMS usage: QMS as support for developing the quality of an offering; QMS as a tool for daily management; and QMS as a tool for standardization and documentation. The purpose of this paper is to explore how these three different types of uses of QMS correlate with management perceptions of quality management in terms of respect, cost and strategic importance. This study focuses on certified QMS, and a QMS is defined as a part of a management system regarding quality, based upon a set of interconnected or interacting elements of an organization to establish the organisation, operation, policies, objectives and processes to achieve those objectives (ISO 9000, 2015). Thus, such a system of elements can be viewed as a tool and support to reach an organisations’ objectives. In the following section, some background to QMS usage and the three ways of using QMS are provided, after which methods, findings and discussion of the findings are given. Finally, conclusions are drawn.

Theoretical background

Born with the ideas of Deming, Shewart, Juran and Ishikawa nearly four decades ago, quality management has evolved to become an established management philosophy and area of research ( Hackman and Wageman, 1995 ). This philosophy has been presented as being based upon three pillars, namely principles, practices and techniques ( Dean and Bowen, 1994 ). The principles are given as customer focus, continuous improvement and teamwork.

The ISO 9001 management system standard, being a common basis for a QMS, has become universal in its application (ISO Survey, 2018), as well as a central theme in quality management research ( Carnerud, 2018 ). ISO 9001 is claimed to have the potential for contributing to quality improvement ( Sousa and Voss, 2002 ) and improved operational performance ( Kaynak, 2003 ; Psomas and Pantouvakis, 2015 ). However, the value and the effect of a QMS is argued to depend on different factors, such as management attitudes and purposes ( Willar et al. , 2015 ), but also on quality management maturity, implementation strategy and people involvement ( Poksinska, 2010 ).

The type of motivation for implementing a QMS is also said to influence the performance of the system. Organisations focusing on real quality improvements and organisational needs achieve higher benefits from their QMS implementation in areas like quality and operational improvement, compared to those organisations that implement and seek certification of their QMS for external motives, for example, image or customer requirements ( Boiral and Amara, 2009 ; del Castillo-Peces et al. , 2018 ; Poksinska et al. , 2002 ; Sampaio et al. , 2009 ). Thus, a QMS implemented based upon external requirements, tends to focus more on compliance and control and less on organisational efficiency ( Alič and Rusjan, 2010 ).

In the following section, three different ways of working with QMS will be outlined. The three ways draw on Maguad (2006) who argued that quality in the 21st century could be categorised based on orientation in three different directions: business management, improvement and compliance. However, it is said that all three orientations must coincide for an organisation to be successful in their quality work ( Maguad, 2006 ).

Quality management systems as a tool for daily management

Maguad (2006) argued that business management-oriented quality demands an integrated deployment of strategy, and attention to critical success factors, including vision of the business, markets, and core processes. It also requires involvement from top management and every employee in continuous improvement efforts ( Maguad, 2006 ). On an overall level, Sadikoglu and Zehir (2010) studied relationships between quality practices and multiple performance measures and revealed that all practices studied – training, employee management, continuous improvement, information and analysis – were significantly and positively correlated with measures of employee performance, innovation performance, and firm performance. For QMS, it has been shown that they have effects not only on effectivity, product and service quality but also on employees and employers, for example, related to health and safety at the workplace ( Levine and Toffel, 2010 ). Furthermore, Levine and Toffel (2010) show that after being certified, firms experienced a growth in both sales and employment considerably quicker compared to firms that were not certified. Thus, the authors argued that management should consider an ISO 9001 certification as valuable.

If QMS is used as a support for managing the organisation, management will likely show respect for quality management and not view it as cost-driving but rather as being of strategic importance.

Quality management systems as a support for developing the quality of the offering

An improvement-oriented view of quality promotes an integrated approach for process improvement, involves the whole organisation, and has a wide range of applications, such as on service and support operations ( Maguad, 2006 ). In a study of service employees who interact with customers, Coo and Verma (2002) found that the employee’s perceptions of the implemented QMS had an impact on service quality of the actual offering, in terms of reliability, responsiveness, assurance, empathy and tangibles ( Parasuraman et al. , 1988 ), and in turn of the firm’s performance. Coo and Verma (2002) further believe that one success factor of these perceptions were strong leaders who were involved in promoting quality management.

If QMS is seen as supportive of the development of the quality of the organisation’s offering, management will likely show respect for quality management, not viewing it as cost-driving but rather as being of strategic importance.

Quality management systems as a tool for documentation and standardization

A focus on providing documentation, developing procedures and ensuring consistency is said to result in a compliance-oriented approach to quality management ( Maguad, 2006 ). Implementing a QMS standard like ISO 9000 drives standardization. How standardization impacts an organisation can depend on three variables: what is standardized, how the implementation is done, and to what extent activities and processes are standardized ( Poksinska, 2007 ). First, if there is a low motivation for implementing a QMS, it is shown to result in that organization only fulfil the minimum requirements of the ISO 9000. Fulfilling only the minimum requirements may result in the implementation of a QMS that focuses only on describing the existing work practices – that is, standardizing present practices instead of practising the standard ( Poksinska , 2007, 2010 ). Second, if the result of a standardization is positive or negative is also affected by how the standard is implemented. Thus, if the standardization is done with employee involvement (enabling), supporting changes to deficient practices, or if the standard is implemented top-down (coercive), where management wants to discipline work ( Poksinska, 2007 ). Finally, the level of standardization needs to be right, as too high a level of standardization will reduce employees’ work motivation ( Poksinska (2007) .

If QMS is used as a tool for documentation and standardization, management will likely show little respect for quality management and view quality management as cost-driving and lacking in strategic importance.

Methodology

Research instrument

The study was based on a concurrent mixed method data collection strategy ( Creswell et al. , 2007 ) using both quantitative and qualitative data. Quantitative data were gathered using a survey instrument, developed through a literature review, input from senior practitioners, as well as researchers, and input from previously validated questionnaires. Specifically, this paper draws on a set of items focusing on the main function of the QMS ( Poksinska et al. , 2006 ) and management’s perceptions of quality ( Elg et al. , 2011 ) ( Table 1 ).

How would you describe the main role or purpose of the QMS?

How is the QMS used in your organisation?

How do you think management view/perceive the QMS?

For the survey, respondents from eight large-sized Swedish organisations (>1000 employees each) participated in the study (see Table 2 ). Each participating organisation identified 30–50 respondents on different hierarchical levels. The respondents within each organisation were chosen from employees who had dedicated time and responsibility for quality work. The total number of responses was 249 (response rate = 81%), the number of respondents per organisation ranged from 16 to 51. For this paper, the subset of questions used in the analysis focused on management perceptions of quality management and the overall view of the QMS. These questions were only asked of respondents with management responsibilities and resulted in a subset of 108 respondents.

For the interviews, the interviewee sample consisted of twelve quality managers (IP 1–12) with dedicated time and responsibility for quality work. Sample selection was based on organisations offering both products and services, and having established quality management work structures. The sampled organisations covered the following industries: forestry industry, equipment manufacturers, electronics industry, mechanical industry, med-tech industry, logistics industry, and aviation engineering. The interviewees in these organisations focused both product and service quality. Selection was also based on each interviewee having broad areas of responsibility for quality work and also unmediated access to higher management levels, thereby ensuring a relevant knowledge base concerning management perceptions of quality management in general, and the QMS in particular.

Data collection

The survey was administered by e-mail, including a customized invitation letter for each organization and a link to the survey (using the Web-based tool SurveyMonkey). The survey was open for one month per organization, including two rounds of reminders. The interviews were recorded and then transcribed verbatim.

Data analysis

Since the analysed statements in the quantitative data are jointly exhaustive, answers for which no alternative was chosen were considered to be missing values. After excluding rows containing missing values, 108 of the original 249 observations remained. Of these, nine had rows containing the answer “no opinion”. Since this answer cannot be interpreted as an ordinal value, these observations were excluded as well, resulting in a sample of 99 observations. Spearman’s rank correlation coefficient was used to evaluate the monotonic relationships between the ordinal variables. To depend the understanding of the correlations, the mixed method design was exploited as qualitative interview data was used to further the comprehension of the correlations. Hence, focus was on understanding the relevance and meaning of the correlations.

For the analysis of the qualitative data, the transcriptions of the interviews were uploaded into the QSR NVivo 12 software program. A coding scheme was devised using the theory of grounded propositions (see above). The interviews were then subjected to a thematic text analysis using a deductive cross-case analysis strategy ( Miles and Huberman, 1994 ). Data analysis was done by first reading through all the interviews. By using the theoretically derived coding scheme, coding can be described as influenced by the theoretical underpinnings of the propositions and as descriptive by “attributing a class of phenomena to a segment of text” ( Miles and Huberman, 1994 , p.57), based on the grounded propositions. The content of the coded data was thematically analysed whereby general similarities (or discrepancies) between the interviewees could be identified. Finally, the thematic content was evaluated against the conceptual and theoretical underpinnings to further understand the data and draw conclusions. An overview of the coding scheme with quotes illustrating how the data analysis was performed is featured in Table 3 . The results per se will be further elaborated on in the findings section.

Each code category was labelled either to signify a positive view – the use of QMS is viewed with respect in daily work, QMS is viewed as cost reducing, and the use QMS is viewed as strategically important – or to signify a negative view – the use of QMS is not viewed with respect in daily work, QMS is viewed as cost increasing and the use of QMS is not viewed as strategically important.

The study took several steps to achieve acceptable research quality, for example all questions in the survey were based on established instruments, and triangulation of data with questionnaire data and interview data was used to corroborate the findings.

On an overall level, the data shows that the respondents to a large extent agree with all the statements regarding the function and use of the QMS in their organisation ( Table 4 ).

It appears that QMS as a “tool to handle documentation”, “tool for standardisation”, and as having a “significant impact on how the organisation works” are the three statements where most respondents to some extent agree and in other words recognise their way of working with QMS. For statements where a group of respondents do not agree at all, the three other statements stand out. The statement for which most respondents do not agree is that QMS is “a tool that supports efficient management of our organisation”, followed by QMS is “a tool that helps us to fulfil our customers’ needs”, and QMS is “a tool for managing our quality work and improve the quality of our products/services”. As QMS and activities related to designing, implementing, and maintaining the system is a large part of what a quality function does, it arguably will influence how managers view quality management overall. Figure 1 shows the correlations between the level of agreement on the statements related to the function of QMS, and management’s view on quality management in terms of respect, cost and strategic importance.

First, P1 ’s focus on a business management-oriented use of QMS relates to two functions of QMS: impact on work and efficient management ( Table 1 ). These two functions of QMS correlate negatively to management viewing quality management as with a lack of respect and as being costly. On the other hand, there is a positive correlation to viewing quality management as being of strategic importance. Hence, the data points in the same directions as outlined in proposition 1. The findings from the interviews partly support proposition 1 in that management views the impact of QMS on efficient management as positive (e.g. IP8, IP10, IP12). For example, IP7 states that: “The current management at […] has a clear quality aware mentality that benefits everybody […] that works with quality”. However, management can also be perceived as showing a “lack of interest to QMS as to the purpose of quality management work” (IP1).

Second, P2 encompasses the statements on QMS as a tool focused on customer needs and a tool impacting product/service quality; these two concepts constitute what this paper refers to as an improvement-oriented use of QMS. In the same way as the statements underlying P1 , the statements of “customer needs” and “product/service quality” correlate positively to management acknowledging the strategic importance of quality management. Moreover, there are negative correlations with quality being viewed with little respect and as a costly activity. Looking at the correlation values, these are largest for the statement regarding “customer needs”, which might depend on a larger variation in the responses. The findings from the interviews are mostly in favour of P2 (e.g. IP4, IP8, IP12). Key customer requirements such as sustainability (IP4), and also the function of collecting customer information and understanding customer needs (IP3) is perceived by the management as being directly facilitated by QMS. As an example, IP12 states that: “Auditing is still a big part, because that’s one way you can tell how you’re adhering to what your customers want”. IP4 described the benefit of QMS supporting organizational success like this: “And we have this in order, it will be a competitive advantage, and it’s coming globally; it’s coming in all areas.” However, there are also perceptions of management only perceiving the use of QMS for improvement as a “tick in the box”. The interviews show various degrees of understanding QMS as a tool for improvement by management levels (e.g. IP2, IP8).

Third and last, P3 refers to a compliance-oriented use of QMS and concerns documentation and standardization. The correlations are small, but the results are mixed as compared to the other two propositions. The statement viewing QMS as a tool for documentation, displays correlations supporting parts of P3 . That is, it positively correlates with little respect for quality management and a view of it as being costly. However, the statement on documentation does not correlate with quality management being seen as strategic. Moving to the other statement on a compliance-oriented QMS use (“standardization”), the correlations do not support P3 . The use of QMS as a tool for standardization negatively correlates with all three views on quality management. It does not appear supportive of a view on quality management as costly, or of it being little respected. However, it does have a negative correlation with quality management being viewed as strategic (as outlined in P3 ). Again, the correlations are small and further investigation is needed. The interview findings related to P3 are somewhat ambiguous. Regarding management perceptions that QMS, primarily used as a tool for documentation, increases both work and costs and also reduces respect, the findings support P3 (e.g. IP1, IP2, IP5, IP6). Concerning perceptions of QMS used as a tool for standardization, statements on QMS as filling regulatory purposes recur (e.g. IP8, IP9, IP11). Standardization is viewed as both an imperative and something that is self-evident and “the right thing to do” (IP8) with references to safety and brand perception in order not to “run into problems” (IP9).

To support improved QMS usage and increase the perceived value added by a QMS, there is a need to move beyond the broad conception of QMS usage and move towards a more detailed analysis. This paper contributes to research on QMS by outlining three different ways of using QMS, rather than studying QMS usage overall. Drawing on Maguad (2006) three types of QMS usage are described as being oriented towards business management, improvement or compliance.

First, the business management-oriented use of QMS is operationalised by QMS “significantly impacting the way an organisation works”, and “is a tool that supports efficient management of an organisation”. As assumed in proposition 1, these functions appear to support that management will likely show respect for quality management and not view it as cost-driving but rather as being of strategic importance. This is in line with previous research by, for example, Bunney and Dale (1997) establishing that deployment of quality initiatives will be more successful if they are perceived as closely connected to – and potentially improving upon – current work practices.

Second, the improvement-oriented use of QMS is based on QMS as “a tool that help us to fulfil our customers’ needs”, and “a tool for managing our quality work and improve the quality of our products/services”. The proposed impact of these functions is supported, thus ensuring respect for quality management and not viewing it as costly but as strategic ( P2 ). Hence, using QMS to fulfil customer needs and improve the quality of the product or service will positively impact management perception of quality management overall. Previous research has shown that improved quality of the product/service will lead to increased customer satisfaction and loyalty ( Honore Petnji Yaya et al. , 2011 ; Parasuraman et al. , 1988 ), and that improved product/service quality is a benefit of QMS ( Psomas and Pantouvakis, 2015 ). Thus, if QMS is used in a way that can be linked to improved quality and customer satisfaction, this will likely impact management perception of the value added by the QMS.

Third, the results are more mixed in relation to P3 that QMS is used as “a tool for documentation” and “standardization”. This would be correlated with management showing little respect for quality management, viewing it as cost-driving, and not viewing it as strategic. As management perception and support is critical for QMS implementation ( Willar et al. , 2015 ), it is critical to minimize the risk with a too strong focus on documentation conveying a view of QMS as bureaucratic ( Allur et al. , 2018 ) rather than a respected and value-adding activity. However, a certification is still of value as a qualifier in certain business relations ( Boiral and Amara, 2009 ; del Castillo-Peces et al. , 2018 ). This might be a reason that the documentation focus does not appear to have the anticipated negative correlation with management viewing quality management as strategic value. Moreover, a standardisation-focussed use of QMS does not appear to reduce respect for quality management nor lead to it being seen as costly. Perhaps this can be linked to Poksinskàs (2007, 2010) notion of practising the standard rather than standardising current practices. In other words, if standardisation is done with an improvement approach rather than one of pure documentation, it will likely be perceived as beneficial. This is also linked to the function of QMS as having “impact on work”, which is classified as a business management-oriented QMS usage. If this is practised and QMS is allowed to impact actual practices, it will likely mean that QMS is used to standardise and at the same time improve existing work practices.

Overall, the findings support literature pointing to challenges of QMS in terms of focus on compliance rather than organisational efficiency ( Alič and Rusjan, 2010 ), and sometimes not being relevant for actual practice ( Poksinska et al. , 2006 ). However, by distinguishing QMS usage in the three orientations presented above, this study indicates that the documentation focus is what might be the cause for many negative perceptions of the value of QMS. On the other hand, many respondents fully agree that QMS is “a tool that helps us to fulfil our customers’ needs”, which has a relatively high correlation with management viewing quality management as strategic. Contrary to the view of limited value from QMS, this paper supports Poksinska (2007) and Lenning and Gremyr (2017) in that there is potential value in QMS, and that this perceived value will increase if QMS usage is mainly business management- and improvement-oriented, although wisely documented and standardised processes are also required to maintain a certified QMS. An important issue highlighted in the interviews is the risk of using QMS as “quality washing” by management. The interviews indicate that there is still a need to further increase knowledge and understanding within higher management levels on the value of QMS.

The data set underlying this paper is limited in size and the correlations established from the quantitative data are small, yet the qualitative data also supports the propositions. To further establish how an organisation should work with QMS to gain as much benefit as possible, more empirical studies on the three orientations (i.e. business management, improvement and compliance oriented) to QMS are suggested.

Conclusions

Based on an extended view of QMS, this paper has elaborated on three types of QMS use: business management, improvement and compliance-oriented use. The purpose was to explore how these three differing types of uses of QMS correlate with management perceptions of quality management in terms of respect, cost, and strategic importance. Overall, the conclusion is that different ways of working with QMS does not only impact the value of QMS per se , rather it also influences management’s respect for and view of quality management. In terms of difference between the three types of QMS usage, there is a correlation between business management- and improvement-oriented uses of QMS with quality management being respected, and viewed as strategic and not cost-driving. Earlier research has suggested a compliance-oriented use of QMS was the reason for many of the negative perceptions of QMS that in turn was suspected to lead to negative views on quality management in general. However, the findings of this study are somewhat contradictory to this and provide a more nuanced picture showing that, in general, compliance-oriented views might not drive negative perceptions and that it is useful to operationalise compliance into documentation and standardisation. It is suggested that a perception of QMS as having limited value is mainly due to a focus on documentation, whereas work on standardization, which is also part of a compliance-oriented QMS, does not carry similar negative implications. In summary, this study highlights how the perceived strategic value of quality management can be increased through a deliberate design, and choice of an organisation’s ways of using QMS.

Correlation matrix

Overview of organisations in the survey

Coding scheme with illustrative examples

Aba , E. , Badar , M. and Hayden , M. ( 2015 ), “ Impact of ISO 9001 certification on firms financial operating performance ”, International Journal of Quality and Reliability Management , Vol. 33 No. 1 , pp. 78 - 89 , available at: https://doi-org.proxy.lib.chalmers.se/10.1108/IJQRM-02-2014-0021

Alič , M. and Rusjan , B. ( 2010 ), “ Contribution of the ISO 9001 internal audit to business performance ”, International Journal of Quality and Reliability Management , Vol. 27 No. 8 , pp. 916 - 937 , available at: https://doi.org/10.11/02656711011075116

Allur , E. , Heras-Saizarbitoria , I. , Boiral , O. and Testa , F. ( 2018 ), “ Quality and environmental management linkage: a review of the literature ”, Sustainability , Vol. 10 No. 11 , p. 4311 , available at: https://doi.org/10.3390/su10114311

Barouch , G. and Kleinhans , S. ( 2015 ), “ Learning from criticisms of quality management ”, International Journal of Quality and Service Sciences , Vol. 7 Nos 2/3 , pp. 201 - 216 ., available at: https://doi.org/10.1108/IJQSS-02-2015-0026

Beer , M. ( 2003 ), “ Why total quality management programs do not persist: the role of management quality and implications for leading a TQM transformation ”, Decision Sciences , Vol. 34 No. 4 , pp. 623 - 642 , available at: https://doi-org/10.1111/j.1540-5414.2003.02640.x

Boiral , O. and Amara , N. ( 2009 ), “ Paradoxes of ISO 9000 performance: a configurational approach ”, Quality Management Journal , Vol. 16 No. 3 , pp. 36 - 60 , available at: https://doi.org/10.1080/10686967.2009.11918240

Bunney , H.S. and Dale , B.G. ( 1997 ), “ The implementation of quality management tools and techniques: a study ”, The TQM Magazine , Vol. 9 No. 3 , pp. 183 - 189 , available at: https://doi-org/10.1108/09544789710168966

Cândido , C.J. , Coelho , L.M. and Peixinho , R.M. ( 2016 ), “ The financial impact of a withdrawn ISO 9001 certificate ”, International Journal of Operations and Production Management , Vol. 36 No. 1 , pp. 23 - 41 , available at: https://doi-org/10.1108/IJOPM-11-2014-0540

Carnerud , D. ( 2018 ), “ 25 Years of quality management research–outlines and trends ”, International Journal of Quality and Reliability Management , Vol. 35 No. 1 , pp. 208 - 231 , available at: https://doi-org/10.1108/IJQRM-01-2017-0013

del Castillo-Peces , C. , Mercado-Idoeta , C. , Prado-Roman , M. and del Castillo-Feito , C. ( 2018 ), “ The influence of motivations and other factors on the results of implementing ISO 9001 standards ”, European Research on Management and Business Economics , Vol. 24 No. 1 , pp. 33 - 41 , available at: https://doi.org/10.1016/j.iedeen.2017.02.002

Coo , L.S. and Verma , R. ( 2002 ), “ Exploring the linkages between quality system, service quality, and performance excellence: Service providers’ perspectives ”, Quality Management Journal , Vol. 9 No. 2 , pp. 44 - 56 , available at: https://doi.org/10.1080/10686967.2002.11919009

Creswell , J.W. , Plano Clark , V.L. , Gutmann , M. and Hanson , W. ( 2007 ), “ Advanced mixed methods research designs ”, in Tashakkori , A. and Teddlie , C. (Eds), Handbook of Mixed Methods in Social and Behavioral Research , Sage , Thousand Oaks, CA , pp. 619 - 637 .

Dean , J.W. and Bowen , D.E. ( 1994 ), “ Management theory and total quality: improving research and practice through theory development ”, Academy of Management Review , Vol. 19 No. 3 , pp. 392 - 418 , available at: https://doi.org/10.5465/AMR.1994.9412271803

Dubey , R. , Gunasekaran , A. , Childe , S.J. , Papadopoulos , T. , Hazen , B.T. and Roubaud , D. ( 2018 ), “ Examining top management commitment to TQM diffusion using institutional and upper echelon theories ”, International Journal of Production Research , Vol. 56 No. 8 , pp. 2988 - 3006 , availabel at: https://doi.org/10.1080/00207543.2017.1394590

Elg , M. , Gremyr , I. , Hellström , A. and Witell , L. ( 2011 ), “ The role of quality managers in contemporary organisations ”, Total Quality Management and Business Excellence , Vol. 22 No. 8 , pp. 795 - 806 , available at: https://doi.org/10.1080/14783363.2011.593899

Gustafsson , R. , Klefsjö , B. , Berggren , E. and Granfors‐Wellemets , U. ( 2001 ), “ Experiences from implementing ISO 9000 in small enterprises – a study of Swedish organisations ”, The TQM Magazine , Vol. 13 No. 4 , pp. 232 - 246 , available at: https://doi.org/10.1108/09544780110366088

Gray , J.V. , Anand , G. and Roth , A.V. ( 2015 ), “ The influence of ISO 9000 certification on process compliance ”, Production and Operations Management , Vol. 24 No. 3 , pp. 369 - 382 , available at: https://doi.org/10.1111/poms.12252

Hackman , J.R. and Wageman , R. ( 1995 ), “ Total quality management: empirical, conceptual, and practical issues ”, Administrative Science Quarterly , Vol. 40 No. 2 , pp. 309 - 342 , available at: https://doi.org/10.2307/2393640

Honore Petnji Yaya , L. , Marimon , F. and Casadesus , M. ( 2011 ), “ Customer’s loyalty and perception of ISO 9001 in online banking ”, Industrial Management and Data Systems , Vol. 111 No. 8 , pp. 1194 - 1213 , available at: https://doi.org/10.1108/02635571111170767

ISO – International Organization for Standardization ( 2018 ), “ ISO survey of certifications to management system standards – full results ”, The ISO Survey , available at: www.iso.org/the-iso-survey.html (accessed 28 November 2020 ).

Iyer , A. , Saranga , H. and Seshadri , S. ( 2013 ), “ Effect of quality management systems and total quality management on productivity before and after: empirical evidence from the indian auto component industry ”, Production and Operations Management , Vol. 22 No. 2 , pp. 283 - 301 , available at: https://doi.org/10.1111/poms.12000

Joiner , T.A. ( 2007 ), “ Total quality management and performance: the role of organization support and co‐worker support ”, International Journal of Quality and Reliability Management , Vol. 24 No. 6 , pp. 617 - 627 , available at: https://doi.org/10.1108/02656710710757808

Kafetzopoulos , D. , Gotzamani , K. and Gkana , V. ( 2015a ), “ Relationship between quality management, innovation and competitiveness. Evidence from Greek companies ”, Journal of Manufacturing Technology Management , Vol. 26 No. 8 , available at: https://doi.org/10.1108/JMTM-02-2015-0007

Kafetzopoulos , D.P. , Psomas , E.L. and Gotzamani , K.D. ( 2015b ), “ The impact of quality management systems on the performance of manufacturing firms ”, International Journal of Quality and Reliability Management , Vol. 32 No. 4 , pp. 381 - 399 , available at: https://doi.org/10.1108/IJQRM-11-2013-0186

Karapetrovic , S. , Fa , M.C. and Saizarbitoria , I.H. ( 2010 ), “ What happened to the ISO 9000 lustre? An eight-year study ”, Total Quality Management and Business Excellence , Vol. 21 No. 3 , pp. 245 - 267 , available at: https://doi.org/10.1080/14783360903553149

Kaynak , H. ( 2003 ), “ The relationship between total quality management practices and their effects on firm performance ”, Journal of Operations Management , Vol. 21 No. 4 , pp. 405 - 435 , available at: https://doi.org/10.1016/S0272-6963(03)00004-4

Lakhal , L. , Pasin , F. and Limam , M. ( 2006 ), “ Quality management practices and their impact on performance ”, International Journal of Quality and Reliability Management , Vol. 23 No. 6 , pp. 625 - 646 , available at: https://doi.org/10.1108/02656710610672461

Lee , C.Y. ( 2004 ), “ Perception and development of total quality management in small manufacturers: an exploratory study in China ”, Journal of Small Business Management , Vol. 42 No. 1 , pp. 102 - 115 , available at: https://doi.org/10.1111/j.1540-627X.2004.00100.x

Lenning , J. and Gremyr , I. ( 2017 ), “ Making internal audits business-relevant ”, Total Quality Management and Business Excellence , Vol. 28 Nos 9/10 , pp. 1106 - 1121 , available at: https://doi.org/10.1080/14783363.2017.1303891

Levine , D.I. and Toffel , M.W. ( 2010 ), “ Quality management and job quality: how the ISO 9001 standard for quality management systems affects employees and employers ”, Management Science , Vol. 56 No. 6 , pp. 978 - 996 , available at: https://doi.org/10.1287/mnsc.1100.1159

Li , D. , Zhao , Y. , Zhang , L. , Chen , X. and Cao , C. ( 2018 ), “ Impact of quality management on green innovation ”, Journal of Cleaner Production , Vol. 170 , pp. 462 - 470 .

Maguad , B.A. ( 2006 ), “ The modern quality movement: Origins, development and trends ”, Total Quality Management and Business Excellence , Vol. 17 No. 2 , pp. 179 - 203 , available at: https://doi.org/10.1080/14783360500450608

Miles , M.B. and Huberman , A.M. ( 1994 ), Qualitative Data Analysis: An Expanded Sourcebook , Sage .

Ochieng , J. , Muturi , D. and Njihia , S.N. ( 2015 ), “ The impact of ISO 9001 implementation on organizational performance in Kenya ”, The TQM Journal , Vol. 27 No. 6 , pp. 761 - 771 , available at: https://doi.org/10.1108/TQM-06-2015-0071

Parasuraman , A. , Zeithaml , V.A. and Berry , L.L. ( 1988 ), “ Servqual: a multiple-item scale for measuring consumer perc ”, Journal of Retailing , Vol. 64 No. 1 , p. 12 .

Poksinska , B. ( 2007 ), “ Does standardization have a negative impact on working conditions? ”, Human Factors and Ergonomics in Manufacturing , Vol. 17 No. 4 , pp. 383 - 394 , available at: https://doi-org/10.1002/hfm.20080

Poksinska , B. ( 2010 ), “ When does ISO 9000 lead to improvements? ”, International Journal of Productivity and Quality Management , Vol. 5 No. 2 , pp. 124 - 136 , available at: http://dx.doi.org/10.1504/IJPQM.2010.030738

Poksinska , B. , Eklund , J.A.E. and Jörn Dahlgaard , J. ( 2006 ), “ ISO 9001:2000 in small organisations: Lost opportunities, benefits and influencing factors ”, International Journal of Quality and Reliability Management , Vol. 23 No. 5 , pp. 490 - 512 , available at: https://doi-org/10.1108/02656710610664578

Poksinska , B. , Jörn Dahlgaard , J. and Antoni , M. ( 2002 ), “ The state of ISO 9000 certification: a study of Swedish organizations ”, The TQM Magazine , Vol. 14 No. 5 , pp. 297 - 306 , available at: https://doi.org/10.1108/09544780210439734

Psomas , E. and Pantouvakis , A. ( 2015 ), “ ISO 9001 overall performance dimensions: an exploratory study ”, The TQM Journal , Vol. 27 No. 5 , pp. 519 - 531 , available at: https://doi.org/10.1108/TQM-04-2014-0037

Rönnbäck , Å. , Witell , L. and Enquist , B. ( 2009 ), “ Quality management systems and value creation ”, International Journal of Quality and Service Sciences , Vol. 1 No. 3 , pp. 241 - 254 , available at: https://doi.org/10.1108/17566690911004186

Sadikoglu , E. and Zehir , C. ( 2010 ), “ Investigating the effects of innovation and employee performance on the relationship between total quality management practices and firm performance: an empirical study of Turkish firms ”, International Journal of Production Economics , Vol. 127 No. 1 , pp. 13 - 26 , available at: https://doi.org/10.1016/j.ijpe.2010.02.013

Sampaio , P. , Saraiva , P. and Guimarães Rodrigues , A. ( 2009 ), “ ISO 9001 certification research: questions, answers and approaches ”, International Journal of Quality and Reliability Management , Vol. 26 No. 1 , pp. 38 - 58 , available at: https://doi.org/10.1108/02656710910924161

Siva , V. , Gremyr , I. , Bergquist , B. , Garvare , R. , Zobel , T. and Isaksson , R. ( 2016 ), “ The support of quality management to sustainable development: a literature review ”, Journal of Cleaner Production , Vol. 138 , pp. 148 - 157 , available at: https://doi.org/10.1016/j.jclepro.2016.01.020

Sousa , R. and Voss , C.A. ( 2002 ), “ Quality management re-visited: a reflective review and agenda for future research ”, Journal of Operations Management , Vol. 20 No. 1 , pp. 91 - 109 , available at: https://doi.org/10.1016/S0272-6963(01)00088-2

Willar , D. , Coffey , V. and Trigunarsyah , B. ( 2015 ), “ Examining the implementation of ISO 9001 in Indonesian construction companies ”, Vol. 27 No. 1 , pp. 94 - 107 , available at: https://doi.org/10.1108/TQM-08-2012-0060

Zelnik , M. , Maletič , M. , Maletič , D. and Gomišček , B. ( 2012 ), “ Quality management systems as a link between management and employees ”, Total Quality Management and Business Excellence , Vol. 23 No. 1 , pp. 45 - 62 , available at: https://doi-org/10.1080/14783363.2011.637781

Acknowledgements

The authors are grateful for the support from the Swedish Quality Management Academy and the organisations participating in this study. Further, we acknowledge financial support from the Production Area of Advance at Chalmers and the HELIX Competence Centre at Linköping University.

Corresponding author

Related articles, we’re listening — tell us what you think, something didn’t work….

Report bugs here

All feedback is valuable

Please share your general feedback

Join us on our journey

Platform update page.

Visit emeraldpublishing.com/platformupdate to discover the latest news and updates

Questions & More Information

Answers to the most commonly asked questions here

A framework for quality management research and an associated measurement instrument

Research output : Contribution to journal › Article › peer-review

Research on quality incorporates a range of concerns, including quality definition and management, and such specific mechanisms as statistical quality control (SQC). However, though research in statistical quality control has evolved in a scientific and rigorous fashion, based on the early works of Shewhart, Juran, Deming and others, the study of other aspects of quality, particularly quality management, has not evolved in a similarly rigorous fashion. Theory development and measurement issues related to reliability and validity are particularly weak in the quality management literature. Starting from a strategic perspective of the organization, this paper identifies and substantiates the key dimensions of quality management, then tests the measurement of those dimensions for reliability and validity. In doing so, it establishes a clear framework for subsequent research and for evaluation of quality management programs by practitioners. In order to specify the important dimensions of quality management, a thorough search of the relevant literature was undertaken. Quality management is defined as an approach to achieving and sustaining high quality output; thus, we employ a process definition, emphasizing inputs (management practices) rather than outputs (quality performance) in our analysis. Quality management is first viewed as an element of the integrated approach known as World Class Manufacturing; quality management supports and is supported by JIT, human resources management, top management support, technology management and strategic management. The key dimensions of quality management are then articulated. Top management support creates an environment in which quality management activities are rewarded. These activities are related to quality information systems, process management, product design, work force management, supplier involvement and customer involvement. They are used in concert to support the continuous improvement of manufacturing capability. As manufacturing capability and quality performance improve, a plant achieves and sustains a competitive advantage. This, in turn, provides feedback, reinforcement and resources to top management, which stimulates continuous improvement. Based on the seven dimensions of quality management identified in this paper, a set of 14 perceptual scales was developed. The scales were assessed for reliability and validity with a sample of 716 respondents at 42 plants in the U.S. in the transportation components, electronics and machinery industries. Reliability is broadly defined as the degree to which scales are free from error and, therefore, consistent. The use of reliable scales provides assurance that the obtained results will be stable. Application of Cronbach's alpha both across the board and by industry and nationality subsamples refined the original group of 14 scales to 11 internally consistent scales. Validity refers to the degree to which scales truly measure the constructs which they are intended to measure. This provides academic and industry users with confidence that the scales measure important constructs which are related to independent measures of the same constructs, and that each scale measures a single construct. It was concluded that the scales, and the instrument as a whole, are valid measures of quality management practices. Thus, the scales may be used with confidence by both researchers and industry users to measure quality management practices, with the ability to generalize beyond the immediate sample. This paper makes several important contributions to the area of quality management. It proposes an emergent theory of quality management and links it to the literature. Because the proposed scales are reliable and valid, they may be used by other researchers for hypothesis testing and by practitioners for assessing quality management practices in their plants and for internal and external benchmarking. Finally, the paper provides a step-by-step approach and criteria for conducting reliability and validity analysis of a measurement instrument.

Publisher link

• 10.1016/S0272-6963(97)90004-8

Other files and links

• Link to publication in Scopus

Fingerprint

• Quality Management Keyphrases 100%
• Measurement Instrument Keyphrases 100%
• Quality Management Research Keyphrases 100%
• Quality Dimensions Keyphrases 33%
• Quality Management Practices Keyphrases 25%
• Specific Industry Economics, Econometrics and Finance 23%
• Quality Assessment Keyphrases 16%
• Quality Performance Keyphrases 16%

T1 - A framework for quality management research and an associated measurement instrument

AU - Flynn, Barbara B.

AU - Schroeder, Roger G.

AU - Sakakibara, Sadao

PY - 1994/3

Y1 - 1994/3

N2 - Research on quality incorporates a range of concerns, including quality definition and management, and such specific mechanisms as statistical quality control (SQC). However, though research in statistical quality control has evolved in a scientific and rigorous fashion, based on the early works of Shewhart, Juran, Deming and others, the study of other aspects of quality, particularly quality management, has not evolved in a similarly rigorous fashion. Theory development and measurement issues related to reliability and validity are particularly weak in the quality management literature. Starting from a strategic perspective of the organization, this paper identifies and substantiates the key dimensions of quality management, then tests the measurement of those dimensions for reliability and validity. In doing so, it establishes a clear framework for subsequent research and for evaluation of quality management programs by practitioners. In order to specify the important dimensions of quality management, a thorough search of the relevant literature was undertaken. Quality management is defined as an approach to achieving and sustaining high quality output; thus, we employ a process definition, emphasizing inputs (management practices) rather than outputs (quality performance) in our analysis. Quality management is first viewed as an element of the integrated approach known as World Class Manufacturing; quality management supports and is supported by JIT, human resources management, top management support, technology management and strategic management. The key dimensions of quality management are then articulated. Top management support creates an environment in which quality management activities are rewarded. These activities are related to quality information systems, process management, product design, work force management, supplier involvement and customer involvement. They are used in concert to support the continuous improvement of manufacturing capability. As manufacturing capability and quality performance improve, a plant achieves and sustains a competitive advantage. This, in turn, provides feedback, reinforcement and resources to top management, which stimulates continuous improvement. Based on the seven dimensions of quality management identified in this paper, a set of 14 perceptual scales was developed. The scales were assessed for reliability and validity with a sample of 716 respondents at 42 plants in the U.S. in the transportation components, electronics and machinery industries. Reliability is broadly defined as the degree to which scales are free from error and, therefore, consistent. The use of reliable scales provides assurance that the obtained results will be stable. Application of Cronbach's alpha both across the board and by industry and nationality subsamples refined the original group of 14 scales to 11 internally consistent scales. Validity refers to the degree to which scales truly measure the constructs which they are intended to measure. This provides academic and industry users with confidence that the scales measure important constructs which are related to independent measures of the same constructs, and that each scale measures a single construct. It was concluded that the scales, and the instrument as a whole, are valid measures of quality management practices. Thus, the scales may be used with confidence by both researchers and industry users to measure quality management practices, with the ability to generalize beyond the immediate sample. This paper makes several important contributions to the area of quality management. It proposes an emergent theory of quality management and links it to the literature. Because the proposed scales are reliable and valid, they may be used by other researchers for hypothesis testing and by practitioners for assessing quality management practices in their plants and for internal and external benchmarking. Finally, the paper provides a step-by-step approach and criteria for conducting reliability and validity analysis of a measurement instrument.

AB - Research on quality incorporates a range of concerns, including quality definition and management, and such specific mechanisms as statistical quality control (SQC). However, though research in statistical quality control has evolved in a scientific and rigorous fashion, based on the early works of Shewhart, Juran, Deming and others, the study of other aspects of quality, particularly quality management, has not evolved in a similarly rigorous fashion. Theory development and measurement issues related to reliability and validity are particularly weak in the quality management literature. Starting from a strategic perspective of the organization, this paper identifies and substantiates the key dimensions of quality management, then tests the measurement of those dimensions for reliability and validity. In doing so, it establishes a clear framework for subsequent research and for evaluation of quality management programs by practitioners. In order to specify the important dimensions of quality management, a thorough search of the relevant literature was undertaken. Quality management is defined as an approach to achieving and sustaining high quality output; thus, we employ a process definition, emphasizing inputs (management practices) rather than outputs (quality performance) in our analysis. Quality management is first viewed as an element of the integrated approach known as World Class Manufacturing; quality management supports and is supported by JIT, human resources management, top management support, technology management and strategic management. The key dimensions of quality management are then articulated. Top management support creates an environment in which quality management activities are rewarded. These activities are related to quality information systems, process management, product design, work force management, supplier involvement and customer involvement. They are used in concert to support the continuous improvement of manufacturing capability. As manufacturing capability and quality performance improve, a plant achieves and sustains a competitive advantage. This, in turn, provides feedback, reinforcement and resources to top management, which stimulates continuous improvement. Based on the seven dimensions of quality management identified in this paper, a set of 14 perceptual scales was developed. The scales were assessed for reliability and validity with a sample of 716 respondents at 42 plants in the U.S. in the transportation components, electronics and machinery industries. Reliability is broadly defined as the degree to which scales are free from error and, therefore, consistent. The use of reliable scales provides assurance that the obtained results will be stable. Application of Cronbach's alpha both across the board and by industry and nationality subsamples refined the original group of 14 scales to 11 internally consistent scales. Validity refers to the degree to which scales truly measure the constructs which they are intended to measure. This provides academic and industry users with confidence that the scales measure important constructs which are related to independent measures of the same constructs, and that each scale measures a single construct. It was concluded that the scales, and the instrument as a whole, are valid measures of quality management practices. Thus, the scales may be used with confidence by both researchers and industry users to measure quality management practices, with the ability to generalize beyond the immediate sample. This paper makes several important contributions to the area of quality management. It proposes an emergent theory of quality management and links it to the literature. Because the proposed scales are reliable and valid, they may be used by other researchers for hypothesis testing and by practitioners for assessing quality management practices in their plants and for internal and external benchmarking. Finally, the paper provides a step-by-step approach and criteria for conducting reliability and validity analysis of a measurement instrument.

UR - http://www.scopus.com/inward/record.url?scp=0028401194&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0028401194&partnerID=8YFLogxK

U2 - 10.1016/S0272-6963(97)90004-8

DO - 10.1016/S0272-6963(97)90004-8

M3 - Article

AN - SCOPUS:0028401194

SN - 0272-6963

JO - Journal of Operations Management

JF - Journal of Operations Management

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Adv Pract Oncol
• v.12(4); 2021 May

Quality Improvement Projects and Clinical Research Studies

Every day, I witness firsthand the amazing things that advanced practitioners and nurse scientists accomplish. Through the conduct of quality improvement (QI) projects and clinical research studies, advanced practitioners and nurse scientists have the opportunity to contribute exponentially not only to their organizations, but also towards personal and professional growth.

Recently, the associate editors and staff at JADPRO convened to discuss the types of articles our readership may be interested in. Since we at JADPRO believe that QI projects and clinical research studies are highly valuable methods to improve clinical processes or seek answers to questions, you will see that we have highlighted various QI and research projects within the Research and Scholarship column of this and future issues. There have also been articles published in JADPRO about QI and research ( Gillespie, 2018 ; Kurtin & Taher, 2020 ). As a refresher, let’s explore the differences between a QI project and clinical research.

Quality Improvement

As leaders in health care, advanced practitioners often conduct QI projects to improve their internal processes or streamline clinical workflow. These QI projects use a multidisciplinary team comprising a team leader as well as nurses, PAs, pharmacists, physicians, social workers, and program administrators to address important questions that impact patients. Since QI projects use strategic processes and methods to analyze existing data and all patients participate, institutional review board (IRB) approval is usually not needed. Common frameworks, such as Lean, Six Sigma, and the Model for Improvement can be used. An attractive aspect of QI projects is that these are generally quicker to conduct and report on than clinical research, and often with quantifiable benefits to a large group within a system ( Table 1 ).

Clinical Research

Conducting clinical research through an IRB-approved study is another area in which advanced practitioners and nurse scientists gain new knowledge and contribute to scientific evidence-based practice. Research is intended for specific groups of patients who are protected from harm through the IRB and ethical principles. Research can potentially benefit a larger group, but benefits to participants are often unknown during the study period.

Clinical research poses many challenges at various stages of what can be a lengthy process. First, the researcher conducts a review of the literature to identify gaps in existing knowledge. Then, the researcher must be diligent in their self-reflection (is this phenomenon worth studying?) and in developing the sampling and statistical methods to ensure validity and reliability of the research ( Higgins & Straub, 2006 ). A team of additional researchers and support staff is integral to completing the research and disseminating findings. A well-designed clinical trial is worth the time and effort it takes to answer important clinical questions.

So, as an advanced practitioner, would a QI project be better to conduct than a clinical research study? That depends. A QI project uses a specific process, measures, and existing data to improve outcomes in a specific group. A research study uses an IRB-approved study protocol, strategic methods, and generates new data to hopefully benefit a larger group.

In This Issue

Both QI projects and clinical research can provide evidence to base one’s interventions on and enhance the lives of patients in one way or another. I hope you will agree that this issue is filled with valuable information on a wide range of topics. In the following pages, you will learn about findings of a QI project to integrate palliative care into ambulatory oncology. In a phenomenological study, Carrasco explores patient communication preferences around cancer symptom reporting during cancer treatment.

We have two excellent review articles for you as well. Rogers and colleagues review the management of hematologic adverse events of immune checkpoint inhibitors, and Lemke reviews the evidence for use of ginseng in the management of cancer-related fatigue. In Grand Rounds, Flagg and Pierce share an interesting case of essential thrombocythemia in a 15-year-old, with valuable considerations in the pediatric population. May and colleagues review practical considerations for integrating biosimilars into clinical practice, and Moore and Thompson review BTK inhibitors in B-cell malignancies.

• Higgins P. A., & Straub A. J. (2006). Understanding the error of our ways: Mapping the concepts of validity and reliability . Nursing Outlook , 54 ( 1 ), 23–29. 10.1016/j.outlook.2004.12.004 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Gillespie T. W. (2018). Do the right study: Quality improvement projects and human subject research—both valuable, simply different . Journal of the Advanced Practitioner in Oncology , 9 ( 5 ), 471–473. 10.6004/jadpro.2018.9.5.1 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Kurtin S. E., & Taher R. (2020). Clinical trial design and drug approval in oncology: A primer for the advanced practitioner in oncology . Journal of the Advanced Practitioner in Oncology , 11 ( 7 ), 736–751. 10.6004/jadpro.2020.11.7.7 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]

• Research article
• Open access
• Published: 15 April 2024

What is quality in long covid care? Lessons from a national quality improvement collaborative and multi-site ethnography

• Trisha Greenhalgh   ORCID: orcid.org/0000-0003-2369-8088 1 ,
• Julie L. Darbyshire 1 ,
• Cassie Lee 2 ,
• Emma Ladds 1 &
• Jenny Ceolta-Smith 3  

BMC Medicine volume  22 , Article number:  159 ( 2024 ) Cite this article

1263 Accesses

59 Altmetric

Metrics details

Long covid (post covid-19 condition) is a complex condition with diverse manifestations, uncertain prognosis and wide variation in current approaches to management. There have been calls for formal quality standards to reduce a so-called “postcode lottery” of care. The original aim of this study—to examine the nature of quality in long covid care and reduce unwarranted variation in services—evolved to focus on examining the reasons why standardizing care was so challenging in this condition.

In 2021–2023, we ran a quality improvement collaborative across 10 UK sites. The dataset reported here was mostly but not entirely qualitative. It included data on the origins and current context of each clinic, interviews with staff and patients, and ethnographic observations at 13 clinics (50 consultations) and 45 multidisciplinary team (MDT) meetings (244 patient cases). Data collection and analysis were informed by relevant lenses from clinical care (e.g. evidence-based guidelines), improvement science (e.g. quality improvement cycles) and philosophy of knowledge.

Participating clinics made progress towards standardizing assessment and management in some topics; some variation remained but this could usually be explained. Clinics had different histories and path dependencies, occupied a different place in their healthcare ecosystem and served a varied caseload including a high proportion of patients with comorbidities. A key mechanism for achieving high-quality long covid care was when local MDTs deliberated on unusual, complex or challenging cases for which evidence-based guidelines provided no easy answers. In such cases, collective learning occurred through idiographic (case-based) reasoning , in which practitioners build lessons from the particular to the general. This contrasts with the nomothetic reasoning implicit in evidence-based guidelines, in which reasoning is assumed to go from the general (e.g. findings of clinical trials) to the particular (management of individual patients).

Not all variation in long covid services is unwarranted. Largely because long covid’s manifestations are so varied and comorbidities common, generic “evidence-based” standards require much individual adaptation. In this complex condition, quality improvement resources may be productively spent supporting MDTs to optimise their case-based learning through interdisciplinary discussion. Quality assessment of a long covid service should include review of a sample of individual cases to assess how guidelines have been interpreted and personalized to meet patients’ unique needs.

Study registration

NCT05057260, ISRCTN15022307.

Peer Review reports

The term “long covid” [ 1 ] means prolonged symptoms following SARS-CoV-2 infection not explained by an alternative diagnosis [ 2 ]. It embraces the US term “post-covid conditions” (symptoms beyond 4 weeks) [ 3 ], the UK terms “ongoing symptomatic covid-19” (symptoms lasting 4–12 weeks) and “post covid-19 syndrome” (symptoms beyond 12 weeks) [ 4 ] and the World Health Organization’s “post covid-19 condition” (symptoms occurring beyond 3 months and persisting for at least 2 months) [ 5 ]. Long covid thus defined is extremely common. In UK, for example, 1.8 million of a population of 67 million met the criteria for long covid in early 2023 and 41% of these had been unwell for more than 2 years [ 6 ].

Long covid is characterized by a constellation of symptoms which may include breathlessness, fatigue, muscle and joint pain, chest pain, memory loss and impaired concentration (“brain fog”), sleep disturbance, depression, anxiety, palpitations, dizziness, gastrointestinal problems such as diarrhea, skin rashes and allergy to food or drugs [ 2 ]. These lead to difficulties with essential daily activities such as washing and dressing, impaired exercise tolerance and ability to work, and reduced quality of life [ 2 , 7 , 8 ]. Symptoms typically cluster (e.g. in different patients, long covid may be dominated by fatigue, by breathlessness or by palpitations and dizziness) [ 9 , 10 ]. Long covid may follow a fairly constant course or a relapsing and remitting one, perhaps with specific triggers [ 11 ]. Overlaps between fatigue-dominant subtypes of long covid, myalgic encephalomyelitis and chronic fatigue syndrome have been hypothesized [ 12 ] but at the time of writing remain unproven.

Long covid has been a contested condition from the outset. Whilst long-term sequelae following other coronavirus (SARS and MERS) infections were already well-documented [ 13 ], SARS-CoV-2 was originally thought to cause a short-lived respiratory illness from which the patient either died or recovered [ 14 ]. Some clinicians dismissed protracted or relapsing symptoms as due to anxiety or deconditioning, especially if the patient had not had laboratory-confirmed covid-19. People with long covid got together in online groups and shared accounts of their symptoms and experiences of such “gaslighting” in their healthcare encounters [ 15 , 16 ]. Some groups conducted surveys on their members, documenting the wide range of symptoms listed in the previous paragraph and showing that whilst long covid is more commonly a sequel to severe acute covid-19, it can (rarely) follow a mild or even asymptomatic acute infection [ 17 ].

Early publications on long covid depicted a post-pneumonia syndrome which primarily affected patients who had been hospitalized (and sometimes ventilated) [ 18 , 19 ]. Later, covid-19 was recognized to be a multi-organ inflammatory condition (the pneumonia, for example, was reclassified as pneumonitis ) and its long-term sequelae attributed to a combination of viral persistence, dysregulated immune response (including auto-immunity), endothelial dysfunction and immuno-thrombosis, leading to damage to the lining of small blood vessels and (thence) interference with transfer of oxygen and nutrients to vital organs [ 20 , 21 , 22 , 23 , 24 ]. But most such studies were highly specialized, laboratory-based and written primarily for an audience of fellow laboratory researchers. Despite demonstrating mean differences in a number of metabolic variables, they failed to identify a reliable biomarker that could be used routinely in the clinic to rule a diagnosis of long covid in or out. Whilst the evidence base from laboratory studies grew rapidly, it had little influence on clinical management—partly because most long covid clinics had been set up with impressive speed by front-line clinical teams to address an immediate crisis, with little or no input from immunologists, virologists or metabolic specialists [ 25 ].

Studies of the patient experience revealed wide geographical variation in whether any long covid services were provided and (if they were) which patients were eligible for these and what tests and treatments were available [ 26 ]. An interim UK clinical guideline for long covid had been produced at speed and published in December 2020 [ 27 ], but it was uncertain about diagnostic criteria, investigations, treatments and prognosis. Early policy recommendations for long covid services in England, based on wide consultation across UK, had proposed a tiered service with “tier 1” being supported self-management, “tier 2” generalist assessment and management in primary care, “tier 3” specialist rehabilitation or respiratory follow-up with oversight from a consultant physician and “tier 4” tertiary care for patients with complications or complex needs [ 28 ]. In 2021, ring-fenced funding was allocated to establish 90 multidisciplinary long covid clinics in England [ 29 ]; some clinics were also set up with local funding in Scotland and Wales. These clinics varied widely in eligibility criteria, referral pathways, staffing mix (some had no doctors at all) and investigations and treatments offered. A further policy document on improving long covid services was published in 2022 [ 30 ]; it recommended that specialist long covid clinics should continue, though the long-term funding of these services remains uncertain [ 31 ]. To build the evidence base for delivering long covid services, major programs of publicly funded research were commenced in both UK [ 32 ] and USA [ 33 ].

In short, at the time this study began (late 2021), there appeared to be much scope for a program of quality improvement which would capture fast-emerging research findings, establish evidence-based standards and ensure these were rapidly disseminated and consistently adopted across both specialist long covid services and in primary care.

Quality improvement collaboratives

The quality improvement movement in healthcare was born in the early 1980s when clinicians and policymakers US and UK [ 34 , 35 , 36 , 37 ] began to draw on insights from outside the sector [ 38 , 39 , 40 ]. Adapting a total quality management approach that had previously transformed the Japanese car industry, they sought to improve efficiency, reduce waste, shift to treating the upstream causes of problems (hence preventing disease) and help all services approach the standards of excellence achieved by the best. They developed an approach based on (a) understanding healthcare as a complex system (especially its key interdependencies and workflows), (b) analysing and addressing variation within the system, (c) learning continuously from real-world data and (d) developing leaders who could motivate people and help them change structures and processes [ 41 , 42 , 43 , 44 ].

Quality improvement collaboratives (originally termed “breakthrough collaboratives” [ 45 ]), in which representatives from different healthcare organizations come together to address a common problem, identify best practice, set goals, share data and initiate and evaluate improvement efforts [ 46 ], are one model used to deliver system-wide quality improvement. It is widely assumed that these collaboratives work because—and to the extent that—they identify, interpret and implement high-quality evidence (e.g. from randomized controlled trials).

Research on why quality improvement collaboratives succeed or fail has produced the following list of critical success factors: taking a whole-system approach, selecting a topic and goal that fits with organizations’ priorities, fostering a culture of quality improvement (e.g. that quality is everyone’s job), engagement of everyone (including the multidisciplinary clinical team, managers, patients and families) in the improvement effort, clearly defining people’s roles and contribution, engaging people in preliminary groundwork, providing organizational-level support (e.g. chief executive endorsement, protected staff time, training and support for teams, resources, quality-focused human resource practices, external facilitation if needed), training in specific quality improvement techniques (e.g. plan-do-study-act cycle), attending to the human dimension (including cultivating trust and working to ensure shared vision and buy-in), continuously generating reliable data on both processes (e.g. current practice) and outcomes (clinical, satisfaction) and a “learning system” infrastructure in which knowledge that is generated feeds into individual, team and organizational learning [ 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 ].

The quality improvement collaborative approach has delivered many successes but it has been criticized at a theoretical level for over-simplifying the social science of human motivation and behaviour and for adopting a somewhat mechanical approach to the study of complex systems [ 55 , 56 ]. Adaptations of the original quality improvement methodology (e.g. from Sweden [ 57 , 58 ]) have placed greater emphasis on human values and meaning-making, on the grounds that reducing the complexities of a system-wide quality improvement effort to a set of abstract and generic “success factors” will miss unique aspects of the case such as historical path dependencies, personalities, framing and meaning-making and micropolitics [ 59 ].

Perhaps this explains why, when the abovementioned factors are met, a quality improvement collaborative’s success is more likely but is not guaranteed, as a systematic review demonstrated [ 60 ]. Some well-designed and well-resourced collaboratives addressing clear knowledge gaps produced few or no sustained changes in key outcome measures [ 49 , 53 , 60 , 61 , 62 ]. To identify why this might be, a detailed understanding of a service’s history, current challenges and contextual constraints is needed. This explains our decision, part-way through the study reported here, to collect rich contextual data on participating sites so as to better explain success or failure of our own collaborative.

Warranted and unwarranted variation in clinical practice

A generation ago, Wennberg described most variation in clinical practice as “unwarranted” (which he defined as variation in the utilization of health care services that cannot be explained by variation in patient illness or patient preferences) [ 63 ]. Others coined the term “postcode lottery” to depict how such variation allegedly impacted on health outcomes [ 64 ]. Wennberg and colleagues’ Atlas of Variation , introduced in 1999 [ 65 ], and its UK equivalent, introduced in 2010 [ 66 ], described wide regional differences in the rates of procedures from arthroscopy to hysterectomy, and were used to prompt services to identify and address examples of under-treatment, mis-treatment and over-treatment. Numerous similar initiatives, mostly based on hospital activity statistics, have been introduced around the world [ 66 , 67 , 68 , 69 ]. Sutherland and Levesque’s proposed framework for analysing variation, for example, has three domains: capacity (broadly, whether sufficient resources are allocated at organizational level and whether individuals have the time and headspace to get involved), evidence (the extent to which evidence-based guidelines exist and are followed), and agency (e.g. whether clinicians are engaged with the issue and the effect of patient choice) [ 70 ].

Whilst it is clearly a good idea to identify unwarranted variation in practice, it is also important to acknowledge that variation can be warranted . The very act of measuring and describing variation carries great rhetorical power, since revealing geographical variation in any chosen metric effectively frames this as a problem with a conceptually simple solution (reducing variation) that will appeal to both politicians and the public [ 71 ]. The temptation to expose variation (e.g. via visualizations such as maps) and address it in mechanistic ways should be resisted until we have fully understood the reasons why it exists, which may include perverse incentives, insufficient opportunities to discuss cases with colleagues, weak or absent feedback on practice, unclear decision processes, contested definitions of appropriate care and professional challenges to guidelines [ 72 ].

Research question, aims and objectives

Research question.

What is quality in long covid care and how can it best be achieved?

To identify best practice and reduce unwarranted variation in UK long covid services.

To explain aspects of variation in long covid services that are or may be warranted.

Our original objectives were to:

Establish a quality improvement collaborative for 10 long covid clinics across UK.

Use quality improvement methods in collaboration with patients and clinic staff to prioritize aspects of care to improve. For each priority topic, identify best (evidence-informed) clinical practice, measure performance in each clinic, compare performance with a best practice benchmark and improve performance.

Produce organizational case studies of participating long covid clinics to explain their origins, evolution, leadership, ethos, population served, patient pathways and place in the wider healthcare ecosystem.

Examine these case studies to explain variation in practice, especially in topics where the quality improvement cycle proves difficult to follow or has limited impact.

The LOCOMOTION study

LOCOMOTION (LOng COvid Multidisciplinary consortium Optimising Treatments and services across the NHS) was a 30-month multi-site case study of 10 long covid clinics (8 in England, 1 in Wales and 1 in Scotland), beginning in 2021, which sought to optimise long covid care. Each clinic offered multidisciplinary care to patients referred from primary or secondary care (and, in some cases, self-referred), and held regular multidisciplinary team (MDT) meetings, mostly online via Microsoft Teams, to discuss cases. A study protocol for LOCOMOTION, with details of ethical approvals, management, governance and patient involvement has been published [ 25 ]. The three main work packages addressed quality improvement, technology-supported patient self-management and phenotyping and symptom clustering. This paper reports on the first work package, focusing mainly on qualitative findings.

Setting up the quality improvement collaborative

We broadly followed standard methodology for “breakthrough” quality improvement collaboratives [ 44 , 45 ], with two exceptions. First, because of geographical distance, continuing pandemic precautions and developments in videoconferencing technology, meetings were held online. Second, unlike in the original breakthrough model, patients were included in the collaborative, reflecting the cultural change towards patient partnerships since the model was originally proposed 40 years ago.

Each site appointed a clinical research fellow (doctor, nurse or allied health professional) funded partly by the LOCOMOTION study and partly with clinical sessions; some were existing staff who were backfilled to take on a research role whilst others were new appointments. The quality improvement meetings were held approximately every 8 weeks on Microsoft Teams and lasted about 2 h; there was an agenda and a chair, and meetings were recorded with consent. The clinical research fellow from each clinic attended, sometimes joined by the clinical lead for that site. In the initial meeting, the group proposed and prioritized topics before merging their consensus with the list of priority topics generated separately by patients (there was much overlap but also some differences).

In subsequent meetings, participants attempted to reach consensus on how to define, measure and achieve quality for each priority topic in turn, implement this approach in their own clinic and monitor its impact. Clinical leads prepared illustrative clinical cases and summaries of the research evidence, which they presented using Microsoft Powerpoint; the group then worked towards consensus on the implications for practice through general discussion. Clinical research fellows assisted with literature searches, collected baseline data from their own clinic, prepared and presented anonymized case examples, and contributed to collaborative goal-setting for improvement. Progress on each topic was reviewed at a later meeting after an agreed interval.

An additional element of this work package was semi-structured interviews with 29 patients, recruited from 9 of the 10 participating sites, about their clinic experiences with a view to feeding into service improvement (in the other site, no patient volunteered).

Our patient advisory group initially met separately from the quality improvement collaborative. They designed a short survey of current practice and sent it to each clinic; the results of this informed a prioritization exercise for topics where they considered change was needed. The patient-generated list was tabled at the quality improvement collaborative discussions, but patients were understandably keen to join these discussions directly. After about 9 months, some patient advisory group members joined the regular collaborative meetings. This dynamic was not without its tensions, since sharing performance data requires trust and there were some concerns about confidentiality when real patient cases were discussed with other patients present.

How evidence-informed quality targets were set

At the time the study began, there were no published large-scale randomized controlled trials of any interventions for long covid. We therefore followed a model used successfully in other quality improvement efforts where research evidence was limited or absent or it did not translate unambiguously into models for current services. In such circumstances, the best evidence may be custom and practice in the best-performing units. The quality improvement effort becomes oriented to what one group of researchers called “potentially better practices”—that is, practices that are “developed through analysis of the processes of care, literature review, and site visits” (page 14) [ 73 ]. The idea was that facilitated discussion among clinical teams, drawing on published research where available but also incorporating clinical experience, established practice and systematic analysis of performance data across participating clinics would surface these “potentially better practices”—an approach which, though not formally tested in controlled trials, appears to be associated with improved outcomes [ 46 , 73 ].

Adding an ethnographic component

Following limited progress made on some topics that had been designated high priority, we interviewed all 10 clinical research fellows (either individually or, in two cases, with a senior clinician present) and 18 other clinic staff (five individually plus two groups of 5 and 8), along with additional informal discussions, to explore the challenges of implementing the changes that had been agreed. These interviews were not audiotaped but detailed notes were made and typed up immediately afterwards. It became evident that some aspects of what the collaborative had deemed “evidence-informed” care were contested by front-line clinic staff, perceived as irrelevant to the service they were delivering, or considered impossible to implement. To unpack these issues further, the research protocol was amended to include an ethnographic component.

TG and EL (academic general practitioners) and JLD (a qualitative researcher with a PhD in the patient experience) attended a total of 45 MDT meetings in participating clinics (mostly online or hybrid). Staff were informed in advance that there would be an observer present; nobody objected. We noted brief demographic and clinical details of cases discussed (but no identifying data), dilemmas and uncertainties on which discussions focused, and how different staff members contributed.

TG made 13 in-person visits to participating long covid clinics. Staff were notified in advance; all were happy to be observed. Visits lasted between 5 and 8 h (54 h in total). We observed support staff booking patients in and processing requests and referrals, and shadowed different clinical staff in turn as they saw patients. Patients were informed of our presence and its purpose beforehand and given the opportunity to decline (three of 53 patients approached did). We discussed aspects of each case with the clinician after the patient left. When invited, we took breaks with staff and used these as an opportunity to ask them informally what it was like working in the clinic.

Ethnographic observation, analysis and reporting was geared to generating a rich interpretive account of the clinical, operational and interpersonal features of each clinic—what Van Maanen calls an “impressionist tales” [ 74 ]. Our work was also guided by the principles set out by Golden-Biddle and Locke, namely authenticity (spending time in the field and basing interpretations on these direct observations), plausibility (creating a plausible account through rich persuasive description) and criticality (e.g. reflexively examining our own assumptions) [ 75 ]. Our collection and analysis of qualitative data was informed by our own professional backgrounds (two general practitioners, one physical therapist, two non-clinicians).

In both MDTs and clinics, we took contemporaneous notes by hand and typed these up immediately afterwards.

Data management and analysis

Typed interview notes and field notes from clinics were collated in a set of Word documents, one for each clinic attended. They were analysed thematically [ 76 ] with attention to the literature on quality improvement and variation (see “ Background ”). Interim summaries were prepared on each clinic, setting out the narrative of how it had been established, its ethos and leadership, setting and staffing, population served and key links with other parts of the local healthcare ecosystem.

Minutes and field notes from the quality improvement collaborative meetings were summarized topic by topic, including initial data collected by the researchers-in-residence, improvement actions taken (or attempted) in that clinic, and any follow-up data shared. Progress or lack of it was interpreted in relation to the contextual case summary for that clinic.

Patient cases seen in clinic, and those discussed by MDTs, were summarized as brief case narratives in Word documents. Using the constant comparative method [ 77 ], we produced an initial synthesis of the clinical picture and principles of management based on the first 10 patient cases seen, and refined this as each additional case was added. Demographic and brief clinical and social details were also logged on Excel spreadsheets. When writing up clinical cases, we used the technique of composite case construction (in which we drew on several actual cases to generate a fictitious one, thereby protecting anonymity whilst preserving key empirical findings [ 78 ]); any names reported in this paper are pseudonyms.

Member checking

A summary was prepared for each clinic, including a narrative of the clinic’s own history and a summary of key quality issues raised across the ten clinics. These summaries included examples from real cases in our dataset. These were shared with the clinical research fellow and a senior clinician from the clinic, and amended in response to feedback. We also shared these summaries with representatives from the patient advisory group.

Overview of dataset

This study generated three complementary datasets. First, the video recordings, minutes, and field notes of 12 quality improvement collaborative meetings, along with the evidence summaries prepared for these meetings and clinic summaries (e.g. descriptions of current practice, audits) submitted by the clinical research fellows. This dataset illustrated wide variation in practice, and (in many topics) gaps or ambiguities in the evidence base.

Second, interviews with staff ( n  = 30) and patients ( n  = 29) from the clinics, along with ethnographic field notes (approximately 100 pages) from 13 in-person clinic visits (54 h), including notes on 50 patient consultations (40 face-to-face, 6 telephone, 4 video). This dataset illustrated the heterogeneity among the ten participating clinics.

Third, field notes (approximately 100 pages), including discussions on 244 clinical cases from the 45 MDT meetings (49 h) that we observed. This dataset revealed further similarities and contrasts among clinics in how patients were managed. In particular, it illustrated how, for the complex patients whose cases were presented at these meetings, teams made sense of, and planned for, each case through multidisciplinary dialogue. This dialogue typically began with one staff member presenting a detailed clinical history along with a narrative of how it had affected the patient’s life and what was at stake for them (e.g. job loss), after which professionals from various backgrounds (nursing, physical therapy, occupational therapy, psychology, dietetics, and different medical specialties) joined in a discussion about what to do.

The ten participating sites are summarized in Table  1 .

In the next two sections, we explore two issues—difficulty defining best practice and the heterogeneous nature of the clinics—that were key to explaining why quality, when pursued in a 10-site collaborative, proved elusive. We then briefly summarize patients’ accounts of their experience in the clinics and give three illustrative examples of the elusiveness of quality improvement using selected topics that were prioritized in our collaborative: outcome measures, investigation of palpitations and management of fatigue. In the final section of the results, we describe how MDT deliberations proved crucial for local quality improvement. Further detail on clinical priority topics will be presented in a separate paper.

“Best practice” in long covid: uncertainty and conflict

The study period (September 2021 to December 2023) corresponded with an exponential increase in published research on long covid. Despite this, the quality improvement collaborative found few unambiguous recommendations for practice. This gap between what the research literature offered and what clinical practice needed was partly ontological (relating what long covid is ). One major bone of contention between patients and clinicians (also evident in discussions with our patient advisory group), for example, was how far (and in whom) clinicians should look for and attempt to treat the various metabolic abnormalities that had been documented in laboratory research studies. The literature on this topic was extensive but conflicting [ 20 , 21 , 22 , 23 , 24 , 79 , 80 , 81 , 82 ]; it was heavy on biological detail but light on clinical application.

Patients were often aware of particular studies that appeared to offer plausible molecular or cellular explanations for symptom clusters along with a drug (often repurposed and off-label) whose mechanism of action appeared to be a good fit with the metabolic chain of causation. In one clinic, for example, we were shown an email exchange between a patient (not medically qualified) and a consultant, in which the patient asked them to reconsider their decision not to prescribe low-dose naltrexone, an opioid receptor antagonist with anti-inflammatory properties. The request included a copy of a peer-reviewed academic paper describing a small, uncontrolled pre-post study (i.e. a weak study design) in which this drug appeared to improve symptoms and functional performance in patients with long covid, as well as a mechanistic argument explaining why the patient felt this drug was a plausible choice in their own case.

This patient’s clinician, in common with most clinicians delivering front-line long covid services, considered that the evidence for such mechanism-based therapies was weak. Clinicians generally felt that this evidence, whilst promising, did not yet support routine measurement of clotting factors, antibodies, immune cells or other biomarkers or the prescription of mechanism-based therapies such as antivirals, anti-inflammatories or anticoagulants. Low-dose naltroxone, for example, is currently being tested in at least one randomized controlled trial (see National Clinical Trials Registry NCT05430152), which had not reported at the time of our observations.

Another challenge to defining best practice was the oft-repeated phrase that long covid is a “diagnosis by exclusion”, but the high prevalence of comorbidities meant that the “pure” long covid patient untainted by other potential explanations for their symptoms was a textbook ideal. In one MDT, for example, we observed a discussion about a patient who had had both swab-positive covid-19 and erythema migrans (a sign of Lyme disease) in the weeks before developing fatigue, yet local diagnostic criteria for each condition required the other to be excluded.

The logic of management in most participating clinics was pragmatic: prompt multidisciplinary assessment and treatment with an emphasis on obtaining a detailed clinical history (including premorbid health status), excluding serious complications (“red flags”), managing specific symptom clusters (for example, physical therapy for breathing pattern disorder), treating comorbidities (for example, anaemia, diabetes or menopause) and supporting whole-person rehabilitation [ 7 , 83 ]. The evidentiary questions raised in MDT discussions (which did not include patients) addressed the practicalities of the rehabilitation model (for example, whether cognitive therapy for neurocognitive complications is as effective when delivered online as it is when delivered in-person) rather than the molecular or cellular mechanisms of disease. For example, the question of whether patients with neurocognitive impairment should be tested for micro-clots or treated with anticoagulants never came up in the MDTs we observed, though we did visit a tertiary referral clinic (the tier 4 clinic in site H), whose lead clinician had a research interest in inflammatory coagulopathies and offered such tests to selected patients.

Because long covid typically produces dozens of symptoms that tend to be uniquely patterned in each patient, the uncertainties on which MDT discussions turned were rarely about general evidence of the kind that might be found in a guideline (e.g. how should fatigue be managed?). Rather they concerned particular case-based clinical decisions (e.g. how should this patient’s fatigue be managed, given the specifics of this case?). An example from our field notes illustrates this:

Physical therapist presents the case of a 39-year-old woman who works as a cleaner on an overnight ferry. Has had long covid for 2 years. Main symptoms are shortness of breath and possible anxiety attacks, especially when at work. She has had a course of physical therapy to teach diaphragmatic breathing but has found that focusing on her breathing makes her more anxious. Patient has to do a lot of bending in her job (e.g. cleaning toilets and under seats), which makes her dizzy, but Active Stand Test was normal. She also has very mild tricuspid incompetence [someone reads out a cardiology report—not hemodynamically significant].

Rehabilitation guidelines (e.g. WHO) recommend phased return to work (e.g. with reduced hours) and frequent breaks. “Tricky!” says someone. The job is intense and busy, and the patient can’t afford not to work. Discussion on whether all her symptoms can be attributed to tension and anxiety. Physical therapist who runs the breathing group says, “No, it’s long covid”, and describes severe initial covid-19 episode and results of serial chest X-rays which showed gradual clearing of ground glass shadows. Team discussion centers on how to negotiate reduced working hours in this particular job, given the overnight ferry shifts. --MDT discussion, Site D

This example raises important considerations about the nature of clinical knowledge in long covid. We return to it in the final section of the “ Results ” and in the “ Discussion ”.

Long covid clinics: a heterogeneous context for quality improvement

Most participating clinics had been established in mid-2020 to follow up patients who had been hospitalized (and perhaps ventilated) for severe acute covid-19. As mass vaccination reduced the severity of acute covid-19 for most people, the patient population in all clinics progressively shifted to include fewer “post-ICU [intensive care unit]” patients (in whom respiratory symptoms almost always dominated), and more people referred by their general practitioners or other secondary care specialties who had not been hospitalized for their acute covid-19 infection, and in whom fatigue, brain fog and palpitations were often the most troubling symptoms. Despite these similarities, the ten clinics had very different histories, geographical and material settings, staffing structures, patient pathways and case mix, as Table  1 illustrates. Below, we give more detail on three example sites.

Site C was established as a generalist “assessment-only” service by a general practitioner with an interest in infectious diseases. It is led jointly by that general practitioner and an occupational therapist, assisted by a wide range of other professionals including speech and language therapy, dietetics, clinical psychology and community-based physical therapy and occupational therapy. It has close links with a chronic fatigue service and a pain clinic that have been running in the locality for over 20 years. The clinic, which is entirely virtual (staff consult either from home or from a small side office in the community trust building), is physically located in a low-rise building on the industrial outskirts of a large town, sharing office space with various community-based health and social care services. Following a 1-h telephone consultation by one of the clinical leads, each patient is discussed at the MDT and then either discharged back to their general practitioner with a detailed management plan or referred on to one of the specialist services. This arrangement evolved to address a particular problem in this locality—that many patients with long covid were being referred by their general practitioner to multiple specialties (e.g. respiratory, neurology, fatigue), leading to a fragmented patient experience, unnecessary specialist assessments and wasteful duplication. The generalist assessment by telephone is oriented to documenting what is often a complex illness narrative (including pre-existing physical and mental comorbidities) and working with the patient to prioritize which symptoms or problems to pursue in which order.

Site E, in a well-regarded inner-city teaching hospital, had been set up in 2020 by a respiratory physician. Its initial ethos and rationale had been “respiratory follow-up”, with strong emphasis on monitoring lung damage via repeated imaging and lung function tests and in ensuring that patients received specialist physical therapy to “re-learn” efficient breathing techniques. Over time, this site has tried to accommodate a more multi-system assessment, with the introduction of a consultant-led infectious disease clinic for patients without a dominant respiratory component, reflecting the shift towards a more fatigue-predominant case mix. At the time of our fieldwork, each patient was seen in turn by a physician, psychologist, occupational therapist and respiratory physical therapist (half an hour each) before all four staff reconvened in a face-to-face MDT meeting to form a plan for each patient. But whilst a wide range of patients with diverse symptoms were discussed at these meetings, there remained a strong focus on respiratory pathology (e.g. tracking improvements in lung function and ensuring that coexisting asthma was optimally controlled).

Site F, one of the first long covid clinics in UK, was set up by a rehabilitation consultant who had been drafted to work on the ICU during the first wave of covid-19 in early 2020. He had a longstanding research interest in whole-patient rehabilitation, especially the assessment and management of chronic fatigue and pain. From the outset, clinic F was more oriented to rehabilitation, including vocational rehabilitation to help patients return to work. There was less emphasis on monitoring lung function or pursuing respiratory comorbidities. At the time of our fieldwork, clinic F offered both a community-based service (“tier 2”) led by an occupational therapist, supported by a respiratory physical therapist and psychologist, and a hospital-based service (“tier 3”) led by the rehabilitation consultant, supported by a wider MDT. Staff in both tiers emphasized that each patient needs a full physical and mental assessment and help to set and work towards achievable goals, whilst staying within safe limits so as to avoid post-exertional symptom exacerbation. Because of the research interest of the lead physician, clinic F adapted well to the growing numbers of patients with fatigue and quickly set up research studies on this cohort [ 84 ].

Details of the other seven sites are shown in Table  1 . Broadly speaking, sites B, E, G and H aligned with the “respiratory follow-up” model and sites F and I aligned with the “rehabilitation” model. Sites A and J had a high-volume, multi-tiered service whose community tier aligned with the “holistic GP assessment” model (site C above) and which also offered a hospital-based, rehabilitation-focused tier. The small service in Scotland (site D) had evolved from an initial respiratory focus to become part of the infectious diseases (ME/CFS) service; Lyme disease (another infectious disease whose sequelae include chronic fatigue) was also prevalent in this region.

The patient experience

Whilst the 10 participating clinics were very diverse in staffing, ethos and patient flows, the 29 patient interviews described remarkably consistent clinic experiences. Almost all identified the biggest problem to be the extended wait of several months before they were seen and the limited awareness (when initially referred) of what long covid clinics could provide. Some talked of how they cried with relief when they finally received an appointment. When the quality improvement collaborative was initially established, waiting times and bottlenecks were patients’ the top priority for quality improvement, and this ranking was shared by clinic staff, who were very aware of how much delays and uncertainties in assessment and treatment compounded patients’ suffering. This issue resolved to a large extent over the study period in all clinics as the referral backlog cleared and the incidence of new cases of long covid fell [ 85 ]; it will be covered in more detail in a separate publication.

Most patients in our sample were satisfied with the care they received when they were finally seen in clinic, especially how they finally felt “heard” after a clinician took a full history. They were relieved to receive affirmation of their experience, a diagnosis of what was wrong and reassurance that they were believed. They were grateful for the input of different members of the multidisciplinary teams and commented on the attentiveness, compassion and skill of allied professionals in particular (“she was wonderful, she got me breathing again”—patient BIR145 talking about a physical therapist). One or two patient participants expressed confusion about who exactly they had seen and what advice they had been given, and some did not realize that a telephone assessment had been an actual clinical consultation. A minority expressed disappointment that an expected investigation had not been ordered (one commented that they had not had any blood tests at all). Several had assumed that the help and advice from the long covid clinic would continue to be offered until they were better and were disappointed that they had been discharged after completing the various courses on offer (since their clinic had been set up as an “assessment only” service).

In the next sections, we give examples of topics raised in the quality improvement collaborative and how they were addressed.

Example quality topic 1: Outcome measures

The first topic considered by the quality improvement collaborative was how (that is, using which measures and metrics) to assess and monitor patients with long covid. In the absence of a validated biomarker, various symptom scores and quality of life scales—both generic and disease-specific—were mooted. Site F had already developed and validated a patient-reported outcome measure (PROM), the C19-YRS (Covid-19 Yorkshire Rehabilitation Scale) and used it for both research and clinical purposes [ 86 ]. It was quickly agreed that, for the purposes of generating comparative research findings across the ten clinics, the C19-YRS should be used at all sites and completed by patients three-monthly. A commercial partner produced an electronic version of this instrument and an app for patient smartphones. The quality improvement collaborative also agreed that patients should be asked to complete the EUROQOL EQ5D, a widely used generic health-related quality of life scale [ 87 ], in order to facilitate comparisons between long covid and other chronic conditions.

In retrospect, the discussions which led to the unopposed adoption of these two measures as a “quality” initiative in clinical care were somewhat aspirational. A review of progress at a subsequent quality improvement meeting revealed considerable variation among clinics, with a wide variety of measures used in different clinics to different degrees. Reasons for this variation were multiple. First, although our patient advisory group were keen that we should gather as much data as possible on the patient experience of this new condition, many clinic patients found the long questionnaires exhausting to complete due to cognitive impairment and fatigue. In addition, whilst patients were keen to answer questions on symptoms that troubled them, many had limited patience to fill out repeated surveys on symptoms that did not trouble them (“it almost felt as if I’ve not got long covid because I didn’t feel like I fit the criteria as they were laying it out”—patient SAL001). Staff assisted patients in completing the measures when needed, but this was time-consuming (up to 45 min per instrument) and burdensome for both staff and patients. In clinics where a high proportion of patients required assistance, staff time was the rate-limiting factor for how many instruments got completed. For some patients, one short instrument was the most that could be asked of them, and the clinician made a judgement on which one would be in their best interests on the day.

The second reason for variation was that the clinical diagnosis and management of particular features, complications and comorbidities of long covid required more nuance than was provided by these relatively generic instruments, and the level of detail sought varied with the specialist interest of the clinic (and the clinician). The modified C19-YRS [ 88 ], for example, contained 19 items, of which one asked about sleep quality. But if a patient had sleep difficulties, many clinicians felt that these needed to be documented in more detail—for example using the 8-item Epworth Sleepiness Scale, originally developed for conditions such as narcolepsy and obstructive sleep apnea [ 89 ]. The “Epworth score” was essential currency for referrals to some but not all specialist sleep services. Similarly, the C19-YRS had three items relating to anxiety, depression and post-traumatic stress disorder, but in clinics where there was a strong focus on mental health (e.g. when there was a resident psychologist), patients were usually invited to complete more specific tools (e.g. the Patient Health Questionnaire 9 [ 90 ], a 9-item questionnaire originally designed to assess severity of depression).

The third reason for variation was custom and practice. Ethnographic visits revealed that paper copies of certain instruments were routinely stacked on clinicians’ desks in outpatient departments and also (in some cases) handed out by administrative staff in waiting areas so that patients could complete them before seeing the clinician. These familiar clinic artefacts tended to be short (one-page) instruments that had a long tradition of use in clinical practice. They were not always fit for purpose. For example, the Nijmegen questionnaire was developed in the 1980s to assess hyperventilation; it was validated against a longer, “gold standard” instrument for that condition [ 91 ]. It subsequently became popular in respiratory clinics to diagnose or exclude breathing pattern disorder (a condition in which the normal physiological pattern of breathing becomes replaced with less efficient, shallower breathing [ 92 ]), so much so that the researchers who developed the instrument published a paper to warn fellow researchers that it had not been validated for this purpose [ 93 ]. Whilst a validated 17-item instrument for breathing pattern disorder (the Self-Evaluation of Breathing Questionnaire [ 94 ]) does exist, it is not in widespread clinical use. Most clinics in LOCOMOTION used Nijmegen either on all patients (e.g. as part of a comprehensive initial assessment, especially if the service had begun as a respiratory follow-up clinic) or when breathing pattern disorder was suspected.

In sum, the use of outcome measures in long covid clinics was a compromise between standardization and contingency. On the one hand, all clinics accepted the need to use “validated” instruments consistently. On the other hand, there were sometimes good reasons why they deviated from agreed practice, including mismatch between the clinic’s priorities as a research site, its priorities as a clinical service, and the particular clinical needs of a patient; the clinic’s—and the clinician’s—specialist focus; and long-held traditions of using particular instruments with which staff and patients were familiar.

Example quality topic 2: Postural orthostatic tachycardia syndrome (POTS)

Palpitations (common in long covid) and postural orthostatic tachycardia syndrome (POTS, a disproportionate acceleration in heart rate on standing, the assumed cause of palpitations in many long covid patients) was the top priority for quality improvement identified by our patient advisory group. Reflecting discussions and evidence (of various kinds) shared in online patient communities, the group were confident that POTS is common in long covid patients and that many cases remain undetected (perhaps misdiagnosed as anxiety). Their request that all long covid patients should be “screened” for POTS prompted a search for, and synthesis of, evidence (which we published in the BMJ [ 95 ]). In sum, that evidence was sparse and contested, but, combined with standard practice in specialist clinics, broadly supported the judicious use of the NASA Lean Test [ 96 ]. This test involves repeated measurements of pulse and blood pressure with the patient first lying and then standing (with shoulders resting against a wall).

The patient advisory group’s request that the NASA Lean Test should be conducted on all patients met with mixed responses from the clinics. In site F, the lead physician had an interest in autonomic dysfunction in chronic fatigue and was keen; he had already published a paper on how to adapt the NASA Lean Test for self-assessment at home [ 97 ]. Several other sites were initially opposed. Staff at site E, for example, offered various arguments:

The test is time-consuming, labor-intensive, and takes up space in the clinic which has an opportunity cost in terms of other potential uses;

The test is unvalidated and potentially misleading (there is a high incidence of both false negative and false positive results);

There is no proven treatment for POTS, so there is no point in testing for it;

It is a specialist test for a specialist condition, so it should be done in a specialist clinic where its benefits and limitations are better understood;

Objective testing does not change clinical management since what we treat is the patient’s symptoms (e.g. by a pragmatic trial of lifestyle measures and medication);

People with symptoms suggestive of dysautonomia have already been “triaged out” of this clinic (that is, identified in the initial telephone consultation and referred directly to neurology or cardiology);

POTS is a manifestation of the systemic nature of long covid; it does not need specific treatment but will improve spontaneously as the patient goes through standard interventions such as active pacing, respiratory physical therapy and sleep hygiene;

Testing everyone, even when asymptomatic, runs counter to the ethos of rehabilitation, which is to “de-medicalize” patients so as to better orient them to their recovery journey.

When clinics were invited to implement the NASA Lean Test on a consecutive sample of patients to resolve a dispute about the incidence of POTS (from “we’ve only seen a handful of people with it since the clinic began” to “POTS is common and often missed”), all but one site agreed to participate. The tertiary POTS centre linked to site H was already running the NASA Lean Test as standard on all patients. Site C, which operated entirely virtually, passed the work to the referring general practitioner by making this test a precondition for seeing the patient; site D, which was largely virtual, sent instructions for patients to self-administer the test at home.

The NASA Lean Test study has been published separately [ 98 ]. In sum, of 277 consecutive patients tested across the eight clinics, 20 (7%) had a positive NASA Lean Test for POTS and a further 28 (10%) a borderline result. Six of 20 patients who met the criteria for POTS on testing had no prior history of orthostatic intolerance. The question of whether this test should be used to “screen” all patients was not answered definitively. But the experience of participating in the study persuaded some sceptics that postural changes in heart rate could be severe in some long covid patients, did not appear to be fully explained by their previously held theories (e.g. “functional”, anxiety, deconditioning), and had likely been missed in some patients. The outcome of this particular quality improvement cycle was thus not a wholescale change in practice (for which the evidence base was weak) but a more subtle increase in clinical awareness, a greater willingness to consider testing for POTS and a greater commitment to contribute to research into this contested condition.

More generally, the POTS audit prompted some clinicians to recognize the value of quality improvement in novel clinical areas. One physician who had initially commented that POTS was not seen in their clinic, for example, reflected:

“ Our clinic population is changing. […] Overall there’s far fewer post-ICU patients with ECMO [extra-corporeal membrane oxygenation] issues and far more long covid from the community, and this is the bit our clinic isn’t doing so well on. We’re doing great on breathing pattern disorder; neuro[logists] are helping us with the brain fogs; our fatigue and occupational advice is ok but some of the dysautonomia symptoms that are more prevalent in the people who were not hospitalized – that’s where we need to improve .” -Respiratory physician, site G (from field visit 6.6.23)

Example quality topic 3: Management of fatigue

Fatigue was the commonest symptom overall and a high priority among both patients and clinicians for quality improvement. It often coexisted with the cluster of neurocognitive symptoms known as brain fog, with both conditions relapsing and remitting in step. Clinicians were keen to systematize fatigue management using a familiar clinical framework oriented around documenting a full clinical history, identifying associated symptoms, excluding or exploring comorbidities and alternative explanations (e.g. poor sleep patterns, depression, menopause, deconditioning), assessing how fatigue affects physical and mental function, implementing a program of physical and cognitive therapy that was sensitive to the patient’s condition and confidence level, and monitoring progress using validated patient-reported outcome measures and symptom diaries.

The underpinning logic of this approach, which broadly reflected World Health Organization guidance [ 99 ], was that fatigue and linked cognitive impairment could be a manifestation of many—perhaps interacting—conditions but that a whole-patient (body and mind) rehabilitation program was the cornerstone of management in most cases. Discussion in the quality improvement collaborative focused on issues such as whether fatigue was so severe that it produced safety concerns (e.g. in a person’s job or with childcare), the pros and cons of particular online courses such as yoga, relaxation and mindfulness (many were viewed positively, though the evidence base was considered weak), and the extent to which respiratory physical therapy had a crossover impact on fatigue (systematic reviews suggested that it may do, but these reviews also cautioned that primary studies were sparse, methodologically flawed, and heterogeneous [ 100 , 101 ]). They also debated the strengths and limitations of different fatigue-specific outcome measures, each of which had been developed and validated in a different condition, with varying emphasis on cognitive fatigue, physical fatigue, effect on daily life, and motivation. These instruments included the Modified Fatigue Impact Scale; Fatigue Severity Scale [ 102 ]; Fatigue Assessment Scale; Functional Assessment Chronic Illness Therapy—Fatigue (FACIT-F) [ 103 ]; Work and Social Adjustment Scale [ 104 ]; Chalder Fatigue Scale [ 105 ]; Visual Analogue Scale—Fatigue [ 106 ]; and the EQ5D [ 87 ]. In one clinic (site F), three of these scales were used in combination for reasons discussed below.

Some clinicians advocated melatonin or nutritional supplements (such as vitamin D or folic acid) for fatigue on the grounds that many patients found them helpful and formal placebo-controlled trials were unlikely ever to be conducted. But neurostimulants used in other fatigue-predominant conditions (e.g. brain injury, stroke), which also lacked clinical trial evidence in long covid, were viewed as inappropriate in most patients because of lack of evidence of clear benefit and hypothetical risk of harm (e.g. adverse drug reactions, polypharmacy).

Whilst the patient advisory group were broadly supportive of a whole-patient rehabilitative approach to fatigue, their primary concern was fatiguability , especially post-exertional symptom exacerbation (PESE, also known as “crashes”). In these, the patient becomes profoundly fatigued some hours or days after physical or mental exertion, and this state can last for days or even weeks [ 107 ]. Patients viewed PESE as a “red flag” symptom which they felt clinicians often missed and sometimes caused. They wanted the quality improvement effort to focus on ensuring that all clinicians were aware of the risks of PESE and acted accordingly. A discussion among patients and clinicians at a quality improvement collaborative meeting raised a new research hypothesis—that reducing the number of repeated episodes of PESE may improve the natural history of long covid.

These tensions around fatigue management played out differently in different clinics. In site C (the GP-led virtual clinic run from a community hub), fatigue was viewed as one manifestation of a whole-patient condition. The lead general practitioner used the metaphor of untangling a skein of wool: “you have to find the end and then gently pull it”. The underlying problem in a fatigued patient, for example, might be an undiagnosed physical condition such as anaemia, disturbed sleep, or inadequate pacing. These required (respectively) the chronic fatigue service (comprising an occupational therapist and specialist psychologist and oriented mainly to teaching the techniques of goal-setting and pacing), a “tiredness” work-up (e.g. to exclude anaemia or menopause), investigation of poor sleep (which, not uncommonly, was due to obstructive sleep apnea), and exploration of mental health issues.

In site G (a hospital clinic which had evolved from a respiratory service), patients with fatigue went through a fatigue management program led by the occupational therapist with emphasis on pacing, energy conservation, avoidance of PESE and sleep hygiene. Those without ongoing respiratory symptoms were often discharged back to their general practitioner once they had completed this; there was no consultant follow-up of unresolved fatigue.

In site F (a rehabilitation clinic which had a longstanding interest in chronic fatigue even before the pandemic), active interdisciplinary management of fatigue was commenced at or near the patient’s first visit, on the grounds that the earlier this began, the more successful it would be. In this clinic, patients were offered a more intensive package: a similar occupational therapy-led fatigue course as those in site G, plus input from a dietician to advise on regular balanced meals and caffeine avoidance and a group-based facilitated peer support program which centred on fatigue management. The dietician spoke enthusiastically about how improving diet in longstanding long covid patients often improved fatigue (e.g. because they had often lost muscle mass and tended to snack on convenience food rather than make meals from scratch), though she agreed there was no evidence base from trials to support this approach.

Pursuing local quality improvement through MDTs

Whilst some long covid patients had “textbook” symptoms and clinical findings, many cases were unique and some were fiendishly complex. One clinician commented that, somewhat paradoxically, “easy cases” were often the post-ICU follow-ups who had resolving chest complications; they tended to do well with a course of respiratory physical therapy and a return-to-work program. Such cases were rarely brought to MDT meetings. “Difficult cases” were patients who had not been hospitalized for their acute illness but presented with a months- or years-long history of multiple symptoms with fatigue typically predominant. Each one was different, as the following example (some details of which have been fictionalized to protect anonymity) illustrates.

The MDT is discussing Mrs Fermah, a 65-year-old homemaker who had covid-19 a year ago. She has had multiple symptoms since, including fluctuating fatigue, brain fog, breathlessness, retrosternal chest pain of burning character, dry cough, croaky voice, intermittent rashes (sometimes on eating), lips going blue, ankle swelling, orthopnoea, dizziness with the room spinning which can be triggered by stress, low back pain, aches and pains in the arms and legs and pins and needles in the fingertips, loss of taste and smell, palpitations and dizziness (unclear if postural, but clear association with nausea), headaches on waking, and dry mouth. She is somewhat overweight (body mass index 29) and admits to low mood. Functionally, she is mostly confined to the house and can no longer manage the stairs so has begun to sleep downstairs. She has stumbled once or twice but not fallen. Her social life has ceased and she rarely has the energy to see her grandchildren. Her 70-year-old husband is retired and generally supportive, though he spends most evenings at his club. Comorbidities include glaucoma which is well controlled and overseen by an ophthalmologist, mild club foot (congenital) and stage 1 breast cancer 20 years ago. Various tests, including a chest X-ray, resting and exercise oximetry and a blood panel, were normal except for borderline vitamin D level. Her breathing questionnaire score suggests she does not have breathing pattern disorder. ECG showed first-degree atrioventricular block and left axis deviation. No clinician has witnessed the blue lips. Her current treatment is online group respiratory physical therapy; a home visit is being arranged to assess her climbing stairs. She has declined a psychologist assessment. The consultant asks the nurse who assessed her: “Did you get a feel if this is a POTS-type dizziness or an ENT-type?” She sighs. “Honestly it was hard to tell, bless her.”—Site A MDT

This patient’s debilitating symptoms and functional impairments could all be due to long covid, yet “evidence-based” guidance for how to manage her complex suffering does not exist and likely never will exist. The question of which (if any) additional blood or imaging tests to do, in what order of priority, and what interventions to offer the patient will not be definitively answered by consulting clinical trials involving hundreds of patients, since (even if these existed) the decision involves weighing this patient’s history and the multiple factors and uncertainties that are relevant in her case. The knowledge that will help the MDT provide quality care to Mrs Fermah is case-based knowledge—accumulated clinical experience and wisdom from managing and deliberating on multiple similar cases. We consider case-based knowledge further in the “ Discussion ”.

Summary of key findings

This study has shown that a quality improvement collaborative of UK long covid clinics made some progress towards standardizing assessment and management in some topics, but some variation remained. This could be explained in part by the fact that different clinics had different histories and path dependencies, occupied a different place in the local healthcare ecosystem, served different populations, were differently staffed, and had different clinical interests. Our patient advisory group and clinicians in the quality improvement collaborative broadly prioritized the same topics for improvement but interpreted them somewhat differently. “Quality” long covid care had multiple dimensions, relating to (among other things) service set-up and accessibility, clinical provision appropriate to the patient’s need (including options for referral to other services locally), the human qualities of clinical and support staff, how knowledge was distributed across (and accessible within) the system, and the accumulated collective wisdom of local MDTs in dealing with complex cases (including multiple kinds of specialist expertise as well as relational knowledge of what was at stake for the patient). Whilst both staff and patients were keen to contribute to the quality improvement effort, the burden of measurement was evident: multiple outcome measures, used repeatedly, were resource-intensive for staff and exhausting for patients.

Strengths and limitations of this study

To our knowledge, we are the first to report both a quality improvement collaborative and an in-depth qualitative study of clinical work in long covid. Key strengths of this work include the diverse sampling frame (with sites from three UK jurisdictions and serving widely differing geographies and demographics); the use of documents, interviews and reflexive interpretive ethnography to produce meaningful accounts of how clinics emerged and how they were currently organized; the use of philosophical concepts to analyse data on how MDTs produced quality care on a patient-by-patient basis; and the close involvement of patient co-researchers and coauthors during the research and writing up.

Limitations of the study include its exclusive UK focus (the external validity of findings to other healthcare systems is unknown); the self-selecting nature of participants in a quality improvement collaborative (our patient advisory group suggested that the MDTs observed in this study may have represented the higher end of a quality spectrum, hence would be more likely than other MDTs to adhere to guidelines); and the particular perspective brought by the researchers (two GPs, a physical therapist and one non-clinical person) in ethnographic observations. Hospital specialists or organizational scholars, for example, may have noticed different things or framed what they observed differently.

Explaining variation in long covid care

Sutherland and Levesque’s framework mentioned in the “ Background ” section does not explain much of the variation found in our study [ 70 ]. In terms of capacity, at the time of this study most participating clinics benefited from ring-fenced resources. In terms of evidence, guidelines existed and were not greatly contested, but as illustrated by the case of Mrs Fermah above, many patients were exceptions to the guideline because of complex symptomatology and relevant comorbidities. In terms of agency, clinicians in most clinics were passionately engaged with long covid (they were pioneers who had set up their local clinic and successfully bid for national ring-fenced resources) and were generally keen to support patient choice (though not if the patient requested tests which were unavailable or deemed not indicated).

Astma et al.’s list of factors that may explain variation in practice (see “ Background ”) includes several that may be relevant to long covid, especially that the definition of appropriate care in this condition remains somewhat contested. But lack of opportunity to discuss cases was not a problem in the clinics in our sample. On the contrary, MDT meetings in each locality gave clinicians multiple opportunities to discuss cases with colleagues and reflect collectively on whether and how to apply particular guidelines.

The key problem was not that clinicians disputed the guidelines for managing long covid or were unaware of them; it was that the guidelines were not self-interpreting . Rather, MDTs had to deliberate on the balance of benefits and harms in different aspects of individual cases. In patients whose symptoms suggested a possible diagnosis of POTS (or who suspected themselves of having POTS), for example, these deliberations were sometimes lengthy and nuanced. Should a test result that is not technically in the abnormal range but close to it be treated as diagnostic, given that symptoms point to this diagnosis? If not, should the patient be told that the test excludes POTS or that it is equivocal? If a cardiology opinion has stated firmly that the patient does not have POTS but the cardiologist is not known for their interest in this condition, should a second specialist opinion be sought? If the gold standard “tilt test” [ 108 ] for POTS (usually available only in tertiary centres) is not available locally, does this patient merit a costly out-of-locality referral? Should the patient’s request for a trial of off-label medication, reflecting discussions in an online support group, be honoured? These are the kinds of questions on which MDTs deliberated at length.

The fact that many cases required extensive deliberation does not necessarily justify variation in practice among clinics. But taking into account the clinics’ very different histories, set-up, and local referral pathways, the variation begins to make sense. A patient who is being assessed in a clinic that functions as a specialist chronic fatigue centre and attracts referrals which reflect this interest (e.g. site F in our sample) will receive different management advice from one that functions as a telephone-only generalist assessment centre and refers on to other specialties (site C in our sample). The wide variation in case mix, coupled with the fact that a different proportion of these cases were highly complex in each clinic (and in different ways), suggests that variation in practice may reflect appropriate rather than inappropriate care.

Our patient advisory group affirmed that many of the findings reported here resonated with their own experience, but they raised several concerns. These included questions about patient groups who may have been missed in our sample because they were rarely discussed in MDTs. The decision to take a case to MDT discussion is taken largely by a clinician, and there was evidence from online support groups that some patients’ requests for their case to be taken to an MDT had been declined (though not, to our knowledge, in the clinics participating in the LOCOMOTION study).

We began this study by asking “what is quality in long covid care?”. We initially assumed that this question referred to a generalizable evidence base, which we felt we could identify, and we believed that we could then determine whether long covid clinics were following the evidence base through conventional audits of structure, process, and outcome. In retrospect, these assumptions were somewhat naïve. On the basis of our findings, we suggest that a better (and more individualized) research question might be “to what extent does each patient with long covid receive evidence-based care appropriate to their needs?”. This question would require individual case review on a sample of cases, tracking each patient longitudinally including cross-referrals, and also interviewing the patient.

Nomothetic versus idiographic knowledge

In a series of lectures first delivered in the 1950s and recently republished [ 109 ], psychiatrist Dr Maurice O’Connor Drury drew on the later philosophy of his friend and mentor Ludwig Wittgenstein to challenge what he felt was a concerning trend: that the nomothetic (generalizable, abstract) knowledge from randomized controlled trials (RCTs) was coming to over-ride the idiographic (personal, situated) knowledge about particular patients. Based on Wittgenstein’s writings on the importance of the particular, Drury predicted—presciently—that if implemented uncritically, RCTs would result in worse, not better, care for patients, since it would go hand-in-hand with a downgrading of experience, intuition, subjective judgement, personal reflection, and collective deliberation.

Much conventional quality improvement methodology is built on an assumption that nomothetic knowledge (for example, findings from RCTs and systematic reviews) is a higher form of knowing than idiographic knowledge. But idiographic, case-based reasoning—despite its position at the very bottom of evidence-based medicine’s hierarchy of evidence [ 110 ]—is a legitimate and important element of medical practice. Bioethicist Kathryn Montgomery, drawing on Aristotle’s notion of praxis , considers clinical practice to be an example of case-based reasoning [ 111 ]. Medicine is governed not by hard and fast laws but by competing maxims or rules of thumb ; the essence of judgement is deciding which (if any) rule should be applied in a particular circumstance. Clinical judgement incorporates science (especially the results of well-conducted research) and makes use of available tools and technologies (including guidelines and decision-support algorithms that incorporate research findings). But rather than being determined solely by these elements, clinical judgement is guided both by the scientific evidence and by the practical and ethical question “what is it best to do, for this individual, given these circumstances?”.

In this study, we observed clinical management of, and MDT deliberations on, hundreds of clinical cases. In the more straightforward ones (for example, recovering pneumonitis), guideline-driven care was not difficult to implement and such cases were rarely brought to the MDT. But cases like Mrs Fermah (see last section of “ Results ”) required much discussion on which aspects of which guideline were in the patient’s best interests to bring into play at any particular stage in their illness journey.

Conclusions

One systematic review on quality improvement collaboratives concluded that “ [those] reporting success generally addressed relatively straightforward aspects of care, had a strong evidence base and noted a clear evidence-practice gap in an accepted clinical pathway or guideline” (page 226) [ 60 ]. The findings from this study suggest that to the extent that such collaboratives address clinical cases that are not straightforward, conventional quality improvement methods may be less useful and even counterproductive.

The question “what is quality in long covid care?” is partly a philosophical one. Our findings support an approach that recognizes and values idiographic knowledge —including establishing and protecting a safe and supportive space for deliberation on individual cases to occur and to value and draw upon the collective learning that occurs in these spaces. It is through such deliberation that evidence-based guidelines can be appropriately interpreted and applied to the unique needs and circumstances of individual patients. We suggest that Drury’s warning about the limitations of nomothetic knowledge should prompt a reassessment of policies that rely too heavily on such knowledge, resulting in one-size-fits-all protocols. We also cautiously hypothesize that the need to centre the quality improvement effort on idiographic rather than nomothetic knowledge is unlikely to be unique to long covid. Indeed, such an approach may be particularly important in any condition that is complex, unpredictable, variable in presentation and clinical course, and associated with comorbidities.

Availability of data and materials

Selected qualitative data (ensuring no identifiable information) will be made available to formal research teams on reasonable request to Professor Greenhalgh at the University of Oxford, on condition that they have research ethics approval and relevant expertise. The quantitative data on NASA Lean Test have been published in full in a separate paper [ 98 ].

Abbreviations

Chronic fatigue syndrome

Intensive care unit

Jenny Ceolta-Smith

Julie Darbyshire

LOng COvid Multidisciplinary consortium Optimising Treatments and services across the NHS

Multidisciplinary team

Myalgic encephalomyelitis

Middle East Respiratory Syndrome

National Aeronautics and Space Association

Occupational therapy/ist

Post-exertional symptom exacerbation

Postural orthostatic tachycardia syndrome

Speech and language therapy

Severe Acute Respiratory Syndrome

Trisha Greenhalgh

United Kingdom

United States

World Health Organization

Perego E, Callard F, Stras L, Melville-JÛhannesson B, Pope R, Alwan N. Why the Patient-Made Term “Long Covid” is needed. Wellcome Open Res. 2020;5:224.

Article   Google Scholar  

Greenhalgh T, Sivan M, Delaney B, Evans R, Milne R: Long covid—an update for primary care. bmj 2022;378:e072117.

Centers for Disease Control and Prevention (US): Long COVID or Post-COVID Conditions (updated 16th December 2022). Atlanta: CDC. Accessed 2nd June 2023 at https://www.cdc.gov/coronavirus/2019-ncov/long-term-effects/index.html ; 2022.

National Institute for Health and Care Excellence (NICE) Scottish Intercollegiate Guidelines Network (SIGN) and Royal College of General Practitioners (RCGP): COVID-19 rapid guideline: managing the long-term effects of COVID-19, vol. Accessed 30th January 2022 at https://www.nice.org.uk/guidance/ng188/resources/covid19-rapid-guideline-managing-the-longterm-effects-of-covid19-pdf-51035515742 . London: NICE; 2022.

Organization WH: Post Covid-19 Condition (updated 7th December 2022), vol. Accessed 2nd June 2023 at https://www.who.int/europe/news-room/fact-sheets/item/post-covid-19-condition#:~:text=It%20is%20defined%20as%20the,months%20with%20no%20other%20explanation . Geneva: WHO; 2022.

Office for National Statistics: Prevalence of ongoing symptoms following coronavirus (COVID-19) infection in the UK: 31st March 2023. London: ONS. Accessed 30th May 2023 at https://www.ons.gov.uk/peoplepopulationandcommunity/healthandsocialcare/conditionsanddiseases/datasets/alldatarelatingtoprevalenceofongoingsymptomsfollowingcoronaviruscovid19infectionintheuk ; 2023.

Crook H, Raza S, Nowell J, Young M, Edison P: Long covid—mechanisms, risk factors, and management. bmj 2021;374.

Sudre CH, Murray B, Varsavsky T, Graham MS, Penfold RS, Bowyer RC, Pujol JC, Klaser K, Antonelli M, Canas LS. Attributes and predictors of long COVID. Nat Med. 2021;27(4):626–31.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Reese JT, Blau H, Casiraghi E, Bergquist T, Loomba JJ, Callahan TJ, Laraway B, Antonescu C, Coleman B, Gargano M: Generalisable long COVID subtypes: findings from the NIH N3C and RECOVER programmes. EBioMedicine 2023;87.

Thaweethai T, Jolley SE, Karlson EW, Levitan EB, Levy B, McComsey GA, McCorkell L, Nadkarni GN, Parthasarathy S, Singh U. Development of a definition of postacute sequelae of SARS-CoV-2 infection. JAMA. 2023;329(22):1934–46.

Brown DA, O’Brien KK. Conceptualising Long COVID as an episodic health condition. BMJ Glob Health. 2021;6(9): e007004.

Article   PubMed   Google Scholar  

Tate WP, Walker MO, Peppercorn K, Blair AL, Edgar CD. Towards a Better Understanding of the Complexities of Myalgic Encephalomyelitis/Chronic Fatigue Syndrome and Long COVID. Int J Mol Sci. 2023;24(6):5124.

Ahmed H, Patel K, Greenwood DC, Halpin S, Lewthwaite P, Salawu A, Eyre L, Breen A, Connor RO, Jones A. Long-term clinical outcomes in survivors of severe acute respiratory syndrome (SARS) and Middle East respiratory syndrome coronavirus (MERS) outbreaks after hospitalisation or ICU admission: a systematic review and meta-analysis. J Rehabil Med. 2020;52(5):1–11.

Google Scholar  

World Health Organisation: Clinical management of severe acute respiratory infection (SARI) when COVID-19 disease is suspected: Interim guidance (13th March 2020). Geneva: WHO. Accessed 3rd January 2023 at https://t.co/JpNdP8LcV8?amp=1 ; 2020.

Rushforth A, Ladds E, Wieringa S, Taylor S, Husain L, Greenhalgh T: Long Covid – the illness narratives. Under review for Sociology of Health and Illness 2021.

Russell D, Spence NJ. Chase J-AD, Schwartz T, Tumminello CM, Bouldin E: Support amid uncertainty: Long COVID illness experiences and the role of online communities. SSM-Qual Res Health. 2022;2: 100177.

Article   PubMed   PubMed Central   Google Scholar  

Ziauddeen N, Gurdasani D, O’Hara ME, Hastie C, Roderick P, Yao G, Alwan NA. Characteristics and impact of Long Covid: Findings from an online survey. PLoS ONE. 2022;17(3): e0264331.

Evans RA, McAuley H, Harrison EM, Shikotra A, Singapuri A, Sereno M, Elneima O, Docherty AB, Lone NI, Leavy OC. Physical, cognitive, and mental health impacts of COVID-19 after hospitalisation (PHOSP-COVID): a UK multicentre, prospective cohort study. Lancet Respir Med. 2021;9(11):1275–87.

Sykes DL, Holdsworth L, Jawad N, Gunasekera P, Morice AH, Crooks MG. Post-COVID-19 symptom burden: what is long-COVID and how should we manage it? Lung. 2021;199(2):113–9.

Altmann DM, Whettlock EM, Liu S, Arachchillage DJ, Boyton RJ: The immunology of long COVID. Nat Rev Immunol 2023:1–17.

Klein J, Wood J, Jaycox J, Dhodapkar RM, Lu P, Gehlhausen JR, Tabachnikova A, Greene K, Tabacof L, Malik AA et al : Distinguishing features of Long COVID identified through immune profiling. Nature 2023.

Chen B, Julg B, Mohandas S, Bradfute SB. Viral persistence, reactivation, and mechanisms of long COVID. Elife. 2023;12: e86015.

Wang C, Ramasamy A, Verduzco-Gutierrez M, Brode WM, Melamed E. Acute and post-acute sequelae of SARS-CoV-2 infection: a review of risk factors and social determinants. Virol J. 2023;20(1):124.

Cervia-Hasler C, Brüningk SC, Hoch T, Fan B, Muzio G, Thompson RC, Ceglarek L, Meledin R, Westermann P, Emmenegger M et al Persistent complement dysregulation with signs of thromboinflammation in active Long Covid Science 2024;383(6680):eadg7942.

Sivan M, Greenhalgh T, Darbyshire JL, Mir G, O’Connor RJ, Dawes H, Greenwood D, O’Connor D, Horton M, Petrou S. LOng COvid Multidisciplinary consortium Optimising Treatments and servIces acrOss the NHS (LOCOMOTION): protocol for a mixed-methods study in the UK. BMJ Open. 2022;12(5): e063505.

Rushforth A, Ladds E, Wieringa S, Taylor S, Husain L, Greenhalgh T. Long covid–the illness narratives. Soc Sci Med. 2021;286: 114326.

National Institute for Health and Care Excellence: COVID-19 rapid guideline: managing the long-term effects of COVID-19, vol. Accessed 4th October 2023 at https://www.nice.org.uk/guidance/ng188/resources/covid19-rapid-guideline-managing-the-longterm-effects-of-covid19-pdf-51035515742 . London: NICE 2020.

NHS England: Long COVID: the NHS plan for 2021/22. London: NHS England. Accessed 2nd August 2022 at https://www.england.nhs.uk/coronavirus/documents/long-covid-the-nhs-plan-for-2021-22/ ; 2021.

NHS England: NHS to offer ‘long covid’ sufferers help at specialist centres. London: NHS England. Accessed 10th October 2020 at https://www.england.nhs.uk/2020/10/nhs-to-offer-long-covid-help/ ; 2020 (7th October).

NHS England: The NHS plan for improving long COVID services, vol. Acessed 4th February 2024 at https://www.england.nhs.uk/publication/the-nhs-plan-for-improving-long-covid-services/ .London: Gov.uk; 2022.

NHS England: Commissioning guidance for post-COVID services for adults, children and young people, vol. Accessed 6th February 2024 at https://www.england.nhs.uk/long-read/commissioning-guidance-for-post-covid-services-for-adults-children-and-young-people/ . London: gov.uk; 2023.

National Institute for Health Research: Researching Long Covid: Adressing a new global health challenge, vol. Accessed 9.8.23 at https://evidence.nihr.ac.uk/collection/researching-long-covid-addressing-a-new-global-health-challenge/ . London: NIHR; 2022.

Subbaraman N. NIH will invest $1 billion to study long COVID. Nature. 2021;591(7850):356–356.

Article   CAS   PubMed   Google Scholar  

Donabedian A. The definition of quality and approaches to its assessment and monitoring. Ann Arbor: Michigan; 1980.

Laffel G, Blumenthal D. The case for using industrial quality management science in health care organizations. JAMA. 1989;262(20):2869–73.

Maxwell RJ. Quality assessment in health. BMJ. 1984;288(6428):1470.

Berwick DM, Godfrey BA, Roessner J. Curing health care: New strategies for quality improvement. The Journal for Healthcare Quality (JHQ). 1991;13(5):65–6.

Deming WE. Out of the Crisis. Cambridge, MA: MIT Press; 1986.

Argyris C: Increasing leadership effectiveness: New York: J. Wiley; 1976.

Juran JM: A history of managing for quality: The evolution, trends, and future directions of managing for quality: Asq Press; 1995.

Institute of Medicine (US): Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academy Press; 2001.

McNab D, McKay J, Shorrock S, Luty S, Bowie P. Development and application of ‘systems thinking’ principles for quality improvement. BMJ Open Qual. 2020;9(1): e000714.

Sampath B, Rakover J, Baldoza K, Mate K, Lenoci-Edwards J, Barker P. ​Whole-System Quality: A Unified Approach to Building Responsive, Resilient Health Care Systems. Boston: Institute for Healthcare Immprovement; 2021.

Batalden PB, Davidoff F: What is “quality improvement” and how can it transform healthcare? In . , vol. 16: BMJ Publishing Group Ltd; 2007: 2–3.

Baker G. Collaborating for improvement: the Institute for Healthcare Improvement’s breakthrough series. New Med. 1997;1:5–8.

Plsek PE. Collaborating across organizational boundaries to improve the quality of care. Am J Infect Control. 1997;25(2):85–95.

Ayers LR, Beyea SC, Godfrey MM, Harper DC, Nelson EC, Batalden PB. Quality improvement learning collaboratives. Qual Manage Healthcare. 2005;14(4):234–47.

Brandrud AS, Schreiner A, Hjortdahl P, Helljesen GS, Nyen B, Nelson EC. Three success factors for continual improvement in healthcare: an analysis of the reports of improvement team members. BMJ Qual Saf. 2011;20(3):251–9.

Dückers ML, Spreeuwenberg P, Wagner C, Groenewegen PP. Exploring the black box of quality improvement collaboratives: modelling relations between conditions, applied changes and outcomes. Implement Sci. 2009;4(1):1–12.

Nadeem E, Olin SS, Hill LC, Hoagwood KE, Horwitz SM. Understanding the components of quality improvement collaboratives: a systematic literature review. Milbank Q. 2013;91(2):354–94.

Shortell SM, Marsteller JA, Lin M, Pearson ML, Wu S-Y, Mendel P, Cretin S, Rosen M: The role of perceived team effectiveness in improving chronic illness care. Medical Care 2004:1040–1048.

Wilson T, Berwick DM, Cleary PD. What do collaborative improvement projects do? Experience from seven countries. Joint Commission J Qual Safety. 2004;30:25–33.

Schouten LM, Hulscher ME, van Everdingen JJ, Huijsman R, Grol RP. Evidence for the impact of quality improvement collaboratives: systematic review. BMJ. 2008;336(7659):1491–4.

Hulscher ME, Schouten LM, Grol RP, Buchan H. Determinants of success of quality improvement collaboratives: what does the literature show? BMJ Qual Saf. 2013;22(1):19–31.

Dixon-Woods M, Bosk CL, Aveling EL, Goeschel CA, Pronovost PJ. Explaining Michigan: developing an ex post theory of a quality improvement program. Milbank Q. 2011;89(2):167–205.

Bate P, Mendel P, Robert G: Organizing for quality: the improvement journeys of leading hospitals in Europe and the United States: CRC Press; 2007.

Andersson-Gäre B, Neuhauser D. The health care quality journey of Jönköping County Council. Sweden Qual Manag Health Care. 2007;16(1):2–9.

Törnblom O, Stålne K, Kjellström S. Analyzing roles and leadership in organizations from cognitive complexity and meaning-making perspectives. Behav Dev. 2018;23(1):63.

Greenhalgh T, Russell J. Why Do Evaluations of eHealth Programs Fail? An Alternative Set of Guiding Principles. PLoS Med. 2010;7(11): e1000360.

Wells S, Tamir O, Gray J, Naidoo D, Bekhit M, Goldmann D. Are quality improvement collaboratives effective? A systematic review. BMJ Qual Saf. 2018;27(3):226–40.

Landon BE, Wilson IB, McInnes K, Landrum MB, Hirschhorn L, Marsden PV, Gustafson D, Cleary PD. Effects of a quality improvement collaborative on the outcome of care of patients with HIV infection: the EQHIV study. Ann Intern Med. 2004;140(11):887–96.

Mittman BS. Creating the evidence base for quality improvement collaboratives. Ann Intern Med. 2004;140(11):897–901.

Wennberg JE. Unwarranted variations in healthcare delivery: implications for academic medical centres. BMJ. 2002;325(7370):961–4.

Bungay H. Cancer and health policy: the postcode lottery of care. Soc Policy Admin. 2005;39(1):35–48.

Wennberg JE, Cooper MM: The Quality of Medical Care in the United States: A Report on the Medicare Program: The Dartmouth Atlas of Health Care 1999: The Center for the Evaluative Clinical Sciences [Internet]. 1999.

DaSilva P, Gray JM. English lessons: can publishing an atlas of variation stimulate the discussion on appropriateness of care? Med J Aust. 2016;205(S10):S5–7.

Gray WK, Day J, Briggs TW, Harrison S. Identifying unwarranted variation in clinical practice between healthcare providers in England: Analysis of administrative data over time for the Getting It Right First Time programme. J Eval Clin Pract. 2021;27(4):743–50.

Wabe N, Thomas J, Scowen C, Eigenstetter A, Lindeman R, Georgiou A. The NSW Pathology Atlas of Variation: Part I—Identifying Emergency Departments With Outlying Laboratory Test-Ordering Practices. Ann Emerg Med. 2021;78(1):150–62.

Jamal A, Babazono A, Li Y, Fujita T, Yoshida S, Kim SA. Elucidating variations in outcomes among older end-stage renal disease patients on hemodialysis in Fukuoka Prefecture, Japan. PLoS ONE. 2021;16(5): e0252196.

Sutherland K, Levesque JF. Unwarranted clinical variation in health care: definitions and proposal of an analytic framework. J Eval Clin Pract. 2020;26(3):687–96.

Tanenbaum SJ. Reducing variation in health care: The rhetorical politics of a policy idea. J Health Polit Policy Law. 2013;38(1):5–26.

Atsma F, Elwyn G, Westert G. Understanding unwarranted variation in clinical practice: a focus on network effects, reflective medicine and learning health systems. Int J Qual Health Care. 2020;32(4):271–4.

Horbar JD, Rogowski J, Plsek PE, Delmore P, Edwards WH, Hocker J, Kantak AD, Lewallen P, Lewis W, Lewit E. Collaborative quality improvement for neonatal intensive care. Pediatrics. 2001;107(1):14–22.

Van Maanen J: Tales of the field: On writing ethnography: University of Chicago Press; 2011.

Golden-Biddle K, Locke K. Appealing work: An investigation of how ethnographic texts convince. Organ Sci. 1993;4(4):595–616.

Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77–101.

Glaser BG. The constant comparative method of qualitative analysis. Soc Probl. 1965;12:436–45.

Willis R. The use of composite narratives to present interview findings. Qual Res. 2019;19(4):471–80.

Vojdani A, Vojdani E, Saidara E, Maes M. Persistent SARS-CoV-2 Infection, EBV, HHV-6 and other factors may contribute to inflammation and autoimmunity in long COVID. Viruses. 2023;15(2):400.

Choutka J, Jansari V, Hornig M, Iwasaki A. Unexplained post-acute infection syndromes. Nat Med. 2022;28(5):911–23.

Connors JM, Ariëns RAS. Uncertainties about the roles of anticoagulation and microclots in postacute sequelae of severe acute respiratory syndrome coronavirus 2 infection. J Thromb Haemost. 2023;21(10):2697–701.

Patel MA, Knauer MJ, Nicholson M, Daley M, Van Nynatten LR, Martin C, Patterson EK, Cepinskas G, Seney SL, Dobretzberger V. Elevated vascular transformation blood biomarkers in Long-COVID indicate angiogenesis as a key pathophysiological mechanism. Mol Med. 2022;28(1):122.

Greenhalgh T, Sivan M, Delaney B, Evans R, Milne R: Long covid—an update for primary care. bmj 2022, 378.

Parkin A, Davison J, Tarrant R, Ross D, Halpin S, Simms A, Salman R, Sivan M. A multidisciplinary NHS COVID-19 service to manage post-COVID-19 syndrome in the community. J Prim Care Commun Health. 2021;12:21501327211010990.

NHS England: COVID-19 Post-Covid Assessment Service, vol. Accessed 5th March 2024 at https://www.england.nhs.uk/statistics/statistical-work-areas/covid-19-post-covid-assessment-service/ . London: NHS England; 2024.

Sivan M, Halpin S, Gee J, Makower S, Parkin A, Ross D, Horton M, O'Connor R: The self-report version and digital format of the COVID-19 Yorkshire Rehabilitation Scale (C19-YRS) for Long Covid or Post-COVID syndrome assessment and monitoring. Adv Clin Neurosci Rehabil 2021;20(3).

The EuroQol Group. EuroQol-a new facility for the measurement of health-related quality of life. Health Policy. 1990;16(3):199–208.

Sivan M, Preston NJ, Parkin A, Makower S, Gee J, Ross D, Tarrant R, Davison J, Halpin S, O’Connor RJ, et al. The modified COVID-19 Yorkshire Rehabilitation Scale (C19-YRSm) patient-reported outcome measure for Long Covid or Post-COVID syndrome. J Med Virol. 2022;94(9):4253–64.

Johns MW. A new method for measuring daytime sleepiness: the Epworth sleepiness scale. Sleep. 1991;14(6):540–5.

Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. 2001;16(9):606–13.

Van Dixhoorn J, Duivenvoorden H. Efficacy of Nijmegen Questionnaire in recognition of the hyperventilation syndrome. J Psychosom Res. 1985;29(2):199–206.

Evans R, Pick A, Lardner R, Masey V, Smith N, Greenhalgh T: Breathing difficulties after covid-19: a guide for primary care. BMJ 2023;381.

Van Dixhoorn J, Folgering H: The Nijmegen Questionnaire and dysfunctional breathing. In . , vol. 1: Eur Respiratory Soc; 2015.

Courtney R, Greenwood KM. Preliminary investigation of a measure of dysfunctional breathing symptoms: The Self Evaluation of Breathing Questionnaire (SEBQ). Int J Osteopathic Med. 2009;12(4):121–7.

Espinosa-Gonzalez A, Master H, Gall N, Halpin S, Rogers N, Greenhalgh T. Orthostatic tachycardia after covid-19. BMJ (Clinical Research ed). 2023;380:e073488–e073488.

PubMed   Google Scholar  

Bungo M, Charles J, Johnson P Jr. Cardiovascular deconditioning during space flight and the use of saline as a countermeasure to orthostatic intolerance. Aviat Space Environ Med. 1985;56(10):985–90.

CAS   PubMed   Google Scholar  

Sivan M, Corrado J, Mathias C. The Adapted Autonomic Profile (Aap) Home-Based Test for the Evaluation of Neuro-Cardiovascular Autonomic Dysfunction. Adv Clin Neurosci Rehabil. 2022;3:10–13. https://doi.org/10.47795/QKBU46715 .

Lee C, Greenwood DC, Master H, Balasundaram K, Williams P, Scott JT, Wood C, Cooper R, Darbyshire JL, Gonzalez AE. Prevalence of orthostatic intolerance in long covid clinic patients and healthy volunteers: A multicenter study. J Med Virol. 2024;96(3): e29486.

World Health Organization: Clinical management of covid-19 - living guideline. Geneva: WHO. Accessed 4th October 2023 at https://www.who.int/publications/i/item/WHO-2019-nCoV-clinical-2021-2 ; 2023.

Ahmed I, Mustafaoglu R, Yeldan I, Yasaci Z, Erhan B: Effect of pulmonary rehabilitation approaches on dyspnea, exercise capacity, fatigue, lung functions and quality of life in patients with COVID-19: A Systematic Review and Meta-Analysis. Arch Phys Med Rehabil 2022.

Dillen H, Bekkering G, Gijsbers S, Vande Weygaerde Y, Van Herck M, Haesevoets S, Bos DAG, Li A, Janssens W, Gosselink R, et al. Clinical effectiveness of rehabilitation in ambulatory care for patients with persisting symptoms after COVID-19: a systematic review. BMC Infect Dis. 2023;23(1):419.

Learmonth Y, Dlugonski D, Pilutti L, Sandroff B, Klaren R, Motl R. Psychometric properties of the fatigue severity scale and the modified fatigue impact scale. J Neurol Sci. 2013;331(1–2):102–7.

Webster K, Cella D, Yost K. The Functional Assessment of Chronic Illness T herapy (FACIT) Measurement System: properties, applications, and interpretation. Health Qual Life Outcomes. 2003;1(1):1–7.

Mundt JC, Marks IM, Shear MK, Greist JM. The Work and Social Adjustment Scale: a simple measure of impairment in functioning. Br J Psychiatry. 2002;180(5):461–4.

Chalder T, Berelowitz G, Pawlikowska T, Watts L, Wessely S, Wright D, Wallace E. Development of a fatigue scale. J Psychosom Res. 1993;37(2):147–53.

Shahid A, Wilkinson K, Marcu S, Shapiro CM: Visual analogue scale to evaluate fatigue severity (VAS-F). In: STOP, THAT and one hundred other sleep scales . edn.: Springer; 2011:399–402.

Parker M, Sawant HB, Flannery T, Tarrant R, Shardha J, Bannister R, Ross D, Halpin S, Greenwood DC, Sivan M. Effect of using a structured pacing protocol on post-exertional symptom exacerbation and health status in a longitudinal cohort with the post-COVID-19 syndrome. J Med Virol. 2023;95(1): e28373.

Kenny RA, Bayliss J, Ingram A, Sutton R. Head-up tilt: a useful test for investigating unexplained syncope. The Lancet. 1986;327(8494):1352–5.

Drury MOC: Science and Psychology. In: The selected writings of Maurice O’Connor Drury: On Wittgenstein, philosophy, religion and psychiatry. edn.: Bloomsbury Publishing; 2017.

Concato J, Shah N, Horwitz RI. Randomized, controlled trials, observational studies, and the hierarchy of research designs. N Engl J Med. 2000;342(25):1887–92.

Mongtomery K: How doctors think: Clinical judgment and the practice of medicine: Oxford University Press; 2005.

Download references

Acknowledgements

We are grateful to clinic staff for allowing us to study their work and to patients for allowing us to sit in on their consultations. We also thank the funder of LOCOMOTION (National Institute for Health Research) and the patient advisory group for lived experience input.

This research is supported by National Institute for Health Research (NIHR) Long Covid Research Scheme grant (Ref COV-LT-0016).

Author information

Authors and affiliations.

Nuffield Department of Primary Care Health Sciences, University of Oxford, Woodstock Rd, Oxford, OX2 6GG, UK

Trisha Greenhalgh, Julie L. Darbyshire & Emma Ladds

Imperial College Healthcare NHS Trust, London, UK

LOCOMOTION Patient Advisory Group and Lived Experience Representative, London, UK

You can also search for this author in PubMed   Google Scholar

Contributions

TG conceptualized the overall study, led the empirical work, supported the quality improvement meetings, conducted the ethnographic visits, led the data analysis, developed the theorization and wrote the first draft of the paper. JLD organized and led the quality improvement meetings, supported site-based researchers to collect and analyse data on their clinic, collated and summarized data on quality topics, and liaised with the patient advisory group. CL conceptualized and led the quality topic on POTS, including exploring reasons for some clinics’ reluctance to conduct testing and collating and analysing the NASA Lean Test data across all sites. EL assisted with ethnographic visits, data analysis, and theorization. JCS contributed lived experience of long covid and also clinical experience as an occupational therapist; she liaised with the wider patient advisory group, whose independent (patient-led) audit of long covid clinics informed the quality improvement prioritization exercise. All authors provided extensive feedback on drafts and contributed to discussions and refinements. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Trisha Greenhalgh .

Ethics declarations

Ethics approval and consent to participate.

LOng COvid Multidisciplinary consortium Optimising Treatments and servIces acrOss the NHS study is sponsored by the University of Leeds and approved by Yorkshire & The Humber—Bradford Leeds Research Ethics Committee (ref: 21/YH/0276) and subsequent amendments.

Patient participants in clinic were approached by the clinician (without the researcher present) and gave verbal informed consent for a clinically qualified researcher to observe the consultation. If they consented, the researcher was then invited to sit in. A written record was made in field notes of this verbal consent. It was impractical to seek consent from patients whose cases were discussed (usually with very brief clinical details) in online MDTs. Therefore, clinical case examples from MDTs presented in the paper are fictionalized cases constructed from multiple real cases and with key clinical details changed (for example, comorbidities were replaced with different conditions which would produce similar symptoms). All fictionalized cases were checked by our patient advisory group to check that they were plausible to lived experience experts.

Consent for publication

No direct patient cases are reported in this manuscript. For details of how the fictionalized cases were constructed and validated, see “Consent to participate” above.

Competing interests

TG was a member of the UK National Long Covid Task Force 2021–2023 and on the Oversight Group for the NICE Guideline on Long Covid 2021–2022. She is a member of Independent SAGE.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Greenhalgh, T., Darbyshire, J.L., Lee, C. et al. What is quality in long covid care? Lessons from a national quality improvement collaborative and multi-site ethnography. BMC Med 22 , 159 (2024). https://doi.org/10.1186/s12916-024-03371-6

Download citation

Received : 04 December 2023

Accepted : 26 March 2024

Published : 15 April 2024

DOI : https://doi.org/10.1186/s12916-024-03371-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Post-covid-19 syndrome
• Quality improvement
• Breakthrough collaboratives
• Warranted variation
• Unwarranted variation
• Improvement science
• Ethnography
• Idiographic reasoning
• Nomothetic reasoning

BMC Medicine

ISSN: 1741-7015

• Submission enquiries: [email protected]
• General enquiries: [email protected]

About Us: Research Quality Management Staff

• About Us: Office of Research & Doctoral Services
• About Us: Student Research Administration Staff
• About Us: Research Ethics Staff
• About Us: Research and Sponsored Programs Staff
• Research Quality Management Staff
• Previous Page: About Us: Research and Sponsored Programs Staff

Research Quality Management

Research Quality Management is charged with maintaining and continuously improving the quality of research associated with the university. It provides support to facilitate student research by reviewing and maintaining the graduate research curriculum. It also provides training to faculty members seeking to work as dissertation chairs of capstone committees and maintains university-level oversight of the CAO review process. Finally, the office provides resources related to planning and writing research studies, research tutorials/guides, locating faculty expertise, and consultations regarding research methods for faculty.

• Methodology Advisors (Faculty Only):  [email protected]
• Walden Participant Pool:   [email protected]  
• ICPSR or other dataset questions:  [email protected]  

Deborah Inman

Marydee A. Spillett

Morris Bidjerano

Reggie Taylor

Arfe Ozcan, PhD

Michelle Stallone Brown

Lee Stadtlander

Shereeza Mohammed, PhD

Zin Htway, PhD

Social media, connect with the office of research and doctoral services.

• Office of Student Disability Services

Walden Resources

Departments.

• Academic Residencies
• Academic Skills
• Career Planning and Development
• Customer Care Team
• Field Experience
• Military Services
• Student Success Advising
• Writing Skills

Centers and Offices

• Center for Social Change
• Office of Academic Support and Instructional Services
• Office of Degree Acceleration
• Office of Research and Doctoral Services
• Office of Student Affairs

Student Resources

• Doctoral Writing Assessment
• Form & Style Review
• Quick Answers
• ScholarWorks
• SKIL Courses and Workshops
• Walden Bookstore
• Walden Catalog & Student Handbook
• Student Safety/Title IX
• Legal & Consumer Information
• Website Terms and Conditions
• Cookie Policy
• Accessibility
• Accreditation
• State Authorization
• Net Price Calculator
• Contact Walden

Walden University is a member of Adtalem Global Education, Inc. www.adtalem.com Walden University is certified to operate by SCHEV © 2024 Walden University LLC. All rights reserved.

• Schools & departments

Research quality

Institutional performance in funding council assessments of research.

This paper is in the following e-collection/theme issue:

Published on 17.4.2024 in Vol 26 (2024)

Service Quality and Residents’ Preferences for Facilitated Self-Service Fundus Disease Screening: Cross-Sectional Study

Authors of this article:

Original Paper

• Senlin Lin 1, 2, 3 * , MSc   ; 
• Yingyan Ma 1, 2, 3, 4 * , PhD   ; 
• Yanwei Jiang 5 * , MPH   ; 
• Wenwen Li 6 , PhD   ; 
• Yajun Peng 1, 2, 3 , BA   ; 
• Tao Yu 1, 2, 3 , BA   ; 
• Yi Xu 1, 2, 3 , MD   ; 
• Jianfeng Zhu 1, 2, 3 , MD   ; 
• Lina Lu 1, 2, 3 , MPH   ; 
• Haidong Zou 1, 2, 3, 4 , MD  

1 Shanghai Eye Diseases Prevention &Treatment Center/ Shanghai Eye Hospital, School of Medicine, Tongji University, Shanghai, China

2 National Clinical Research Center for Eye Diseases, Shanghai, China

3 Shanghai Engineering Research Center of Precise Diagnosis and Treatment of Eye Diseases, Shanghai, China

4 Shanghai General Hospital, School of Medicine, Shanghai Jiao Tong University, Shanghai, China

5 Shanghai Hongkou Center for Disease Control and Prevention, Shanghai, China

6 School of Management, Fudan University, Shanghai, China

*these authors contributed equally

Corresponding Author:

Haidong Zou, MD

Shanghai Eye Diseases Prevention &Treatment Center/ Shanghai Eye Hospital

School of Medicine

Tongji University

No 1440, Hongqqiao Road

Shanghai, 200336

Phone: 86 02162539696

Email: [email protected]

Background: Fundus photography is the most important examination in eye disease screening. A facilitated self-service eye screening pattern based on the fully automatic fundus camera was developed in 2022 in Shanghai, China; it may help solve the problem of insufficient human resources in primary health care institutions. However, the service quality and residents’ preference for this new pattern are unclear.

Objective: This study aimed to compare the service quality and residents’ preferences between facilitated self-service eye screening and traditional manual screening and to explore the relationships between the screening service’s quality and residents’ preferences.

Methods: We conducted a cross-sectional study in Shanghai, China. Residents who underwent facilitated self-service fundus disease screening at one of the screening sites were assigned to the exposure group; those who were screened with a traditional fundus camera operated by an optometrist at an adjacent site comprised the control group. The primary outcome was the screening service quality, including effectiveness (image quality and screening efficiency), physiological discomfort, safety, convenience, and trustworthiness. The secondary outcome was the participants’ preferences. Differences in service quality and the participants’ preferences between the 2 groups were compared using chi-square tests separately. Subgroup analyses for exploring the relationships between the screening service’s quality and residents’ preference were conducted using generalized logit models.

Results: A total of 358 residents enrolled; among them, 176 (49.16%) were included in the exposure group and the remaining 182 (50.84%) in the control group. Residents’ basic characteristics were balanced between the 2 groups. There was no significant difference in service quality between the 2 groups (image quality pass rate: P =.79; average screening time: P =.57; no physiological discomfort rate: P =.92; safety rate: P =.78; convenience rate: P =.95; trustworthiness rate: P =.20). However, the proportion of participants who were willing to use the same technology for their next screening was significantly lower in the exposure group than in the control group ( P <.001). Subgroup analyses suggest that distrust in the facilitated self-service eye screening might increase the probability of refusal to undergo screening ( P =.02).

Conclusions: This study confirms that the facilitated self-service fundus disease screening pattern could achieve good service quality. However, it was difficult to reverse residents’ preferences for manual screening in a short period, especially when the original manual service was already excellent. Therefore, the digital transformation of health care must be cautious. We suggest that attention be paid to the residents’ individual needs. More efficient man-machine collaboration and personalized health management solutions based on large language models are both needed.

Introduction

Vision impairment and blindness are caused by a variety of eye diseases, including cataracts, glaucoma, uncorrected refractive error, age-related macular degeneration, diabetic retinopathy, and other eye diseases [ 1 ]. They not only reduce economic productivity but also harm the quality of life and increase mortality [ 2 - 6 ]. In 2020, an estimated 43.3 million individuals were blind, and 1.06 billion individuals aged 50 years and older had distance or near vision impairment [ 7 ]. With an increase in the aging population, the number of individuals affected by vision loss has increased substantially [ 1 ].

High-quality public health care for eye disease prevention, such as effective screening, can assist in eliminating approximately 57% of all blindness cases [ 8 ]. Digital technologies, such as telemedicine, 5G telecommunications, the Internet of Things, and artificial intelligence (AI), have provided the potential to improve the accessibility, availability, and productivity of existing resources and the overall efficiency of eye care services [ 9 , 10 ]. The use of digital technology not only reduces the cost of eye disease screening and improves its efficiency, but also assists residents living in remote areas to gain access to eye disease screening [ 11 - 13 ]. Therefore, an increasing number of countries (or regions) are attempting to establish eye screening systems based on digital technology [ 9 ].

Fundus photography is the most important examination in eye disease screening because the vast majority of diagnoses of blinding retinal diseases are based on fundus photographs. Diagnoses can be made by human experts or AI software. However, traditional fundus cameras must be operated by optometrists, who are usually in short supply in primary health care institutions when faced with the large demand for screening services.

Fortunately, the fully automatic fundus camera has been developed on the basis of digital technologies including AI, industrial automation, sensors, and voice navigation. It can automatically identify the person’s left and right eyes, search for pupils, adjust the lens position and shooting focus, and provide real-time voice feedback during the process, helping the residents to understand the current inspection steps clearly and cooperatively complete the inspection. Therefore, a facilitated self-service eye screening pattern has been newly established in 2022 in Shanghai, China.

However, evidence is inadequate about whether this new screening pattern performs well and whether the residents prefer it. Therefore, this cross-sectional study aims to compare the service quality and residents’ preferences of this new screening pattern with that of the traditional screening pattern. We aimed to (1) investigate whether the facilitated self-service eye screening can achieve service quality similar to that of traditional manual screening, (2) compare residents’ preferences between the facilitated self-service eye screening and traditional manual screening, and (3) explore the relationship between the screening service quality and residents’ preferences.

Study Setting

This study was conducted in Shanghai, China, in 2022. Since 2010, Shanghai has conducted an active community-based fundus disease telemedicine screening program. After 2018, an AI model was adopted ( Figure 1 ). At the end of 2021, the fully automatic fundus camera was adopted, and the facilitated self-service fundus disease screening pattern was established ( Figure 1 ). Within this new pattern, residents could perform fundus photography by themselves without professionals’ assistance ( Multimedia Appendix 1 ). The fundus images were sent to the cloud server center of the AI model, and the screening results were fed back immediately.

Study Design

We conducted a cross-sectional study at 2 adjacent screening sites. These 2 sites were expected to be very similar in terms of their socioeconomic and educational aspects since they were located next to each other. One site provided facilitated self-service fundus disease screening, and the residents who participated therein comprised the exposure group; the other site provided screening with a traditional fundus camera operated by an optometrist, and the residents who participated therein comprised the control group. All the adult residents could participant in our screening program, but their data were used for analysis only if they signed the informed consent form. Residents could opt out of the study at any time during the screening.

In the exposure group, the residents were assessed using an updated version of the nonmydriatic fundus camera Kestrel 3100m (Shanghai Top View Industrial Co Ltd) with a self-service module. In the process of fundus photography, the residents pressed the “Start” button by themselves. All checking steps (including focusing, shooting, and image quality review) were undertaken automatically by the fundus camera ( Figure 2 ). Screening data were transmitted to the AI algorithm on a cloud-based server center through the telemedicine platform, and the screening results were fed back immediately. Residents were fully informed that the assessment was fully automated and not performed by the optometrist.

In the control group, the residents were assessed using the basic version of the same nonmydriatic fundus camera. The optical components were identical to those in the exposure group but without the self-service module. In the process of fundus photography, all steps were carried out by the optometrist (including focusing, shooting, and image quality review). Screening data were transmitted to the AI algorithm on a cloud-based server center through the telemedicine platform, and the screening results were fed back immediately. Residents were also fully informed.

Measures and Outcomes

The primary outcome was the screening service’s quality. Based on the World Health Organization’s recommendations for the evaluation of AI-based medical devices [ 14 ] and the European Union’s Assessment List for Trustworthy Artificial Intelligence [ 15 ], 5 dimensions were selected to reflect the service quality of eye disease screening: effectiveness, physiological discomfort, safety, convenience, and trustworthiness.

Furthermore, effectiveness was based on 2 indicators: image quality and screening efficiency. A staff member recorded the time required for each resident to take fundus photographs (excluding the time taken for diagnosis) at the screening site. Then, a professional ophthalmologist evaluated the quality of each fundus photograph after the on-site experiment. The ophthalmologist was blinded to the grouping of participants. Image quality was assessed on the basis of the image quality pass rate, expressed as the number of eyes with high-quality fundus images per 100 eyes. Screening efficiency was assessed on the basis of the average screening time, expressed as the mean of the time required for each resident to take fundus photographs.

To assess physiological discomfort, safety, convenience, and trustworthiness of screening services, residents were asked to finish a questionnaire just after they received the screening results. A 5-point Likert scale was adopted for each dimension, from the best to the worst, except for the physiological discomfort ( Multimedia Appendix 2 ). A no physiological discomfort rate was expressed as the number of residents who chose the “There is no physiological discomfort during the screening” per 100 individuals in each group. Safety rate is expressed as the number of residents who chose “The screening is very safe” or “The screening is safe” per 100 individuals in each group. Convenience rate is expressed as the number of residents who chose “The screening is very convenient” or “The screening is convenient” per 100 individuals in each group. The trustworthiness rate is expressed as the number of residents who chose “The screening result is very trustworthy” or “The screening result is trustworthy” per 100 individuals in each group.

The secondary outcome was the preference rate, expressed as the number of residents who were willing to use the same technology for their next screening per 100 individuals. In detail, in the exposure group, the preference rate was expressed as the number of the residents who preferred facilitated self-service eye screening per 100 individuals, while in the control group, it was expressed as the number of residents who preferred traditional manual screening per 100 individuals.

To understand the residents’ preference, a video displaying the processes of both facilitated self-service eye screening and traditional manual screening was shown to the residents. Then, the following question was asked: “At your next eye disease screening, you can choose either facilitated self-service eye screening or traditional manual screening. Which one do you prefer?” A total of 4 alternatives were set: “Prefer traditional manual screening,” “Prefer facilitated self-service eye screening,” “Both are acceptable,” and “Neither is acceptable (Refusal of screening).” Each resident could choose only 1 option, which best reflected their preference.

Sample Size

The rule of events per variable was used for sample size estimation. In this study, 2 logit models were established for the 2 groups separately, each containing 8 independent variables. We set 10 events per variable in general. According to a previous study [ 16 ], when the decision-making process had high uncertainty, the proportion of individuals who preferred the algorithms was about 50%. This led us to arrive at a sample size of 160 (8 variables multiplied by 10 events each, with 50% of individuals potentially preferring facilitated screening [ie, 50% of 8×10]) for each group.

Every dimension of the screening service quality and the preference rate were calculated separately. Chi-square and t tests were used to test whether the service quality or the residents’ preferences differed between the 2 groups. A total of 7 hypotheses were tested, as shown in Textbox 1 .

• H1: image quality pass rate exposure group ≠ image quality pass rate control group H0: image quality pass rate exposure group =image quality pass rate control group
• H1: screening time exposure group ≠screening time control group H0: screening time exposure group =screening time control group
• H1: no discomfort rate exposure group ≠no discomfort rate control group H0: no discomfort rate exposure group = no discomfort rate control group
• H1: safety rate exposure group ≠safety rate control group H0: safety rate exposure group = safety rate control group
• H1: convenience rate exposure group ≠convenience rate control group H0: convenience rate exposure group = convenience rate control group
• H1: trustworthiness rate exposure group ≠trustworthiness rate control group H0: trustworthiness rate exposure group = trustworthiness rate control group
• H1: preference rate exposure group ≠preference rate control group H0: preference rate exposure group = preference rate control group

If any of the hypotheses among hypotheses 1-6 ( Textbox 1 ) were significant, it indicated that the service quality was different between facilitated self-service eye screening and traditional manual screening. If hypothesis 7 was significant, it meant that the residents’ preference for facilitated self-service eye screening was different from that for traditional manual screening.

Additionally, subgroup analyses in the exposure and control groups were conducted to explore the relationships between the screening service quality and the residents’ preferences, using generalized logit models. The option “Prefer facilitated self-service eye screening” was used as the reference level for the dependent variable in the models. The independent variables included age, sex, image quality, screening efficiency, physiological discomfort, safety, convenience, and trustworthiness. All statistics were performed using SAS (version 9.4; SAS Institute).

Ethical Considerations

The study adhered to the ethical principles of the Declaration of Helsinki and was approved by the Shanghai General Hospital Ethics Committee (2022SQ272). All participants provided written informed consent before participating in this study. The study data were anonymous, and no identification of individual participants in any images of the manuscript or supplementary material is possible.

Participants’ Characteristics

A total of 358 residents enrolled; among them, 176 (49.16%) were in the exposure group and the remaining 182 (50.84%) were in the control group. Residents’ basic characteristics were balanced between the 2 groups. The mean age was 65.05 (SD 12.28) years for the exposure group and 63.96 (SD 13.06) years for the control group; however, this difference was nonsignificant ( P =.42). The proportion of women was 67.05% (n=118) for the exposure group and 62.09% (n=113) for the control group; this difference was also nonsignificant between the 2 groups ( P =.33).

Screening Service Quality

In the exposure group, high-quality fundus images were obtained for 268 out of 352 eyes (image quality pass rate=76.14%; Figure 3 ). The average screening time was 81.03 (SD 36.98) seconds ( Figure 3 ). In the control group, high-quality fundus images were obtained for 274 out of 364 eyes (image quality pass rate=75.27%; Figure 3 ). The average screening time was 78.22 (SD 54.01) seconds ( Figure 3 ). There was no significant difference in the image quality pass rate ( χ 2 1 =0.07, P =.79) and average screening time ( t 321.01 =–0.58 [Welch–Satterthwaite–adjusted df ], P =.56) between the 2 groups ( Figure 3 ).

For the other dimensions, detailed information is shown in Figure 3 . There were no significant differences between any of these rates between the 2 groups (no physiological discomfort rate: χ 2 1 =0.01, P =.92; safety rate: χ 2 1 =0.08, P =.78; convenience rate: χ 2 1 =0.004, P =.95; trustworthiness rate: χ 2 1 =1.63, P =.20).

Residents’ Preferences

In the exposure group, 120 (68.18%) residents preferred traditional manual screening, 19 (10.80%) preferred facilitated self-service eye screening, 19 (10.80%) preferred both, and the remaining 18 (10.23%) preferred neither. In the control group, 123 (67.58%) residents preferred traditional manual screening, 14 (7.69%) preferred facilitated self-service eye screening, 20 (10.99%) preferred both, and the remaining 25 (13.74%) preferred neither.

The proportion of residents who chose the category “Prefer facilitated self-service eye screening” in the exposure group was significantly lower than that of residents who chose the category “Prefer traditional manual screening” in the control group ( χ 2 1 =120.57, P <.001; Figure 3 ).

Subgroup Analyses

In the exposure group, 4 generalized logit models were generated ( Table 1 ). Regarding the effectiveness of facilitated self-service eye screening, neither the image quality nor the screening time had an impact on the residents’ preferences. Regarding the other dimensions for facilitated self-service eye screening service quality, models 3 and 4 demonstrated that distrust in the results of facilitated self-service eye screening might decrease the probability of preferring this screening service and increase the probability of preferring neither of the 2 screening services.

a Age and gender were adjusted in model 1. Age, gender, image quality, and screening efficiency were adjusted in model 2. Age, gender, physiological discomfort, safety, convenience, and trustworthiness were adjusted in model 3. Age, gender, image quality, screening efficiency, physiological discomfort, safety, convenience, and trustworthiness were adjusted in model 4.

b In the exposure group, distrust in the results of facilitated self-service eye screening might decrease the probability of preferring this screening service and increase the probability of preferring neither the traditional nor the facilitated self-service screening services.

c Not available.

In the control group, another 4 generalized logit models were generated ( Table 2 ). Men were more likely to choose a preference both screening services. The probability of preferring manual screening might increase with age, as long as the probability of preferring facilitated self-service eye screening decreased. Regarding the effectiveness of traditional manual screening, neither the image quality pass rate nor the screening time had an impact on the residents’ preferences. For the other dimensions of the quality of traditional manual screening, models 7 and 8 showed that if the residents feel unsafe about traditional manual screening, their preference for traditional manual screening might decrease, and they might turn to facilitated self-service eye screening.

a Age and gender were adjusted in model 5. Age, gender, image quality, and screening efficiency were adjusted in model 6. Age, gender, physiological discomfort, safety, convenience, and trustworthiness were adjusted in model 7. Age, gender, image quality, screening efficiency, physiological discomfort, safety, convenience, and trustworthiness were adjusted in model 8.

b In the control group, if the residents feel unsafe about traditional manual screening, their preference for traditional manual screening might decrease, and they might turn to facilitated self-service eye screening.

A new fundus disease screening pattern was established using the fully automatic fundus camera without any manual intervention. Our findings suggest that facilitated self-service eye screening can achieve a service quality similar to that of traditional manual screening. The study further evaluated the residents’ preferences and associated factors for the newly established self-service fundus disease screening. Our study found that the residents’ preference for facilitated self-service eye screening is significantly less than that for traditional manual screening. This implies that the association between the service quality of the screening technology and residents’ preferences was weak, suggesting that aversion to the algorithm might exist. In addition, the subgroup analyses suggest that even the high quality of facilitated self-service eye screening cannot increase the residents’ preference for this new screening pattern. Worse still, distrust in the results of this new pattern may lead to lower usage of eye disease screening services as a whole. To the best of our knowledge, this study is one of the first to evaluate service quality and residents’ preferences for facilitated self-service fundus disease screening.

Previous studies have suggested that people significantly prefer manual services to algorithms in the field of medicine [ 16 - 18 ]. Individuals have an aversion to algorithms underlying digital technology, especially when they see errors in the algorithm’s functioning [ 18 ]. The preference for algorithms does not increase even if the residents are told that the algorithm outperforms human doctors [ 19 , 20 ]. Our results confirm that fundus image quality in the exposure group is similar to that in the control group in our study, and both are similar to or even better than those reported in previous studies [ 21 , 22 ]. However, the preference for facilitated self-service fundus disease screening is significantly less than that for traditional manual screening. One possible explanation is that uniqueness neglect—a concern that algorithm providers are less able than human providers to account for residents’ (or patients’) unique characteristics and circumstances—drives consumer resistance to digital medical technology [ 23 ]. Therefore, personalized health management solutions based on large language models should be developed urgently [ 24 ] to meet the residents’ individual demands. In addition, a survey of population preferences for medical AI indicated that the most important factor for the public is that physicians are ultimately responsible for diagnosis and treatment planning [ 25 ]. As a result, man-machine collaboration, such as human supervision, is still necessary [ 26 ], especially in the early stages of digital transformation to help residents understand and accept the digital technologies.

Furthermore, our study suggests that distrust in the results of facilitated self-service fundus disease screening may cause residents to abandon eye disease screening, irrespective of whether it is provided using this new screening pattern or via the traditional manual screening pattern. This is critical to digital transformation in medicine. This implies that if the digital technology does not perform well, residents will not only be averse to the digital technology itself but also be more likely to abandon health care services as a whole. Digital transformation is a fundamental change to the health care delivery system. This implies that it can self-disrupt its ability to question the practices and production models of existing health care services. As a result, it may become incompatible with the existing models, processes, activities, and even cultures [ 27 ]. Therefore, it is important to assess whether the adoption of digital technologies contributes to health system objectives in an optimal manner, and this assessment should be carried out at the level of health services but not at the level of digital transformation [ 28 ].

The most prominent limitation of our study is that it was conducted only in Shanghai, China. Because of the sound health care system in Shanghai, residents have already received high-quality eye disease screening services before the adoption of the facilitated self-service eye screening pattern. Consequently, residents are bound to demand more from this new pattern. This situation is quite different from that in lower-income regions. Digital technology was adapted in poverty-stricken areas to build an eye care system, but it did not replace the original system that is based on manually delivered services [ 13 ]. Therefore, the framing effect may be weak [ 29 ], and there is little practical value in comparing digital technology and manual services in these regions. Second, our study is an observational study and blind grouping was not practical due to the special characteristics of fundus examination. However, we have attempted to use blind processing whenever possible. For instance, ophthalmologists’ evaluation of image quality was conducted in a blinded manner. Third, the manner in which we inquired about residents’ preferences might affect the results. For example, participants in the exposure group generally have experience with manual screening, but those in the control group may not have had enough experience with facilitated screening despite having been shown a video. This might make the participants in the control group more likely to choose manual screening because the new technology was unfamiliar. Finally, individual-level socioeconomic factors or educational level were not recorded, so we cannot rule out the influence of these factors on residents’ preferences.

In summary, this study confirms that the facilitated self-service fundus disease screening pattern could achieve high service quality. The preference of the residents for this new mode, however, was not ideal. It was difficult to reverse residents’ preference for manual screening in a short period, especially when the original manual service was already excellent. Therefore, the digital transformation of health care must proceed with caution. We suggest that attention be paid to the residents’ individual needs. Although more efficient man-machine collaboration is necessary to help the public understand and accept new technologies, personalized health management solutions based on large language models are required.

Acknowledgments

This study was funded by the Shanghai Public Health Three-Year Action Plan (GWVI-11.1-30, GWVI-11.1-22), Science and Technology Commission of Shanghai Municipality (20DZ1100200 and 23ZR1481000), Shanghai Municipal Health Commission (2022HP61, 2022YQ051, and 20234Y0062), Shanghai First People's Hospital featured research projects (CCTR-2022C08) and Medical Research Program of Hongkou District Health Commission (Hongwei2202-07).

Data Availability

Data are available from the corresponding author upon reasonable request.

Authors' Contributions

SL, YM, and YJ contributed to the conceptualization and design of the study. SL, YM, YJ, YP, TY, and YX collected the data. SL and YM analyzed the data. SL, YM, and YJ drafted the manuscript. WL, YX, JZ, LL, and HZ extensively revised the manuscript. All authors read and approved the final manuscript submitted.

Conflicts of Interest

None declared.

Video of the non-mydriatic fundus camera Kestrel-3100m with the self-service module.

Questions for screening service quality.

• GBD 2019 BlindnessVision Impairment Collaborators, Vision Loss Expert Group of the Global Burden of Disease Study. Causes of blindness and vision impairment in 2020 and trends over 30 years, and prevalence of avoidable blindness in relation to VISION 2020: the Right to Sight: an analysis for the Global Burden of Disease Study. Lancet Glob Health. Feb 2021;9(2):e144-e160. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Marques AP, Ramke J, Cairns J, Butt T, Zhang JH, Muirhead D, et al. Global economic productivity losses from vision impairment and blindness. EClinicalMedicine. May 2021;35:100852. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Jan C, Li S, Kang M, Liu L, Li H, Jin L, et al. Association of visual acuity with educational outcomes: a prospective cohort study. Br J Ophthalmol. Nov 18, 2019;103(11):1666-1671. [ CrossRef ] [ Medline ]
• Chai YX, Gan ATL, Fenwick EK, Sui AY, Tan BKJ, Quek DQY, et al. Relationship between vision impairment and employment. Br J Ophthalmol. Mar 16, 2023;107(3):361-366. [ CrossRef ] [ Medline ]
• Nayeni M, Dang A, Mao AJ, Malvankar-Mehta MS. Quality of life of low vision patients: a systematic review and meta-analysis. Can J Ophthalmol. Jun 2021;56(3):151-157. [ CrossRef ] [ Medline ]
• Wang L, Zhu Z, Scheetz J, He M. Visual impairment and ten-year mortality: the Liwan Eye Study. Eye (Lond). Aug 19, 2021;35(8):2173-2179. [ FREE Full text ] [ CrossRef ] [ Medline ]
• GBD 2019 BlindnessVision Impairment Collaborators, Vision Loss Expert Group of the Global Burden of Disease Study. Trends in prevalence of blindness and distance and near vision impairment over 30 years: an analysis for the Global Burden of Disease Study. Lancet Glob Health. Feb 2021;9(2):e130-e143. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Cheng C, Wang N, Wong TY, Congdon N, He M, Wang YX, et al. Vision Loss Expert Group of the Global Burden of Disease Study. Prevalence and causes of vision loss in East Asia in 2015: magnitude, temporal trends and projections. Br J Ophthalmol. May 28, 2020;104(5):616-622. [ CrossRef ] [ Medline ]
• Li JO, Liu H, Ting DS, Jeon S, Chan RP, Kim JE, et al. Digital technology, tele-medicine and artificial intelligence in ophthalmology: a global perspective. Prog Retin Eye Res. May 2021;82:100900. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ting DSW, Pasquale LR, Peng L, Campbell JP, Lee AY, Raman R, et al. Artificial intelligence and deep learning in ophthalmology. Br J Ophthalmol. Feb 25, 2019;103(2):167-175. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Xie Y, Nguyen QD, Hamzah H, Lim G, Bellemo V, Gunasekeran DV, et al. Artificial intelligence for teleophthalmology-based diabetic retinopathy screening in a national programme: an economic analysis modelling study. Lancet Digit Health. May 2020;2(5):e240-e249. [ CrossRef ]
• Tang J, Liang Y, O'Neill C, Kee F, Jiang J, Congdon N. Cost-effectiveness and cost-utility of population-based glaucoma screening in China: a decision-analytic Markov model. Lancet Glob Health. Jul 2019;7(7):e968-e978. [ CrossRef ]
• Xiao X, Xue L, Ye L, Li H, He Y. Health care cost and benefits of artificial intelligence-assisted population-based glaucoma screening for the elderly in remote areas of China: a cost-offset analysis. BMC Public Health. Jun 04, 2021;21(1):1065. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Generating Evidence for Artificial Intelligence Based Medical Devices: A Framework for Training Validation and Evaluation. World Health Organization. URL: https://www.who.int/publications/i/item/9789240038462 [accessed 2024-03-27]
• The Assessment List for Trustworthy Artificial Intelligence. URL: https://altai.insight-centre.org/ [accessed 2024-03-27]
• Dietvorst BJ, Bharti S. People reject algorithms in uncertain decision domains because they have diminishing sensitivity to forecasting error. Psychol Sci. Oct 11, 2020;31(10):1302-1314. [ CrossRef ] [ Medline ]
• DeCamp M, Tilburt JC. Why we cannot trust artificial intelligence in medicine. Lancet Digit Health. Dec 2019;1(8):e390. [ CrossRef ]
• Frank D, Elbæk CT, Børsting CK, Mitkidis P, Otterbring T, Borau S. Drivers and social implications of artificial intelligence adoption in healthcare during the COVID-19 pandemic. PLoS One. Nov 22, 2021;16(11):e0259928. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Juravle G, Boudouraki A, Terziyska M, Rezlescu C. Trust in artificial intelligence for medical diagnoses. Prog Brain Res. 2020;253:263-282. [ CrossRef ] [ Medline ]
• Liu X, Faes L, Kale AU, Wagner SK, Fu DJ, Bruynseels A, et al. A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis. Lancet Digit Health. Oct 2019;1(6):e271-e297. [ CrossRef ]
• Scanlon PH, Foy C, Malhotra R, Aldington SJ. The influence of age, duration of diabetes, cataract, and pupil size on image quality in digital photographic retinal screening. Diabetes Care. Oct 2005;28(10):2448-2453. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Cen L, Ji J, Lin J, Ju S, Lin H, Li T, et al. Automatic detection of 39 fundus diseases and conditions in retinal photographs using deep neural networks. Nat Commun. Aug 10, 2021;12(1):4828. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Longoni C, Bonezzi A, Morewedge C. Resistance to medical artificial intelligence. J Consum Res. 2019;46:650. [ CrossRef ]
• Huang AS, Hirabayashi K, Barna L, Parikh D, Pasquale LR. Assessment of a Large Language Model's Responses to Questions and Cases About Glaucoma and Retina Management. JAMA Ophthalmol. Feb 22, 2024. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ploug T, Sundby A, Moeslund TB, Holm S. Population preferences for performance and explainability of artificial intelligence in health care: choice-based conjoint survey. J Med Internet Res. Dec 13, 2021;23(12):e26611. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Young AT, Amara D, Bhattacharya A, Wei ML. Patient and general public attitudes towards clinical artificial intelligence: a mixed methods systematic review. Lancet Digit Health. Sep 2021;3(9):e599-e611. [ CrossRef ]
• Alami H, Gagnon M, Fortin J. Digital health and the challenge of health systems transformation. Mhealth. Aug 08, 2017;3:31-31. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ricciardi W, Pita Barros P, Bourek A, Brouwer W, Kelsey T, Lehtonen L, et al. Expert Panel on Effective Ways of Investing in Health (EXPH). How to govern the digital transformation of health services. Eur J Public Health. Oct 01, 2019;29(Supplement_3):7-12. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Khan WU, Shachak A, Seto E. Understanding decision-making in the adoption of digital health technology: the role of behavioral economics' prospect theory. J Med Internet Res. Feb 07, 2022;24(2):e32714. [ FREE Full text ] [ CrossRef ] [ Medline ]

Abbreviations

Edited by A Mavragani; submitted 06.01.23; peer-reviewed by B Li, A Bate, CW Pan; comments to author 13.09.23; revised version received 15.10.23; accepted 12.03.24; published 17.04.24.

©Senlin Lin, Yingyan Ma, Yanwei Jiang, Wenwen Li, Yajun Peng, Tao Yu, Yi Xu, Jianfeng Zhu, Lina Lu, Haidong Zou. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 17.04.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.