how to do a literature search for a literature review

How to undertake a literature search: a step-by-step guide

Affiliation.

• 1 Literature Search Specialist, Library and Archive Service, Royal College of Nursing, London.
• PMID: 32279549
• DOI: 10.12968/bjon.2020.29.7.431

Undertaking a literature search can be a daunting prospect. Breaking the exercise down into smaller steps will make the process more manageable. This article suggests 10 steps that will help readers complete this task, from identifying key concepts to choosing databases for the search and saving the results and search strategy. It discusses each of the steps in a little more detail, with examples and suggestions on where to get help. This structured approach will help readers obtain a more focused set of results and, ultimately, save time and effort.

Keywords: Databases; Literature review; Literature search; Reference management software; Research questions; Search strategy.

• Databases, Bibliographic*
• Information Storage and Retrieval / methods*
• Nursing Research
• Review Literature as Topic*

Help us improve our Library guides with this 5 minute survey . We appreciate your feedback!

• UOW Library
• Key guides for students

Literature Review

How to search effectively.

• Find examples of literature reviews
• How to write a literature review
• Grey literature

The  Literature searching interactive tutorial  includes self-paced, guided activities to assist you in developing  effective search skills..

1. Identify search words

Analyse your research topic or question.

• What are the main ideas?
• What concepts or theories have you already covered?
• Write down your main ideas, synonyms, related words and phrases.
• If you're looking for specific types of research, use these suggested terms: qualitative, quantitative, methodology, review, survey, test, trend (and more).
• Be aware of UK and US spelling variations. E.g. organisation OR organization, ageing OR aging.
• Interactive Keyword Builder
• Identifying effective keywords

2. Connect your search words

Find results with one or more search words.

Use OR between words that mean the same thing.

E.g.  adolescent  OR  teenager

This search will find results with either (or both) of the search words.

Find results with two search words

Use AND between words which represent the main ideas in the question.

E.g. adolescent AND “physical activity”

This will find results with both of the search words.

Exclude search words

Use NOT to exclude words that you don’t want in your search results.

E.g. (adolescent OR teenager) NOT “young adult”

3. Use search tricks

Search for different word endings.

Truncation *

The asterisk symbol * will help you search for different word endings.

E.g. teen* will find results with the words: teen, teens, teenager, teenagers

Specific truncation symbols will vary. Check the 'Help' section of the database you are searching.

Search for common phrases

Phrase searching “...........”

Double quotation marks help you search for common phrases and make your results more relevant.

E.g. “physical activity” will find results with the words physical activity together as a phrase.

Search for spelling variations within related terms

Wildcards ?

Wildcard symbols allow you to search for spelling variations within the same or related terms.

E.g. wom?n will find results with women OR woman

Specific wild card symbols will vary. Check the 'Help' section of the database you are searching.

Search terms within specific ranges of each other

Proximity  w/#

Proximity searching allows you to specify where your search terms will appear in relation to each other.

E.g.  pain w/10 morphine will search for pain within ten words of morphine

Specific proximity symbols will vary. Check the 'Help' section of the database you are searching.

4. Improve your search results

All library databases are different and you can't always search and refine in the same way. Try to be consistent when transferring your search in the library databases you have chosen.

Narrow and refine your search results by:

• year of publication or date range (for recent or historical research)
• document or source type (e.g. article, review or book)
• subject or keyword (for relevance). Try repeating your search using the 'subject' headings or 'keywords' field to focus your search
• searching in particular fields, i.e. citation and abstract. Explore the available dropdown menus to change the fields to be searched.

When searching, remember to:

Adapt your search and keep trying.

Searching for information is a process and you won't always get it right the first time. Improve your results by changing your search and trying again until you're happy with what you have found.

Keep track of your searches

Keeping track of searches saves time as you can rerun them, store references, and set up regular alerts for new research relevant to your topic.

Most library databases allow you to register with a personal account. Look for a 'log in', 'sign in' or 'register' button to get started.

• Literature review search tracker (Excel spreadsheet)

Manage your references

There are free and subscription reference management programs available on the web or to download on your computer.

• EndNote - The University has a license for EndNote. It is available for all students and staff, although is recommended for postgraduates and academic staff.
• Zotero - Free software recommended for undergraduate students.
• Previous: How to write a literature review
• Next: Where to search when doing a literature review
• Last Updated: May 28, 2024 9:42 AM
• URL: https://uow.libguides.com/literaturereview

Insert research help text here

LIBRARY RESOURCES

Library homepage

Library SEARCH

A-Z Databases

STUDY SUPPORT

Academic Skills Centre

Referencing and citing

Digital Skills Hub

MORE UOW SERVICES

UOW homepage

Student support and wellbeing

IT Services

On the lands that we study, we walk, and we live, we acknowledge and respect the traditional custodians and cultural knowledge holders of these lands.

Copyright & disclaimer | Privacy & cookie usage

• UNC Libraries
• HSL Academic Process
• Systematic Reviews
• Step 3: Conduct Literature Searches

Systematic Reviews: Step 3: Conduct Literature Searches

Created by health science librarians.

• Step 1: Complete Pre-Review Tasks
• Step 2: Develop a Protocol

About Step 3: Conduct Literature Searches

Partner with a librarian, systematic searching process, choose a few databases, search with controlled vocabulary and keywords, acknowledge outdated or offensive terminology, helpful tip - building your search, use nesting, boolean operators, and field tags, build your search, translate to other databases and other searching methods, document the search, updating your review.

• Searching FAQs
• Step 4: Manage Citations
• Step 5: Screen Citations
• Step 6: Assess Quality of Included Studies
• Step 7: Extract Data from Included Studies
• Step 8: Write the Review

  Check our FAQ's

   Email us

   Call (919) 962-0800

   Make an appointment with a librarian

  Request a systematic or scoping review consultation

Search the FAQs

In Step 3, you will design a search strategy to find all of the articles related to your research question. You will:

• Define the main concepts of your topic
• Choose which databases you want to search
• List terms to describe each concept
• Add terms from controlled vocabulary like MeSH
• Use field tags to tell the database where to search for terms
• Combine terms and concepts with Boolean operators AND and OR
• Translate your search strategy to match the format standards for each database
• Save a copy of your search strategy and details about your search

There are many factors to think about when building a strong search strategy for systematic reviews. Librarians are available to provide support with this step of the process.

Click an item below to see how it applies to Step 3: Conduct Literature Searches.

Reporting your review with PRISMA

For PRISMA, there are specific items you will want to report from your search.  For this step, review the PRISMA-S checklist.

• PRISMA-S for Searching
• Specify all databases, registers, websites, organizations, reference lists, and other sources searched or consulted to identify studies. Specify the date when each source was last searched or consulted. Present the full search strategies for all databases, registers and websites, including any filters and limits used.
• For information on how to document database searches and other search methods on your PRISMA flow diagram, visit our FAQs "How do I document database searches on my PRISMA flow diagram?" and "How do I document a grey literature search for my PRISMA flow diagram?"

Managing your review with Covidence

For this step of the review, in Covidence you can:

• Document searches in Covidence review settings so all team members can view
• Add keywords from your search to be highlighted in green or red while your team screens articles in your review settings

How a librarian can help with Step 3

When designing and conducting literature searches, a librarian can advise you on :

• How to create a search strategy with Boolean operators, database-specific syntax, subject headings, and appropriate keywords 
• How to apply previously published systematic review search strategies to your current search
• How to test your search strategy's performance 
• How to translate a search strategy from one database's preferred structure to another

The goal of a systematic retrieve is to find all results that are relevant to your topic. Because systematic review searches can be quite extensive and retrieve large numbers of results, an important aspect of systematic searching is limiting the number of irrelevant results that need to be screened. Librarians are experts trained in literature searching and systematic review methodology. Ask us a question or partner with a librarian to save time and improve the quality of your review. Our comparison chart detailing two tiers of partnership provides more information on how librarians can collaborate with and contribute to systematic review teams.

Search Process

• Use controlled vocabulary, if applicable
• Include synonyms/keyword terms
• Choose databases, websites, and/or registries to search
• Translate to other databases
• Search using other methods (e.g. hand searching)
• Validate and peer review the search

Databases can be multidisciplinary or subject specific. Choose the best databases for your research question. Databases index various journals, so in order to be comprehensive, it is important to search multiple databases when conducting a systematic review. Consider searching databases with more diverse or global coverage (i.e., Global Index Medicus) when appropriate. A list of frequently used databases is provided below. You can access UNC Libraries' full listing of databases on the HSL website (arranged alphabetically or by subject ).

Generally speaking, when literature searching, you are not searching the full-text article. Instead, you are searching certain citation data fields, like title, abstract, keyword, controlled vocabulary terms, and more. When developing a literature search, a good place to start is to identify searchable concepts of the research question, and then expand by adding other terms to describe those concepts. Read below for more information and examples on how to develop a literature search, as well as find tips and tricks for developing more comprehensive searches.

Identify search concepts and terms for each

Start by identifying the main concepts of your research question. If unsure, try using a question framework to help identify the main searchable concepts. PICO is one example of a question framework and is used specifically for clinical questions. If your research question doesn't fit into the PICO model well, view other examples of question frameworks and try another!

View our example in PICO format

Question: for patients 65 years and older, does an influenza vaccine reduce the future risk of pneumonia, controlled vocabulary.

Controlled vocabulary is a set of terminology assigned to citations to describe the content of each reference. Searching with controlled vocabulary can improve the relevancy of search results. Many databases assign controlled vocabulary terms to citations, but their naming schema is often specific to each database. For example, the controlled vocabulary system searchable via PubMed is MeSH, or Medical Subject Headings. More information on searching MeSH can be found on the HSL PubMed Ten Tips Legacy Guide .

Note: Controlled vocabulary may be outdated, and some databases allow users to submit requests to update terminology.

View Controlled Vocabulary for our example PICO

As mentioned above, databases with controlled vocabulary often use their own unique system. A listing of controlled vocabulary systems by database is shown below.

Keyword Terms

Not all citations are indexed with controlled vocabulary terms, however, so it is important to combine controlled vocabulary searches with keyword, or text word, searches. 

Authors often write about the same topic in varied ways and it is important to add these terms to your search in order to capture most of the literature. For example, consider these elements when developing a list of keyword terms for each concept:

• American versus British spelling
• hyphenated terms
• quality of life
• satisfaction
• vaccination
• influenza vaccination

There are several resources to consider when searching for synonyms. Scan the results of preliminary searches to identify additional terms. Look for synonyms, word variations, and other possibilities in Wikipedia, other encyclopedias or dictionaries, and databases. For example, PubChem lists additional drug names and chemical compounds.

Display Controlled Vocabulary and Keywords for our example PICO

Combining controlled vocabulary and text words in PubMed would look like this:

"Influenza Vaccines"[Mesh] OR "influenza vaccine" OR "influenza vaccines" OR "flu vaccine" OR "flu vaccines" OR "flu shot" OR "flu shots" OR "influenza virus vaccine" OR "influenza virus vaccines"

Social and cultural norms have been rapidly changing around the world. This has led to changes in the vocabulary used, such as when describing people or populations. Library and research terminology changes more slowly, and therefore can be considered outdated, unacceptable, or overly clinical for use in conversation or writing.

For our example with people 65 years and older, APA Style Guidelines recommend that researchers use terms like “older adults” and “older persons” and forgo terms like “senior citizens” and “elderly” that connote stereotypes. While these are current recommendations, researchers will recognize that terms like “elderly” have previously been used in the literature. Therefore, removing these terms from the search strategy may result in missed relevant articles. 

Research teams need to discuss current and outdated terminology and decide which terms to include in the search to be as comprehensive as possible. The research team or a librarian can search for currently preferred terms in glossaries, dictionaries, published guidelines, and governmental or organizational websites. The University of Michigan Library provides suggested wording to use in the methods section when antiquated, non-standard, exclusionary, or potentially offensive terms are included in the search.

Check the methods sections or supplementary materials of published systematic reviews for search strategies to see what terminology they used. This can help inform your search strategy by using MeSH terms or keywords you may not have thought of. However, be aware that search strategies will differ in their comprehensiveness.

You can also run a preliminary search for your topic, sort the results by Relevance or Best Match, and skim through titles and abstracts to identify terminology from relevant articles that you should include in your search strategy.

Nesting is a term that describes organizing search terms inside parentheses. This is important because, just like their function in math, commands inside a set of parentheses occur first. Parentheses let the database know in which order terms should be combined. 

Always combine terms for a single concept inside a parentheses set. For example: 

( "Influenza Vaccines"[Mesh] OR "influenza vaccine" OR "influenza vaccines" OR "flu vaccine" OR "flu vaccines" OR "flu shot" OR "flu shots" OR "influenza virus vaccine" OR "influenza virus vaccines" )

Additionally, you may nest a subset of terms for a concept inside a larger parentheses set, as seen below. Pay careful attention to the number of parenthesis sets and ensure they are matched, meaning for every open parentheses you also have a closed one.

( "Influenza Vaccines"[Mesh] OR "influenza vaccine" OR "influenza vaccines" OR "flu vaccine" OR "flu vaccines" OR "flu shot" OR "flu shots" OR "influenza virus vaccine" OR "influenza virus vaccines" OR   (( flu OR influenza ) AND ( vaccine OR vaccines OR vaccination OR immunization )))

Boolean operators

Boolean operators are used to combine terms in literature searches. Searches are typically organized using the Boolean operators OR or AND. OR is used to combine search terms for the same concept (i.e., influenza vaccine). AND is used to combine different concepts (i.e., influenza vaccine AND older adults AND pneumonia). An example of how Boolean operators can affect search retrieval is shown below. Using AND to combine the three concepts will only retrieve results where all are present. Using OR to combine the concepts will retrieve results that use all separately or together. It is important to note that, generally speaking, when you are performing a literature search you are only searching the title, abstract, keywords and other citation data. You are not searching the full-text of the articles.

The last major element to consider when building systematic literature searches are field tags. Field tags tell the database exactly where to search. For example, you can use a field tag to tell a database to search for a term in just the title, the title and abstract, and more. Just like with controlled vocabulary, field tag commands are different for every database.

If you do not manually apply field tags to your search, most databases will automatically search in a set of citation data points. Databases may also overwrite your search with algorithms if you do not apply field tags. For systematic review searching, best practice is to apply field tags to each term for reproducibility.

For example:

("Influenza Vaccines"[Mesh] OR "influenza vaccine"[tw] OR "influenza vaccines"[tw] OR "flu vaccine"[tw] OR "flu vaccines"[tw] OR "flu shot"[tw] OR "flu shots"[tw] OR "influenza virus vaccine"[tw] OR "influenza virus vaccines"[tw] OR ((flu[tw] OR influenza[tw]) AND (vaccine[tw] OR vaccines[tw] OR vaccination[tw] OR immunization[tw])))

View field tags for several health databases

For more information about how to use a variety of databases, check out our guides on searching.

• Searching PubMed guide Guide to searching Medline via the PubMed database
• Searching Embase guide Guide to searching Embase via embase.com
• Searching Scopus guide Guide to searching Scopus via scopus.com
• Searching EBSCO Databases guide Guide to searching CINAHL, PsycInfo, Global Health, & other databases via EBSCO

Combining search elements together

Organizational structure of literature searches is very important. Specifically, how terms are grouped (or nested) and combined with Boolean operators will drastically impact search results. These commands tell databases exactly how to combine terms together, and if done incorrectly or inefficiently, search results returned may be too broad or irrelevant.

For example, in PubMed:

(influenza OR flu) AND vaccine is a properly combined search and it produces around 50,000 results.

influenza OR flu AND vaccine is not properly combined.  Databases may read it as everything about influenza OR everything about (flu AND vaccine), which would produce more results than needed.

We recommend one or more of the following:

• put all your synonyms together inside a set of parentheses, then put AND between the closing parenthesis of one set and the opening parenthesis of the next set
• use a separate search box for each set of synonyms
• run each set of synonyms as a separate search, and then combine all your searches
• ask a librarian if your search produces too many or too few results

View the proper way to combine MeSH terms and Keywords for our example PICO

Question: for patients 65 years and older, does an influenza vaccine reduce the future risk of pneumonia , translating search strategies to other databases.

Databases often use their own set of terminology and syntax. When searching multiple databases, you need to adjust the search slightly to retrieve comparable results. Our sections on Controlled Vocabulary and Field Tags have information on how to build searches in different databases.  Resources to help with this process are listed below.

• Polyglot search A tool to translate a PubMed or Ovid search to other databases
• Search Translation Resources (Cornell) A listing of resources for search translation from Cornell University
• Advanced Searching Techniques (King's College London) A collection of advanced searching techniques from King's College London

Other searching methods

Hand searching.

Literature searches can be supplemented by hand searching. One of the most popular ways this is done with systematic reviews is by searching the reference list and citing articles of studies included in the review. Another method is manually browsing key journals in your field to make sure no relevant articles were missed. Other sources that may be considered for hand searching include: clinical trial registries, white papers and other reports, pharmaceutical or other corporate reports, conference proceedings, theses and dissertations, or professional association guidelines.

Searching grey literature

Grey literature typically refers to literature not published in a traditional manner and often not retrievable through large databases and other popular resources. Grey literature should be searched for inclusion in systematic reviews in order to reduce bias and increase thoroughness. There are several databases specific to grey literature that can be searched.

• Open Grey Grey literature for Europe
• OAIster A union catalog of millions of records representing open access resources from collections worldwide
• Grey Matters: a practical tool for searching health-related grey literature (CADTH) From CADTH, the Canadian Agency for Drugs and Technologies in Health, Grey Matters is a practical tool for searching health-related grey literature. The MS Word document covers a grey literature checklist, including national and international health technology assessment (HTA) web sites, drug and device regulatory agencies, clinical trial registries, health economics resources, Canadian health prevalence or incidence databases, and drug formulary web sites.
• Duke Medical Center Library: Searching for Grey Literature A good online compilation of resources by the Duke Medical Center Library.

Systematic review quality is highly dependent on the literature search(es) used to identify studies. To follow best practices for reporting search strategies, as well as increase reproducibility and transparency, document various elements of the literature search for your review. To make this process more clear, a statement and checklist for reporting literature searches has been developed and and can be found below.

• PRISMA-S: Reporting Literature Searches in Systematic Reviews
• Section 4.5 Cochrane Handbook - Documenting and reporting the search process

At a minimum, document and report certain elements, such as databases searched, including name (i.e., Scopus) and platform (i.e. Elsevier), websites, registries, and grey literature searched. In addition, this also may include citation searching and reaching out to experts in the field. Search strategies used in each database or source should be documented, along with any filters or limits, and dates searched. If a search has been updated or was built upon previous work, that should be noted as well. It is also helpful to document which search terms have been tested and decisions made for term inclusion or exclusion by the team. Last, any peer review process should be stated as well as the total number of records identified from each source and how deduplication was handled. 

If you have a librarian on your team who is creating and running the searches, they will handle the search documentation.

You can document search strategies in word processing software you are familiar with like Microsoft Word or Excel, or Google Docs or Sheets. A template, and separate example file, is provided below for convenience. 

• Search Strategy Documentation Template
• Search Strategy Documentation Example

*Some databases like PubMed are being continually updated with new technology and algorithms. This means that searches may retrieve different results than when originally run, even with the same filters, date limits, etc.

When you decide to update a systematic review search, there are two ways of identifying new articles:  

1. rerun the original search strategy without any changes. .

Rerun the original search strategy without making any changes.  Import the results into your citation manager, and remove all articles duplicated from the original set of search results.

2. Rerun the original search strategy and add an entry date filter.

Rerun the original search strategy and add a date filter for when the article was added to the database ( not the publication date).  An entry date filter will find any articles added to the results since you last ran the search, unlike a publication date filter, which would only find more recent articles.

Some examples of entry date filters for articles entered since December 31, 2021 are:

• PubMed:   AND ("2021/12/31"[EDAT] : "3000"[EDAT])
• Embase: AND [31-12-2021]/sd
• CINAHL:   AND EM 20211231-20231231
• PsycInfo: AND RD 20211231-20231231
• Scopus:   AND LOAD-DATE AFT 20211231  

Your PRISMA flow diagram

For more information about updating the PRISMA flow diagram for your systematic review, see the information on filling out a PRISMA flow diagram for review updates on the Step 8: Write the Review page of the guide.

• << Previous: Step 2: Develop a Protocol
• Next: Step 4: Manage Citations >>
• Last Updated: May 16, 2024 3:24 PM
• URL: https://guides.lib.unc.edu/systematic-reviews

Literature Searching

In this guide.

• Introduction
• Steps for searching the literature in PubMed
• Step 1 - Formulate a search question
• Step 2- Identify primary concepts and gather synonyms
• Step 3 - Locate subject headings (MeSH)
• Step 4 - Combine concepts using Boolean operators
• Step 5 - Refine search terms and search in PubMed
• Step 6 - Apply limits

Steps for Searching the Literature

Searching is an iterative process and often requires re-evaluation and testing by adding or changing keywords and the ways they relate to each other. To guide your search development, you can follow the search steps below. For more information on each step, navigate to its matching tab on the right menu. 

1. Formulate a clear, well-defined, answerable search question

Generally, the basic literature search process begins with formulating a clear, well-defined research question. Asking the right research question is essential to creating an effective search. Your research question(s) must be well-defined and answerable. If the question is too broad, your search will yield more information than you can possibly look through.

2. Identify primary concepts and gather synonyms

Your research question will also help identify the primary search concepts. This will allow you to think about how you want the concepts to relate to each other. Since different authors use different terminology to refer to the same concept, you will need to gather synonyms and all the ways authors might express them. However, it is important to balance the terms so that the synonyms do not go beyond the scope of how you've defined them.

3. Locate subject headings (MeSH)

Subject databases like PubMed use 'controlled vocabularies' made up of subject headings that are preassigned to indexed articles that share a similar topic. These subject headings are organized hierarchically within a family tree of broader and narrower concepts. In PubMed and MEDLINE, the subject headings are called Medical Subject Headings (MeSH). By including MeSH terms in your search, you will not have to think about word variations, word endings, plural or singular forms, or synonyms. Some topics or concepts may even have more than one appropriate MeSH term. There are also times when a topic or concept may not have a MeSH term. 

4. Combine concepts using Boolean operators AND/OR

Once you have identified your search concepts, synonyms, and MeSH terms, you'll need to put them together using nesting and Boolean operators (e.g. AND, OR, NOT). Nesting uses parentheses to put search terms into groups. Boolean operators are used to combine similar and different concepts into one query. 

5. Refine search terms and search in PubMed

There are various database search tactics you can use, such as field tags to limit the search to certain fields, quotation marks for phrase searching, and proximity operators to search a number of spaces between terms to refine your search terms. The constructed search string is ready to be pasted into PubMed. 

6. Apply limits (optional)

If you're getting too many results, you can further refine your search results by using limits on the left box of the results page. Limits allow you to narrow your search by a number of facets such as year, journal name, article type, language, age, etc. 

Depending on the nature of the literature review, the complexity and comprehensiveness of the search strategies and the choice of databases can be different. Please contact the Lane Librarians if you have any questions. 

The type of information you gather is influenced by the type of information source or database you select to search. Bibliographic databases contain references to published literature, such as journal articles, conference abstracts, books, reports, government and legal publications, and patents. Literature reviews typically synthesis indexed, peer-reviewed articles (i.e. works that generally represent the latest original research and have undergone rigorous expert screening before publication), and gray literature (i.e. materials not formally published by commercial publishers or peer-reviewed journals). PubMed offers a breadth of health sciences literature and is a good starting point to locate journal articles.

What is PubMed?

PubMed is a free search engine accessing primarily the MEDLINE database of references and abstracts on life sciences and biomedical topics. Available to the public online since 1996, PubMed was developed and is maintained by the  National Center for Biotechnology Information (NCBI) , at the  U.S. National Library of Medicine (NLM) , located at the  National Institutes of Health (NIH) .

MEDLINE is the National Library of Medicine’s (NLM) premier bibliographic database that contains more than 27 million references to journal articles from more than 5,200 worldwide journals in life sciences with a concentration on biomedicine. The Literature Selection Technica Review Committee (LSTRC) reviews and selects journals for MEDLINE based on the research quality and impact of the journals. A distinctive feature of MEDLINE is that the records are indexed with NLM  Medical Subject Headings  (MeSH).

PubMed also contains citations for  PubMed Central (PMC)  articles. PMC is a full-text archive that includes articles from journals reviewed and selected by NLM for archiving (current and historical), as well as individual articles collected for archiving in compliance with funder policies.  PubMed allows users to search keywords in the bibliographic data, but not the full text of the PMC articles.

How to Access PubMed?

To access PubMed, go to the Lane Library homepage and click PubMed in "Top Resources" on the left. This PubMed link is coded with Find Fulltext @ Lane Library Stanford that links you to Lane's full-text articles online. 

• << Previous: Introduction
• Next: Step 1 - Formulate a search question >>
• Last Updated: Jan 9, 2024 10:30 AM
• URL: https://laneguides.stanford.edu/LitSearch

Purdue Online Writing Lab Purdue OWL® College of Liberal Arts

Writing a Literature Review

Welcome to the Purdue OWL

This page is brought to you by the OWL at Purdue University. When printing this page, you must include the entire legal notice.

Copyright ©1995-2018 by The Writing Lab & The OWL at Purdue and Purdue University. All rights reserved. This material may not be published, reproduced, broadcast, rewritten, or redistributed without permission. Use of this site constitutes acceptance of our terms and conditions of fair use.

A literature review is a document or section of a document that collects key sources on a topic and discusses those sources in conversation with each other (also called synthesis ). The lit review is an important genre in many disciplines, not just literature (i.e., the study of works of literature such as novels and plays). When we say “literature review” or refer to “the literature,” we are talking about the research ( scholarship ) in a given field. You will often see the terms “the research,” “the scholarship,” and “the literature” used mostly interchangeably.

Where, when, and why would I write a lit review?

There are a number of different situations where you might write a literature review, each with slightly different expectations; different disciplines, too, have field-specific expectations for what a literature review is and does. For instance, in the humanities, authors might include more overt argumentation and interpretation of source material in their literature reviews, whereas in the sciences, authors are more likely to report study designs and results in their literature reviews; these differences reflect these disciplines’ purposes and conventions in scholarship. You should always look at examples from your own discipline and talk to professors or mentors in your field to be sure you understand your discipline’s conventions, for literature reviews as well as for any other genre.

A literature review can be a part of a research paper or scholarly article, usually falling after the introduction and before the research methods sections. In these cases, the lit review just needs to cover scholarship that is important to the issue you are writing about; sometimes it will also cover key sources that informed your research methodology.

Lit reviews can also be standalone pieces, either as assignments in a class or as publications. In a class, a lit review may be assigned to help students familiarize themselves with a topic and with scholarship in their field, get an idea of the other researchers working on the topic they’re interested in, find gaps in existing research in order to propose new projects, and/or develop a theoretical framework and methodology for later research. As a publication, a lit review usually is meant to help make other scholars’ lives easier by collecting and summarizing, synthesizing, and analyzing existing research on a topic. This can be especially helpful for students or scholars getting into a new research area, or for directing an entire community of scholars toward questions that have not yet been answered.

What are the parts of a lit review?

Most lit reviews use a basic introduction-body-conclusion structure; if your lit review is part of a larger paper, the introduction and conclusion pieces may be just a few sentences while you focus most of your attention on the body. If your lit review is a standalone piece, the introduction and conclusion take up more space and give you a place to discuss your goals, research methods, and conclusions separately from where you discuss the literature itself.

Introduction:

• An introductory paragraph that explains what your working topic and thesis is
• A forecast of key topics or texts that will appear in the review
• Potentially, a description of how you found sources and how you analyzed them for inclusion and discussion in the review (more often found in published, standalone literature reviews than in lit review sections in an article or research paper)
• Summarize and synthesize: Give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: Don’t just paraphrase other researchers – add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically Evaluate: Mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: Use transition words and topic sentence to draw connections, comparisons, and contrasts.

Conclusion:

• Summarize the key findings you have taken from the literature and emphasize their significance
• Connect it back to your primary research question

How should I organize my lit review?

Lit reviews can take many different organizational patterns depending on what you are trying to accomplish with the review. Here are some examples:

• Chronological : The simplest approach is to trace the development of the topic over time, which helps familiarize the audience with the topic (for instance if you are introducing something that is not commonly known in your field). If you choose this strategy, be careful to avoid simply listing and summarizing sources in order. Try to analyze the patterns, turning points, and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred (as mentioned previously, this may not be appropriate in your discipline — check with a teacher or mentor if you’re unsure).
• Thematic : If you have found some recurring central themes that you will continue working with throughout your piece, you can organize your literature review into subsections that address different aspects of the topic. For example, if you are reviewing literature about women and religion, key themes can include the role of women in churches and the religious attitude towards women.
• Qualitative versus quantitative research
• Empirical versus theoretical scholarship
• Divide the research by sociological, historical, or cultural sources
• Theoretical : In many humanities articles, the literature review is the foundation for the theoretical framework. You can use it to discuss various theories, models, and definitions of key concepts. You can argue for the relevance of a specific theoretical approach or combine various theorical concepts to create a framework for your research.

What are some strategies or tips I can use while writing my lit review?

Any lit review is only as good as the research it discusses; make sure your sources are well-chosen and your research is thorough. Don’t be afraid to do more research if you discover a new thread as you’re writing. More info on the research process is available in our "Conducting Research" resources .

As you’re doing your research, create an annotated bibliography ( see our page on the this type of document ). Much of the information used in an annotated bibliography can be used also in a literature review, so you’ll be not only partially drafting your lit review as you research, but also developing your sense of the larger conversation going on among scholars, professionals, and any other stakeholders in your topic.

Usually you will need to synthesize research rather than just summarizing it. This means drawing connections between sources to create a picture of the scholarly conversation on a topic over time. Many student writers struggle to synthesize because they feel they don’t have anything to add to the scholars they are citing; here are some strategies to help you:

• It often helps to remember that the point of these kinds of syntheses is to show your readers how you understand your research, to help them read the rest of your paper.
• Writing teachers often say synthesis is like hosting a dinner party: imagine all your sources are together in a room, discussing your topic. What are they saying to each other?
• Look at the in-text citations in each paragraph. Are you citing just one source for each paragraph? This usually indicates summary only. When you have multiple sources cited in a paragraph, you are more likely to be synthesizing them (not always, but often
• Read more about synthesis here.

The most interesting literature reviews are often written as arguments (again, as mentioned at the beginning of the page, this is discipline-specific and doesn’t work for all situations). Often, the literature review is where you can establish your research as filling a particular gap or as relevant in a particular way. You have some chance to do this in your introduction in an article, but the literature review section gives a more extended opportunity to establish the conversation in the way you would like your readers to see it. You can choose the intellectual lineage you would like to be part of and whose definitions matter most to your thinking (mostly humanities-specific, but this goes for sciences as well). In addressing these points, you argue for your place in the conversation, which tends to make the lit review more compelling than a simple reporting of other sources.

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

• Knowledge Base
• Dissertation
• What is a Literature Review? | Guide, Template, & Examples

What is a Literature Review? | Guide, Template, & Examples

Published on 22 February 2022 by Shona McCombes . Revised on 7 June 2022.

What is a literature review? A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research.

There are five key steps to writing a literature review:

• Search for relevant literature
• Evaluate sources
• Identify themes, debates and gaps
• Outline the structure
• Write your literature review

A good literature review doesn’t just summarise sources – it analyses, synthesises, and critically evaluates to give a clear picture of the state of knowledge on the subject.

Instantly correct all language mistakes in your text

Be assured that you'll submit flawless writing. Upload your document to correct all your mistakes.

Table of contents

Why write a literature review, examples of literature reviews, step 1: search for relevant literature, step 2: evaluate and select sources, step 3: identify themes, debates and gaps, step 4: outline your literature review’s structure, step 5: write your literature review, frequently asked questions about literature reviews, introduction.

• Quick Run-through
• Step 1 & 2

When you write a dissertation or thesis, you will have to conduct a literature review to situate your research within existing knowledge. The literature review gives you a chance to:

• Demonstrate your familiarity with the topic and scholarly context
• Develop a theoretical framework and methodology for your research
• Position yourself in relation to other researchers and theorists
• Show how your dissertation addresses a gap or contributes to a debate

You might also have to write a literature review as a stand-alone assignment. In this case, the purpose is to evaluate the current state of research and demonstrate your knowledge of scholarly debates around a topic.

The content will look slightly different in each case, but the process of conducting a literature review follows the same steps. We’ve written a step-by-step guide that you can follow below.

Prevent plagiarism, run a free check.

Writing literature reviews can be quite challenging! A good starting point could be to look at some examples, depending on what kind of literature review you’d like to write.

• Example literature review #1: “Why Do People Migrate? A Review of the Theoretical Literature” ( Theoretical literature review about the development of economic migration theory from the 1950s to today.)
• Example literature review #2: “Literature review as a research methodology: An overview and guidelines” ( Methodological literature review about interdisciplinary knowledge acquisition and production.)
• Example literature review #3: “The Use of Technology in English Language Learning: A Literature Review” ( Thematic literature review about the effects of technology on language acquisition.)
• Example literature review #4: “Learners’ Listening Comprehension Difficulties in English Language Learning: A Literature Review” ( Chronological literature review about how the concept of listening skills has changed over time.)

You can also check out our templates with literature review examples and sample outlines at the links below.

Download Word doc Download Google doc

Before you begin searching for literature, you need a clearly defined topic .

If you are writing the literature review section of a dissertation or research paper, you will search for literature related to your research objectives and questions .

If you are writing a literature review as a stand-alone assignment, you will have to choose a focus and develop a central question to direct your search. Unlike a dissertation research question, this question has to be answerable without collecting original data. You should be able to answer it based only on a review of existing publications.

Make a list of keywords

Start by creating a list of keywords related to your research topic. Include each of the key concepts or variables you’re interested in, and list any synonyms and related terms. You can add to this list if you discover new keywords in the process of your literature search.

• Social media, Facebook, Instagram, Twitter, Snapchat, TikTok
• Body image, self-perception, self-esteem, mental health
• Generation Z, teenagers, adolescents, youth

Search for relevant sources

Use your keywords to begin searching for sources. Some databases to search for journals and articles include:

• Your university’s library catalogue
• Google Scholar
• Project Muse (humanities and social sciences)
• Medline (life sciences and biomedicine)
• EconLit (economics)
• Inspec (physics, engineering and computer science)

You can use boolean operators to help narrow down your search:

Read the abstract to find out whether an article is relevant to your question. When you find a useful book or article, you can check the bibliography to find other relevant sources.

To identify the most important publications on your topic, take note of recurring citations. If the same authors, books or articles keep appearing in your reading, make sure to seek them out.

You probably won’t be able to read absolutely everything that has been written on the topic – you’ll have to evaluate which sources are most relevant to your questions.

For each publication, ask yourself:

• What question or problem is the author addressing?
• What are the key concepts and how are they defined?
• What are the key theories, models and methods? Does the research use established frameworks or take an innovative approach?
• What are the results and conclusions of the study?
• How does the publication relate to other literature in the field? Does it confirm, add to, or challenge established knowledge?
• How does the publication contribute to your understanding of the topic? What are its key insights and arguments?
• What are the strengths and weaknesses of the research?

Make sure the sources you use are credible, and make sure you read any landmark studies and major theories in your field of research.

You can find out how many times an article has been cited on Google Scholar – a high citation count means the article has been influential in the field, and should certainly be included in your literature review.

The scope of your review will depend on your topic and discipline: in the sciences you usually only review recent literature, but in the humanities you might take a long historical perspective (for example, to trace how a concept has changed in meaning over time).

Remember that you can use our template to summarise and evaluate sources you’re thinking about using!

Take notes and cite your sources

As you read, you should also begin the writing process. Take notes that you can later incorporate into the text of your literature review.

It’s important to keep track of your sources with references to avoid plagiarism . It can be helpful to make an annotated bibliography, where you compile full reference information and write a paragraph of summary and analysis for each source. This helps you remember what you read and saves time later in the process.

You can use our free APA Reference Generator for quick, correct, consistent citations.

To begin organising your literature review’s argument and structure, you need to understand the connections and relationships between the sources you’ve read. Based on your reading and notes, you can look for:

• Trends and patterns (in theory, method or results): do certain approaches become more or less popular over time?
• Themes: what questions or concepts recur across the literature?
• Debates, conflicts and contradictions: where do sources disagree?
• Pivotal publications: are there any influential theories or studies that changed the direction of the field?
• Gaps: what is missing from the literature? Are there weaknesses that need to be addressed?

This step will help you work out the structure of your literature review and (if applicable) show how your own research will contribute to existing knowledge.

• Most research has focused on young women.
• There is an increasing interest in the visual aspects of social media.
• But there is still a lack of robust research on highly-visual platforms like Instagram and Snapchat – this is a gap that you could address in your own research.

There are various approaches to organising the body of a literature review. You should have a rough idea of your strategy before you start writing.

Depending on the length of your literature review, you can combine several of these strategies (for example, your overall structure might be thematic, but each theme is discussed chronologically).

Chronological

The simplest approach is to trace the development of the topic over time. However, if you choose this strategy, be careful to avoid simply listing and summarising sources in order.

Try to analyse patterns, turning points and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred.

If you have found some recurring central themes, you can organise your literature review into subsections that address different aspects of the topic.

For example, if you are reviewing literature about inequalities in migrant health outcomes, key themes might include healthcare policy, language barriers, cultural attitudes, legal status, and economic access.

Methodological

If you draw your sources from different disciplines or fields that use a variety of research methods , you might want to compare the results and conclusions that emerge from different approaches. For example:

• Look at what results have emerged in qualitative versus quantitative research
• Discuss how the topic has been approached by empirical versus theoretical scholarship
• Divide the literature into sociological, historical, and cultural sources

Theoretical

A literature review is often the foundation for a theoretical framework . You can use it to discuss various theories, models, and definitions of key concepts.

You might argue for the relevance of a specific theoretical approach, or combine various theoretical concepts to create a framework for your research.

Like any other academic text, your literature review should have an introduction , a main body, and a conclusion . What you include in each depends on the objective of your literature review.

The introduction should clearly establish the focus and purpose of the literature review.

If you are writing the literature review as part of your dissertation or thesis, reiterate your central problem or research question and give a brief summary of the scholarly context. You can emphasise the timeliness of the topic (“many recent studies have focused on the problem of x”) or highlight a gap in the literature (“while there has been much research on x, few researchers have taken y into consideration”).

Depending on the length of your literature review, you might want to divide the body into subsections. You can use a subheading for each theme, time period, or methodological approach.

As you write, make sure to follow these tips:

• Summarise and synthesise: give an overview of the main points of each source and combine them into a coherent whole.
• Analyse and interpret: don’t just paraphrase other researchers – add your own interpretations, discussing the significance of findings in relation to the literature as a whole.
• Critically evaluate: mention the strengths and weaknesses of your sources.
• Write in well-structured paragraphs: use transitions and topic sentences to draw connections, comparisons and contrasts.

In the conclusion, you should summarise the key findings you have taken from the literature and emphasise their significance.

If the literature review is part of your dissertation or thesis, reiterate how your research addresses gaps and contributes new knowledge, or discuss how you have drawn on existing theories and methods to build a framework for your research. This can lead directly into your methodology section.

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a dissertation , thesis, research paper , or proposal .

There are several reasons to conduct a literature review at the beginning of a research project:

• To familiarise yourself with the current state of knowledge on your topic
• To ensure that you’re not just repeating what others have already done
• To identify gaps in knowledge and unresolved problems that your research can address
• To develop your theoretical framework and methodology
• To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

The literature review usually comes near the beginning of your  dissertation . After the introduction , it grounds your research in a scholarly field and leads directly to your theoretical framework or methodology .

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

McCombes, S. (2022, June 07). What is a Literature Review? | Guide, Template, & Examples. Scribbr. Retrieved 31 May 2024, from https://www.scribbr.co.uk/thesis-dissertation/literature-review/

Is this article helpful?

Shona McCombes

Other students also liked, how to write a dissertation proposal | a step-by-step guide, what is a theoretical framework | a step-by-step guide, what is a research methodology | steps & tips.

Reference management. Clean and simple.

How to write a literature review in 6 steps

What is a literature review?

How to write a literature review, 1. determine the purpose of your literature review, 2. do an extensive search, 3. evaluate and select literature, 4. analyze the literature, 5. plan the structure of your literature review, 6. write your literature review, other resources to help you write a successful literature review, frequently asked questions about writing a literature review, related articles.

A literature review is an assessment of the sources in a chosen topic of research.

A good literature review does not just summarize sources. It analyzes the state of the field on a given topic and creates a scholarly foundation for you to make your own intervention. It demonstrates to your readers how your research fits within a larger field of study.

In a thesis, a literature review is part of the introduction, but it can also be a separate section. In research papers, a literature review may have its own section or it may be integrated into the introduction, depending on the field.

➡️ Our guide on what is a literature review covers additional basics about literature reviews.

• Identify the main purpose of the literature review.
• Do extensive research.
• Evaluate and select relevant sources.
• Analyze the sources.
• Plan a structure.
• Write the review.

In this section, we review each step of the process of creating a literature review.

In the first step, make sure you know specifically what the assignment is and what form your literature review should take. Read your assignment carefully and seek clarification from your professor or instructor if needed. You should be able to answer the following questions:

• How many sources do I need to include?
• What types of sources should I review?
• Should I evaluate the sources?
• Should I summarize, synthesize or critique sources?
• Do I need to provide any definitions or background information?

In addition to that, be aware that the narrower your topic, the easier it will be to limit the number of sources you need to read in order to get a good overview of the topic.

Now you need to find out what has been written on the topic and search for literature related to your research topic. Make sure to select appropriate source material, which means using academic or scholarly sources , including books, reports, journal articles , government documents and web resources.

➡️ If you’re unsure about how to tell if a source is scholarly, take a look at our guide on how to identify a scholarly source .

Come up with a list of relevant keywords and then start your search with your institution's library catalog, and extend it to other useful databases and academic search engines like:

• Google Scholar
• Science.gov

➡️ Our guide on how to collect data for your thesis might be helpful at this stage of your research as well as the top list of academic search engines .

Once you find a useful article, check out the reference list. It should provide you with even more relevant sources. Also, keep a note of the:

• authors' names
• page numbers

Keeping track of the bibliographic information for each source will save you time when you’re ready to create citations. You could also use a reference manager like Paperpile to automatically save, manage, and cite your references.

Read the literature. You will most likely not be able to read absolutely everything that is out there on the topic. Therefore, read the abstract first to determine whether the rest of the source is worth your time. If the source is relevant for your topic:

• Read it critically.
• Look for the main arguments.
• Take notes as you read.
• Organize your notes using a table, mind map, or other technique.

Now you are ready to analyze the literature you have gathered. While your are working on your analysis, you should ask the following questions:

• What are the key terms, concepts and problems addressed by the author?
• How is this source relevant for my specific topic?
• How is the article structured? What are the major trends and findings?
• What are the conclusions of the study?
• How are the results presented? Is the source credible?
• When comparing different sources, how do they relate to each other? What are the similarities, what are the differences?
• Does the study help me understand the topic better?
• Are there any gaps in the research that need to be filled? How can I further my research as a result of the review?

Tip: Decide on the structure of your literature review before you start writing.

There are various ways to organize your literature review:

• Chronological method : Writing in the chronological method means you are presenting the materials according to when they were published. Follow this approach only if a clear path of research can be identified.
• Thematic review : A thematic review of literature is organized around a topic or issue, rather than the progression of time.
• Publication-based : You can order your sources by publication, if the way you present the order of your sources demonstrates a more important trend. This is the case when a progression revealed from study to study and the practices of researchers have changed and adapted due to the new revelations.
• Methodological approach : A methodological approach focuses on the methods used by the researcher. If you have used sources from different disciplines that use a variety of research methods, you might want to compare the results in light of the different methods and discuss how the topic has been approached from different sides.

Regardless of the structure you chose, a review should always include the following three sections:

• An introduction, which should give the reader an outline of why you are writing the review and explain the relevance of the topic.
• A body, which divides your literature review into different sections. Write in well-structured paragraphs, use transitions and topic sentences and critically analyze each source for how it contributes to the themes you are researching.
• A conclusion , which summarizes the key findings, the main agreements and disagreements in the literature, your overall perspective, and any gaps or areas for further research.

➡️ If your literature review is part of a longer paper, visit our guide on what is a research paper for additional tips.

➡️ UNC writing center: Literature reviews

➡️ How to write a literature review in 3 steps

➡️ How to write a literature review in 30 minutes or less

The goal of a literature review is to asses the state of the field on a given topic in preparation for making an intervention.

A literature review should have its own independent section. You should indicate clearly in the table of contents where it can be found, and address this section as “Literature Review.”

There is no set amount of words for a literature review; the length depends on the research. If you are working with a large amount of sources, then it will be long. If your paper does not depend entirely on references, then it will be short.

Most research papers include a literature review. By assessing the available sources in your field of research, you will be able to make a more confident argument about the topic.

Literature reviews are most commonly found in theses and dissertations. However, you find them in research papers as well.

Harvey Cushing/John Hay Whitney Medical Library

• Collections
• Research Help

YSN Doctoral Programs: Steps in Conducting a Literature Review

• Biomedical Databases
• Global (Public Health) Databases
• Soc. Sci., History, and Law Databases
• Grey Literature
• Trials Registers
• Data and Statistics
• Public Policy
• Google Tips
• Recommended Books
• Steps in Conducting a Literature Review

What is a literature review?

A literature review is an integrated analysis -- not just a summary-- of scholarly writings and other relevant evidence related directly to your research question.  That is, it represents a synthesis of the evidence that provides background information on your topic and shows a association between the evidence and your research question.

A literature review may be a stand alone work or the introduction to a larger research paper, depending on the assignment.  Rely heavily on the guidelines your instructor has given you.

Why is it important?

A literature review is important because it:

• Explains the background of research on a topic.
• Demonstrates why a topic is significant to a subject area.
• Discovers relationships between research studies/ideas.
• Identifies major themes, concepts, and researchers on a topic.
• Identifies critical gaps and points of disagreement.
• Discusses further research questions that logically come out of the previous studies.

APA7 Style resources

APA Style Blog - for those harder to find answers

1. Choose a topic. Define your research question.

Your literature review should be guided by your central research question.  The literature represents background and research developments related to a specific research question, interpreted and analyzed by you in a synthesized way.

• Make sure your research question is not too broad or too narrow.  Is it manageable?
• Begin writing down terms that are related to your question. These will be useful for searches later.
• If you have the opportunity, discuss your topic with your professor and your class mates.

2. Decide on the scope of your review

How many studies do you need to look at? How comprehensive should it be? How many years should it cover? 

• This may depend on your assignment.  How many sources does the assignment require?

3. Select the databases you will use to conduct your searches.

Make a list of the databases you will search. 

Where to find databases:

• use the tabs on this guide
• Find other databases in the Nursing Information Resources web page
• More on the Medical Library web page
• ... and more on the Yale University Library web page

4. Conduct your searches to find the evidence. Keep track of your searches.

• Use the key words in your question, as well as synonyms for those words, as terms in your search. Use the database tutorials for help.
• Save the searches in the databases. This saves time when you want to redo, or modify, the searches. It is also helpful to use as a guide is the searches are not finding any useful results.
• Review the abstracts of research studies carefully. This will save you time.
• Use the bibliographies and references of research studies you find to locate others.
• Check with your professor, or a subject expert in the field, if you are missing any key works in the field.
• Ask your librarian for help at any time.
• Use a citation manager, such as EndNote as the repository for your citations. See the EndNote tutorials for help.

Review the literature

Some questions to help you analyze the research:

• What was the research question of the study you are reviewing? What were the authors trying to discover?
• Was the research funded by a source that could influence the findings?
• What were the research methodologies? Analyze its literature review, the samples and variables used, the results, and the conclusions.
• Does the research seem to be complete? Could it have been conducted more soundly? What further questions does it raise?
• If there are conflicting studies, why do you think that is?
• How are the authors viewed in the field? Has this study been cited? If so, how has it been analyzed?

Tips: 

• Review the abstracts carefully.  
• Keep careful notes so that you may track your thought processes during the research process.
• Create a matrix of the studies for easy analysis, and synthesis, across all of the studies.
• << Previous: Recommended Books
• Last Updated: Jan 4, 2024 10:52 AM
• URL: https://guides.library.yale.edu/YSNDoctoral

How to: systematic review literature searching

Searching literature is one of the most important elements of a systematic review. A well planned search strategy in the right databases ensures you have a robust list of results to whittle down as part of your PRISMA workflow. We’ve answered some of the common questions we get asked about searching literature as part of a systematic review. Take a look – have we missed anything?

How do you do a literature search for a systematic review?

The literature search element of a systematic review is the next step after you’ve created a well-defined question that the systematic review itself is trying to answer.

Best-practice is to perform your literature search across at least three separate databases. The two most common are Embase and MEDLINE, with the third (and fourth and fifth!) varying depending on the subject area of your systematic review. Additional databases can include sources of grey literature like ClinicalTrials.gov or databases of systematic reviews such as Cochrane (with a lot of Cochrane content being available in both Embase and MEDLINE).

Your literature search needs to use a well-constructed and thorough search strategy. Following a framework like PICO (Problem, Intervention, Comparison, Outcome) or SPIDER (Sample, Phenomenon of Interest, Design, Evaluation, Research type) can help you ensure you’re not missing anything. It can also help to use a database search platform that allows you to save and edit your searches. This can make it easier to demonstrate your search strategy is truly reproducible and exhaustive.

Your search strategy will also benefit from using Embase and MEDLINE’s thesauri to refine your search keywords. Using medical synonyms (like those built into  Dialog ) can also be helpful, both for saving your time and ensuring you’re not missing out on any common synonyms for your search terms.

And as a final point, make sure you’re able to refine your results consistently across all the databases you’re using. This is easier if you’re using a single platform to search multiple databases as you can limit date ranges, publications, author, etc. with a single click, rather than trying to do this consistently across individual databases.

What databases are used for systematic reviews?

The databases you choose for your systematic review literature search should be as extensive as possible to avoid any potential publication bias. At the very least, most people agree that MEDLINE and Embase have to be included within your systematic review literature search. You should also choose specific databases related to the topic of your systematic review question (such as PsycINFO for psychological-focused studies or ESPICOM  for  systematic reviews looking at medical devices). Cochrane also recommends you include its Central Register of Controlled Trials (CENTRAL) database in your search if you want your systematic review to eventually appear in CENTRAL.

It is also a good idea to use grey literature within your list of sources. Some useful databases for grey literature and pre-print content include the U.S. National Library of Medicine’s ClinicalTrials.gov, ProQuest Dissertations and Theses and the Morressier database of conference posters.

What is PRISMA for systematic reviews?

PRISMA stands for Preferred Reporting Items for Systematic Reviews and Meta-Analyses and is a set of guidelines designed to improve the quality of systematic reviews.

PRISMA includes a 27 item checklist that covers everything from the report title and abstract to declaring competing interests and support (you can access it on PRISMA’s website  here ). There is also a PRISMA workflow that can help narrow down the results of your systematic review literature searches in an objective and repeatable way (access the flowchart on PRISMA’s website  here ).

How many databases should be searched for a systematic review?

At least three databases should be used for systematic review literature searches. As explained above, the most common are Embase and MEDLINE, with the third database being chosen based on the subject area of the systematic review.

Additional databases can be used to provide sources of grey literature, such as pre-print working papers, dissertations and theses and conference papers and posters.

What is grey literature for a systematic review?

Grey literature is any content that is considered unpublished in the sense it doesn’t appear through traditional publishing channels (i.e. peer-reviewed databases). As a result grey literature covers a very wide range of content types and sources and can include:

• Conference posters
• Conference abstracts and proceedings
• Pre-print / working papers
• Dissertations and theses
• Clinical studies

Can you include grey literature in a systematic review?

Grey literature can be a very important resource in systematic reviews as it can provide extra evidence and sources outside of the main databases and journals. While it can be harder to find grey literature (unless you’re searching databases that specifically contain grey literature content), it can help a systematic review be more balanced and thorough.

The main drawback with grey literature is ensuring the quality and objectivity of the content, as it won’t necessarily have been reviewed in the same way as content from the main publication sources.

How do you find grey literature for a systematic review?

There are a number of databases that contain grey literature, including ProQuest Dissertations & Theses, Publicly Available Content, ClinicalTrials.gov and Morressier. If you’re doing a systematic review literature search on Dialog these databases can be added alongside MEDLINE and Embase as part of your search process.

How do you develop a search term for a systematic review?

To develop a search strategy for a systematic review, we recommend the following approach:

• i.e. Can antibiotics help alleviate the symptoms of a sore throat?
• Choose the most appropriate databases (MEDLINE, Embase and SciSearch, for example) and identify the most cost-effective and efficient way to search and access this content (such as a database search platform like Dialog)
• P opulation/ P roblem  –  sore throat
• I ntervention  –  antibiotics
• C omparison  –  e.g. anti-inflammatories, placebo
• O utcomes  –  alleviation, therapeutic effect
• Then decide what content you will include and exclude
• Use thesauri to find preferred terms and explosions in MEDLINE and Embase
• Use thesaurus subheadings to refine terms in MEDLINE and Embase
• Use medical synonyms to increase recall of free text terms

Related posts

Journal citation reports 2024 preview: unified rankings for more inclusive journal assessment.

Beyond overall survival: Time to agree on the value of alternative oncology endpoints?

Introducing the Clarivate Academic AI Platform

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• PLoS Comput Biol
• v.9(7); 2013 Jul

Ten Simple Rules for Writing a Literature Review

Marco pautasso.

1 Centre for Functional and Evolutionary Ecology (CEFE), CNRS, Montpellier, France

2 Centre for Biodiversity Synthesis and Analysis (CESAB), FRB, Aix-en-Provence, France

Literature reviews are in great demand in most scientific fields. Their need stems from the ever-increasing output of scientific publications [1] . For example, compared to 1991, in 2008 three, eight, and forty times more papers were indexed in Web of Science on malaria, obesity, and biodiversity, respectively [2] . Given such mountains of papers, scientists cannot be expected to examine in detail every single new paper relevant to their interests [3] . Thus, it is both advantageous and necessary to rely on regular summaries of the recent literature. Although recognition for scientists mainly comes from primary research, timely literature reviews can lead to new synthetic insights and are often widely read [4] . For such summaries to be useful, however, they need to be compiled in a professional way [5] .

When starting from scratch, reviewing the literature can require a titanic amount of work. That is why researchers who have spent their career working on a certain research issue are in a perfect position to review that literature. Some graduate schools are now offering courses in reviewing the literature, given that most research students start their project by producing an overview of what has already been done on their research issue [6] . However, it is likely that most scientists have not thought in detail about how to approach and carry out a literature review.

Reviewing the literature requires the ability to juggle multiple tasks, from finding and evaluating relevant material to synthesising information from various sources, from critical thinking to paraphrasing, evaluating, and citation skills [7] . In this contribution, I share ten simple rules I learned working on about 25 literature reviews as a PhD and postdoctoral student. Ideas and insights also come from discussions with coauthors and colleagues, as well as feedback from reviewers and editors.

Rule 1: Define a Topic and Audience

How to choose which topic to review? There are so many issues in contemporary science that you could spend a lifetime of attending conferences and reading the literature just pondering what to review. On the one hand, if you take several years to choose, several other people may have had the same idea in the meantime. On the other hand, only a well-considered topic is likely to lead to a brilliant literature review [8] . The topic must at least be:

• interesting to you (ideally, you should have come across a series of recent papers related to your line of work that call for a critical summary),
• an important aspect of the field (so that many readers will be interested in the review and there will be enough material to write it), and
• a well-defined issue (otherwise you could potentially include thousands of publications, which would make the review unhelpful).

Ideas for potential reviews may come from papers providing lists of key research questions to be answered [9] , but also from serendipitous moments during desultory reading and discussions. In addition to choosing your topic, you should also select a target audience. In many cases, the topic (e.g., web services in computational biology) will automatically define an audience (e.g., computational biologists), but that same topic may also be of interest to neighbouring fields (e.g., computer science, biology, etc.).

Rule 2: Search and Re-search the Literature

After having chosen your topic and audience, start by checking the literature and downloading relevant papers. Five pieces of advice here:

• keep track of the search items you use (so that your search can be replicated [10] ),
• keep a list of papers whose pdfs you cannot access immediately (so as to retrieve them later with alternative strategies),
• use a paper management system (e.g., Mendeley, Papers, Qiqqa, Sente),
• define early in the process some criteria for exclusion of irrelevant papers (these criteria can then be described in the review to help define its scope), and
• do not just look for research papers in the area you wish to review, but also seek previous reviews.

The chances are high that someone will already have published a literature review ( Figure 1 ), if not exactly on the issue you are planning to tackle, at least on a related topic. If there are already a few or several reviews of the literature on your issue, my advice is not to give up, but to carry on with your own literature review,

The bottom-right situation (many literature reviews but few research papers) is not just a theoretical situation; it applies, for example, to the study of the impacts of climate change on plant diseases, where there appear to be more literature reviews than research studies [33] .

• discussing in your review the approaches, limitations, and conclusions of past reviews,
• trying to find a new angle that has not been covered adequately in the previous reviews, and
• incorporating new material that has inevitably accumulated since their appearance.

When searching the literature for pertinent papers and reviews, the usual rules apply:

• be thorough,
• use different keywords and database sources (e.g., DBLP, Google Scholar, ISI Proceedings, JSTOR Search, Medline, Scopus, Web of Science), and
• look at who has cited past relevant papers and book chapters.

Rule 3: Take Notes While Reading

If you read the papers first, and only afterwards start writing the review, you will need a very good memory to remember who wrote what, and what your impressions and associations were while reading each single paper. My advice is, while reading, to start writing down interesting pieces of information, insights about how to organize the review, and thoughts on what to write. This way, by the time you have read the literature you selected, you will already have a rough draft of the review.

Of course, this draft will still need much rewriting, restructuring, and rethinking to obtain a text with a coherent argument [11] , but you will have avoided the danger posed by staring at a blank document. Be careful when taking notes to use quotation marks if you are provisionally copying verbatim from the literature. It is advisable then to reformulate such quotes with your own words in the final draft. It is important to be careful in noting the references already at this stage, so as to avoid misattributions. Using referencing software from the very beginning of your endeavour will save you time.

Rule 4: Choose the Type of Review You Wish to Write

After having taken notes while reading the literature, you will have a rough idea of the amount of material available for the review. This is probably a good time to decide whether to go for a mini- or a full review. Some journals are now favouring the publication of rather short reviews focusing on the last few years, with a limit on the number of words and citations. A mini-review is not necessarily a minor review: it may well attract more attention from busy readers, although it will inevitably simplify some issues and leave out some relevant material due to space limitations. A full review will have the advantage of more freedom to cover in detail the complexities of a particular scientific development, but may then be left in the pile of the very important papers “to be read” by readers with little time to spare for major monographs.

There is probably a continuum between mini- and full reviews. The same point applies to the dichotomy of descriptive vs. integrative reviews. While descriptive reviews focus on the methodology, findings, and interpretation of each reviewed study, integrative reviews attempt to find common ideas and concepts from the reviewed material [12] . A similar distinction exists between narrative and systematic reviews: while narrative reviews are qualitative, systematic reviews attempt to test a hypothesis based on the published evidence, which is gathered using a predefined protocol to reduce bias [13] , [14] . When systematic reviews analyse quantitative results in a quantitative way, they become meta-analyses. The choice between different review types will have to be made on a case-by-case basis, depending not just on the nature of the material found and the preferences of the target journal(s), but also on the time available to write the review and the number of coauthors [15] .

Rule 5: Keep the Review Focused, but Make It of Broad Interest

Whether your plan is to write a mini- or a full review, it is good advice to keep it focused 16 , 17 . Including material just for the sake of it can easily lead to reviews that are trying to do too many things at once. The need to keep a review focused can be problematic for interdisciplinary reviews, where the aim is to bridge the gap between fields [18] . If you are writing a review on, for example, how epidemiological approaches are used in modelling the spread of ideas, you may be inclined to include material from both parent fields, epidemiology and the study of cultural diffusion. This may be necessary to some extent, but in this case a focused review would only deal in detail with those studies at the interface between epidemiology and the spread of ideas.

While focus is an important feature of a successful review, this requirement has to be balanced with the need to make the review relevant to a broad audience. This square may be circled by discussing the wider implications of the reviewed topic for other disciplines.

Rule 6: Be Critical and Consistent

Reviewing the literature is not stamp collecting. A good review does not just summarize the literature, but discusses it critically, identifies methodological problems, and points out research gaps [19] . After having read a review of the literature, a reader should have a rough idea of:

• the major achievements in the reviewed field,
• the main areas of debate, and
• the outstanding research questions.

It is challenging to achieve a successful review on all these fronts. A solution can be to involve a set of complementary coauthors: some people are excellent at mapping what has been achieved, some others are very good at identifying dark clouds on the horizon, and some have instead a knack at predicting where solutions are going to come from. If your journal club has exactly this sort of team, then you should definitely write a review of the literature! In addition to critical thinking, a literature review needs consistency, for example in the choice of passive vs. active voice and present vs. past tense.

Rule 7: Find a Logical Structure

Like a well-baked cake, a good review has a number of telling features: it is worth the reader's time, timely, systematic, well written, focused, and critical. It also needs a good structure. With reviews, the usual subdivision of research papers into introduction, methods, results, and discussion does not work or is rarely used. However, a general introduction of the context and, toward the end, a recapitulation of the main points covered and take-home messages make sense also in the case of reviews. For systematic reviews, there is a trend towards including information about how the literature was searched (database, keywords, time limits) [20] .

How can you organize the flow of the main body of the review so that the reader will be drawn into and guided through it? It is generally helpful to draw a conceptual scheme of the review, e.g., with mind-mapping techniques. Such diagrams can help recognize a logical way to order and link the various sections of a review [21] . This is the case not just at the writing stage, but also for readers if the diagram is included in the review as a figure. A careful selection of diagrams and figures relevant to the reviewed topic can be very helpful to structure the text too [22] .

Rule 8: Make Use of Feedback

Reviews of the literature are normally peer-reviewed in the same way as research papers, and rightly so [23] . As a rule, incorporating feedback from reviewers greatly helps improve a review draft. Having read the review with a fresh mind, reviewers may spot inaccuracies, inconsistencies, and ambiguities that had not been noticed by the writers due to rereading the typescript too many times. It is however advisable to reread the draft one more time before submission, as a last-minute correction of typos, leaps, and muddled sentences may enable the reviewers to focus on providing advice on the content rather than the form.

Feedback is vital to writing a good review, and should be sought from a variety of colleagues, so as to obtain a diversity of views on the draft. This may lead in some cases to conflicting views on the merits of the paper, and on how to improve it, but such a situation is better than the absence of feedback. A diversity of feedback perspectives on a literature review can help identify where the consensus view stands in the landscape of the current scientific understanding of an issue [24] .

Rule 9: Include Your Own Relevant Research, but Be Objective

In many cases, reviewers of the literature will have published studies relevant to the review they are writing. This could create a conflict of interest: how can reviewers report objectively on their own work [25] ? Some scientists may be overly enthusiastic about what they have published, and thus risk giving too much importance to their own findings in the review. However, bias could also occur in the other direction: some scientists may be unduly dismissive of their own achievements, so that they will tend to downplay their contribution (if any) to a field when reviewing it.

In general, a review of the literature should neither be a public relations brochure nor an exercise in competitive self-denial. If a reviewer is up to the job of producing a well-organized and methodical review, which flows well and provides a service to the readership, then it should be possible to be objective in reviewing one's own relevant findings. In reviews written by multiple authors, this may be achieved by assigning the review of the results of a coauthor to different coauthors.

Rule 10: Be Up-to-Date, but Do Not Forget Older Studies

Given the progressive acceleration in the publication of scientific papers, today's reviews of the literature need awareness not just of the overall direction and achievements of a field of inquiry, but also of the latest studies, so as not to become out-of-date before they have been published. Ideally, a literature review should not identify as a major research gap an issue that has just been addressed in a series of papers in press (the same applies, of course, to older, overlooked studies (“sleeping beauties” [26] )). This implies that literature reviewers would do well to keep an eye on electronic lists of papers in press, given that it can take months before these appear in scientific databases. Some reviews declare that they have scanned the literature up to a certain point in time, but given that peer review can be a rather lengthy process, a full search for newly appeared literature at the revision stage may be worthwhile. Assessing the contribution of papers that have just appeared is particularly challenging, because there is little perspective with which to gauge their significance and impact on further research and society.

Inevitably, new papers on the reviewed topic (including independently written literature reviews) will appear from all quarters after the review has been published, so that there may soon be the need for an updated review. But this is the nature of science [27] – [32] . I wish everybody good luck with writing a review of the literature.

Acknowledgments

Many thanks to M. Barbosa, K. Dehnen-Schmutz, T. Döring, D. Fontaneto, M. Garbelotto, O. Holdenrieder, M. Jeger, D. Lonsdale, A. MacLeod, P. Mills, M. Moslonka-Lefebvre, G. Stancanelli, P. Weisberg, and X. Xu for insights and discussions, and to P. Bourne, T. Matoni, and D. Smith for helpful comments on a previous draft.

Funding Statement

This work was funded by the French Foundation for Research on Biodiversity (FRB) through its Centre for Synthesis and Analysis of Biodiversity data (CESAB), as part of the NETSEED research project. The funders had no role in the preparation of the manuscript.

Your web browser is outdated and may be insecure

The RCN recommends using an updated browser such as Microsoft Edge or Google Chrome

How to undertake a literature search

Introduction.

Undertaking a literature search can be a daunting prospect. By breaking the exercise down into smaller steps, you can make the process more manageable. The following ten steps will help you complete the task from identifying key concepts to choosing databases for your search and saving your results and search strategy. It discusses each of the steps in a little more detail with examples and suggestions of where to get help.

There are ten steps to undertaking a literature search which we'll take you through below:

🎬 - Indicates a video is available with more information.

Please click on the boxes below to get a bit more detail on each step.

First, write out your title and check that you understand all the terms. Look up the meaning of any you don’t understand. An online dictionary or medical encyclopaedia may help with this.

If your search is for a dissertation, you may need to choose your own research question. In this case, you will need to consider whether there is likely to be enough research on your topic. Alternatively, if your topic is too broad, you could be overwhelmed by the number of references.

One way of checking how much is written on your topic is to use Library Search. Most libraries offer a Library Search or discovery tool. It provides a quick search across all the library’s holdings. You can also limit your search by date or type of document. If you just need a few references to help you write an essay, Library Search may be helpful. It also gives quick access to full text items.

Next, you need to identify your key concepts. One way to do this is to look at your title and identify the most important words. Ignore words that tell you what to do with the information you find eg evaluate, assess, compare, as these are not generally used as search terms. In the example below, key concepts have been highlighted:

Evaluate the effectiveness of a mindfulness intervention on the health-related quality of life of rheumatoid arthritis patients

Another way to do this is to break down your title using the PEO framework:

P = Population    E = Exposure    O = Outcome 

This works well where there is no comparison between two types of treatment or intervention.

In our example:

P = rheumatoid arthritis patients

E = mindfulness

O =  health related quality of life

Other question formats are available such as PICO or SPIDER

Tip: Not all search topics will include every element of PICO – some include fewer items.

Once you have identified the key concepts, it’s important to think of any other terms or phrases that might have a very similar meaning. Including such synonyms will make your search as thorough as possible. For example, if your topic is looking for articles on Staff attitudes , you might also use the terms:

• Staff perceptions 
• Staff opinions
• Stereotyping
• Labelling 

If the database you are using has a list of subject heading s , this may help you to find the most appropriate term for your subject. Some databases provide definitions for terms used in the database and may suggest related terms.

A comprehensive search will usually include both subject headings from databases and terms that you have thought of yourself.

Tip: Often your search term will be a phrase instead of a single word. To carry out phrase searches, use double quotes, for example “problem drinking”.

Once you have chosen your search terms, you need to think about the best databases for your topic. The databases you choose will depend on the search question and the libraries you have access to.

Tip: It’s well worth taking a few minutes to get to know the databases available on the Library webpages and what they cover.

The next step is to combine your search terms in such a way that you only retrieve the more relevant references for your search question. In order to do this you need to build a search strategy . This involves using Boolean operators such as AND , OR and NOT .

AND narrows the results of the search by ensuring that all the search terms are present in the results. 

OR broadens the results of the search by ensuring that any of the search terms are present in the results.

NOT limits the results by rejecting a particular search term. Be careful with NOT because it will exclude any results containing that search term regardless of whether other parts of the article might have been of interest.

OR will broaden your number of results while AND will produce fewer results.

Try using this  Search-plan-worksheet   to break your topic down into concepts. These can then be linked together when you run the search. You can also add synonyms within each concept box. The yellow limits box is a prompt to think about any limits you want to apply when searching. This leads us to Step 6.

Tip: Most databases will allow you to use a truncation sign (*) or wildcard (?) to pick up various different endings to words or alternative spellings.

For example:  alcohol* would pick up alcohol, alcoholic, alcoholism, etc

Sm?th would find Smith and Smyth

The next step is to think about any other restrictions you want to make to your results.

Common limiters found on databases include:

• Peer reviewed articles
• Research articles
• Age group (adult, child, older person)
• Document type

Not all databases allow all of the limiters above.

When writing a dissertation, primary research articles are normally required. Where the database allows you, try limiting to research articles only.

Non-research materials can also be useful as an overview of your topic; for example a literature review can give an analysis of what has already been written on a topic.

The video below includes a demonstration of how limits can be applied using the CINAHL database as an example:

CINAHL - advanced

Once you have identified all your search terms and any limits you want to apply, you are ready to run your search on the databases you have chosen. 

Once you have some search results, you can look through them and start to select those that look relevant to your literature search. It is likely you will reject some because they are not quite what you wanted but there will be others that can be marked for further attention.

The title of an article on its own may not tell you very much; read the abstract quite carefully to see if the article is relevant or not.

Tip: You can show more details for each record by clicking on the article title. On some databases, there may be an abstract for the article which you can open. 

If you find you are either generating more results than you can possibly look at or too few results to write about, be prepared to adjust your search terms and the way they are combined.

If you get too many results you could try: •limiting to just the most recent material •adding another term or concept and linking it using “AND” •limiting to a particular country or geographical area such as UK

If you get too few results, you might try: •expanding your date range •removing any geographical limits you have applied  •removing the least important term or concept

Tip: Be prepared to try other databases and keep searching until you feel confident you have found enough relevant material.

Once you have selected some articles that look relevant for your piece of work, you will need to save them so that your hard work is not wasted.

At the same time, you will want to save your search strategy . This is a record of the terms you searched, how you combined them, any limits you applied and how many results you found.

You will also need to choose a way to save your results. One way is to email the results to yourself and this can be done from all the databases .

Another way is to export your results to reference management software such as Zotero, RefWorks, EndNote or Mendeley. This software allows you to collect, organise and cite research. It is suitable for managing references over a long period of time. 

The RCN Library and Museum provides help with using Zotero . 

Tip: Keep a record of all the databases you use as you carry out your search. It is also a good idea to note where you found any references you subsequently use for your dissertation.

The final step is to obtain the full text of the articles identified in your search which you believe may be useful for your assignment. If you are lucky, many of these will be available electronically and you may just be able to follow a link to the full text.

Alternatively you can copy and paste your article title into the Library search box  and if it is available as full text, a hyperlink will be shown which will link you to the document.

If you are studying elsewhere and have access to a university or hospital library, they may subscribe to different journals to the RCN Library so it is worth exploring what they can offer. If your library does not have either an electronic copy or a physical copy, you may need to request the article by interlibrary loan .

Tip: It is also worth using Google or other browsers to check for the article title you require. Sometimes the article has been made freely available on the internet by the authors.

Boolean operators – words (AND, OR and NOT) which can be used to combine search terms in order to widen or limit the search results.

Database – this is an online collection of citations to journal articles which have been indexed to make retrieval easier. Some databases which also provide full text access to the articles.

Limits – these are options within a database which allow search results to be broken down further. Common limits are year(s) of publication, document type and language. MEDLINE and CINAHL allow age limits too.

Search Strategy – the list of search terms and limits used to retrieve relevant articles from a database in order to answer a search question.

Subject headings – terms that have been assigned to describe a concept that may have many alternative keywords. All these alternative keywords or terms are brought together under the umbrella of this single term. Most health-related databases use subject headings.

Additional information

If after following these steps, you still can’t find what you are looking for, remember that there is always help available at your library. The RCN Library and Museum offers a range of help materials via our Literature searching and training pages . These include: • Databases guides in electronic and printed formats • Video tutorials on how to search the databases • 1-1 training sessions pre-bookable via the RCN website face to face or via zoom

A reading list is also available on dissertation and essay support which provides suggestions for key resources, books and journal articles which may help. Click on the link below to access this list:

Dissertation and essay support reading list

Here are other resources you may also find helpful. You will find links to each resource below too:

• Aveyard H (2019) Doing a literature review in health and social care: a practical guide . 4th edn. London: Open University Press.
• Bettany-Satlikov J (2016) How to do a systematic literature review in nursing: a step-by-step guide . 2nd edn. Maidenhead: Open University Press.
• Coughlan M and Cronin P (2016) Doing a literature review in nursing, health and social care . 2nd edn. Los Angeles: Sage.
• De Brún C, Pearce-Smith N, Heneghan C, Perera R and Badenoch D (2014) Searching skills toolkit: finding the evidence . 2nd edn. Chichester: Wiley Blackwell / BMJ Books.
• Hewitt-Taylor J (2017) The essential guide to doing a health and social care literature review . London: Routledge. 

Critical Appraisal subject guide

Your spaces.

• RCNi Profile
• Steward Portal
• RCN Foundation
• RCN Library
• RCN Starting Out

Work & Venue

• RCNi Nursing Jobs
• Work for the RCN
• RCN Working with us

Further Info

• Manage Cookie Preferences
• Modern slavery statement
• Accessibility
• Press office

Connect with us:

© 2024 Royal College of Nursing

Conducting a Literature Review

• Getting Started
• Developing a Question
• Searching the Literature
• Identifying Peer-Reviewed Resources
• Managing Results
• Analyzing the Literature
• Writing the Review

Need Help? Ask Your librarian!

Search Strategies

• Boolean Operators

Once you have identified the key concepts of your research question (see "Developing a Question"), you can use those concepts to develop keywords for your search strategy. The following tips and techniques will help you design a precise and relevant search strategy.

Keywords are any words you might use to search the record of an article, book, or other material in library databases. The database searches through the metadata (such as title, authors, publication, abstract, etc.) to find resources that contain the word you searched, and may also search through the full text of the material.

Keywords are most successful when you're searching for the words that the authors use to describe the research topic, as most databases will search for those specific words within the record of the article. To increase your chance of returning relevant results, consider all of the words that  might  be used to describe the research you're trying to find, and try some of these out in sample searches to determine which words return the best results.

Search Tips - Keywords

• Search for singular and plural terms together: (physician OR physicians)
• Search for both the American and British spelling of words: (behavior OR behaviour)
• Search for synonyms of terms together: (teenager OR adolescent)
• Search for phrases inside of quotation marks: ("young adult")

Use Boolean operators to combine keywords for more precise search results. 

AND - If the term must be included in your search:

influenza AND vaccine

OR - If terms are interchangeable, i.e. synonyms. Place OR'd terms within parentheses:

(influenza OR flu) AND vaccine

NOT - If a term should not be included in your search. This Boolean operator is rarely necessary for literature reviews.

(influenza OR flu) AND vaccine NOT H1N1

Note how we've used parentheses in the examples above. Search strings like these are similar to mathematical equations, where you perform the actions within the parentheses before proceeding from left to right to run the search. For example, using the search [(influenza OR flu) AND vaccine] will find results that have a term relating to influenza/flu, as well as the term vaccine.

If we moved the parentheses, it would be a very different search. [influenza OR (flu AND vaccine)] will provide results that use the term influenza, as well as results that use both the terms flu and vaccine. This means you would get results having to do with influenza but perhaps nothing to do with vaccination. 

Here are a few examples of how this search would be different depending on the arrangement of booleans and keywords. The area highlighted in pink represents the search results that would be returned with this search.

Truncation allows you to quickly include all variations of a word in your search. Use the root of the keyword and add an asterisk (*). For example:

nurs* = nurse, nurses, nursing, nursery

IMPORTANT: Notice that "nursery" is also retrieved in the above search. Truncation will save you from having to include a large number of synonyms, but it will also add a certain number of irrelevant results. You can limit this effect by using the NOT Boolean operator, i.e. NOT nursery.

Wild cards allow you to replace a letter in a keyword to retrieve all variations of the spelling. For example:

p?ediatric = pediatric, paediatric

Free-Text vs. Thesaurus Searching

While you can search any word as a keyword, databases also contain an official list of the terms they use to describe the subject of each article, called Subject Headings. You can look up Subject Headings in the thesaurus of the database, using the thesaurus's search box to pull up the recommended Subject Heading for a given keyword. When searching specifically for Subject Headings, the database will only search the Subject Headings field within the record of each article (ie, not the title, abstract, etc.). This is a much more targeted method of searching, and is an excellent addition to your search strategy. 

A strong search strategy will use both free-text (keyword) searching and thesaurus searching, to ensure that all relevant articles have been retrieved by the search. The lists below outline the strengths and weaknesses of both types of search strategies.

Free-Text Searching

• Natural language words describing your topic
• More flexible search strategy - can use any term in any combination
• Database looks for keywords anywhere in the record - not necessarily connected together
• May yield too many or too few results
• May yield many irrelevant results

Thesaurus Searching

• Pre-defined "controlled vocabulary" words used to describe the content of each item in a database
• Less flexible search strategy - need to know the exact controlled vocabulary term
• Database looks for subjects only in the subject heading or descriptor field, where the most relevant words appear
• If too many results, you can use subheadings to focus on one aspect of a broader topic
• Results are usually very relevant to the topic

MIT Libraries. Database Search Tips: Keywords vs. Subjects. https://libguides.mit.edu/c.php?g=175963&p=1160804

Each database has their own thesaurus. You will need to adapt your search strategy for each database to take advantage of their unique thesaurus.

PubMed uses MeSH terms (Medical Subject Headings). You can learn more about finding and using MeSH terms here:

• The Basics of Medical Subject Headings (MeSH) in MEDLINE/PubMed: A Tutorial

CINAHL uses CINAHL Headings. You can learn more about finding and use these terms here:

• Using CINAHL/MeSH Headings

In other databases, look for a link with the terms "headings", "subject headings", or "thesaurus" to find the appropriate thesaurus terms for your search.

Citation Searching

Citation searching is a search strategy that allows you to search either forward or backwards time through the literature based on an identified relevant article:

You can search forward in time by using databases that allow you to search for other articles that have cited the identified relevant article. (Web of Science and Google Scholar can do this automatically.)

• Web of Science (Clarivate Analytics)
• Google Scholar

You can search backward in time by reviewing the reference list of the identified relevant article for additional article citations.

For more information about how to perform citation searches, check out this guide from the University of Toledo Libraries:

• How To: Cited Reference Searches in Web of Science Guide from the University of Toledo

Retrieving Materials

Select a database.

When searching for articles, it is best to use an appropriate subject database rather than the SearchIT catalog. Be sure to select your database from the Spokane Academic Library homepage to ensure that you will have access to full-text articles.

"Find It @ WSU" Button in PubMed

When you have found an article that you would like to read in its entirety, look for the "Find It @ WSU" Button. This button will take you to the article entry in the SearchIT catalog.

Here's what that looks like in PubMed.

"Find It @ WSU" Button in CINAHL

"Find It @ WSU" Button in PsycINFO

Accessing the Full-Text Article

After selecting the "Find It @ WSU" Button, you will be taken to the article entry in SearchIT. Select the link under the Access Options box to be directed to the full-text article. 

If an article is not available in the WSU Libraries collection, you can request the article through interlibrary loan by selecting the link under "Access Options".

See the Using Interlibrary Loan section for more information.

• << Previous: Developing a Question
• Next: Identifying Peer-Reviewed Resources >>
• Last Updated: Apr 16, 2024 3:40 PM
• URL: https://libguides.libraries.wsu.edu/litreview

• Technical Support
• Find My Rep

You are here

How to Do Your Literature Review

• Gary Thomas - University of Birmingham
• Description

This book will walk you through every step of crafting an excellent literature review. It will show you where to find sources, how to assess their quality, and how to combine and integrate these sources into the best possible literature review. From scoping the field to searching the literature to synthesising your findings, this book hits all the spots.

Presenting a clear and detailed roadmap to ensure you don’t miss a step, the book includes:

· Advice on the best ways to scan and search the literature.

· All the latest information and advice on using AI tools for searching literature.

· Case studies and real-life examples from a range of disciplines so you can learn from other researchers who have been in your shoes.

· DIY activities so you can practice your skills and get to grips with key concepts.

· Guidance on how to analyse and synthesise the findings from your literature searches.

· Advice and guidance on organising your writing.

For tutors, each chapter is accompanied by a PowerPoint presentation that will map onto a 10 or 11 week module.

Written in Gary’s signature easy-to-read style, this book is an essential companion for anyone conducting a literature review in the applied social sciences.

See what’s new to this edition by selecting the Features tab on this page. Should you need additional information or have questions regarding the HEOA information provided for this title, including what is new to this edition, please email [email protected] . Please include your name, contact information, and the name of the title for which you would like more information. For information on the HEOA, please go to http://ed.gov/policy/highered/leg/hea08/index.html .

For assistance with your order: Please email us at [email protected] or connect with your SAGE representative.

SAGE 2455 Teller Road Thousand Oaks, CA 91320 www.sagepub.com

Supplements

Preview this book, for instructors, select a purchasing option.

• Submission Guidelines

safety learning system header

Learning Objectives

(1) Explain steps in conducting a literature search

(2) Identify resources to utilize in a literature search

(3) Perform an online literature search using U of U Health resources

Valentina is a third year pediatric resident who notices that many of the teenagers she sees in clinic use their phones to play games and connect with friends and family members. She wonders if there could be an app for teenagers to manage their chronic diseases, specifically type 1 diabetes. But where does she begin? 

What is a literature search?

iterature search is a comprehensive exploration of published literature with the purpose of finding scholarly articles on a specific topic . Managing and organizing selected scholarly works can also be useful.

Why do a literature search?

Literature search is a critical component for any evidence-based project. It helps you to understand the complexity of a clinical issue, gives you insight into the scope of a problem, and provides you with best treatment approaches and the best available evidence on the topic. Without this step, your evidence-based practice project cannot move forward.

Five steps for literature search success

There are several steps involved in conducting a literature search. You may discover more along the way, but these steps will provide a good foundation. 

Plan using PICO(T) to develop your clinical question and formulate a search strategy.

Identify a database to search.

Conduct your search in one or more databases.

Select relevant articles .

Organize your results . Remember that searching the literature is a process.

#1: Plan using PICO(T)

The PICO(T) question framework is a formula for developing answerable, researchable questions. Using PICO(T) guides you in your search for evidence and may even help you be more efficient in the process ( Click here to learn all about PICO(T) ). 

Once you have your PICO(T) question you can formulate a search strategy by identifying key words, synonyms and subject headings. These can help you determine which databases to use. 

#2: Identify a database

For your search, you will need to consult a variety of resources to find information on your topic. While some of these resources will overlap, each also contains unique information that you won’t find in other databases.  

The "Big 3" databases: Embase, PubMed, and Scopus are always important to search because they contain large numbers of citations and have a fairly broad scope. ( Click here to access these databases and others in the library's A to Z database.) 

In addition to searching these expansive databases, try one that is more topic specific.

We are here to help.

If you are conducting a literature search and are not certain of the details, don't panic! U of U Health has a wealth of resources, including experienced librarians, to help you through the process. Learn more here. 

Utah’s Epic-embedded librarian support

Did you know you can request evidence-based information from the library directly through Epic?  Contact us through Epic’s Message Basket.

Eccles Health Sciences medical librarians are able to provide expertise in articulating the clinical question, identifying appropriate data sources, and locating the best evidence in the shortest amount of time. You can also send a message to ASK EHSL .

#3: Conduct your search

Now that you have identified pertinent databases, it is time to begin the search!

Use the key words that you’ve identified from your PICO(T) question to start searching. You might start your search broadly, with just a few key words, and then add more once you see the scope of the literature. If the initial search doesn't produce many results, you can play with removing some key words and adding more granular detail.

In our intro case study, Valentina’s population is teenagers with type 1 diabetes and her intervention is a mobile app. Watch the video below to see how Valentina uses the powerful Embase PICO search feature to identify synonyms for type 1 diabetes, mobile apps, and teenagers.

Example of   Embase using PICO Why use Embase? This search casts a wider net than most databases for more results.

Common Search Terms and Symbols

AND Includes both keywords Narrows search OR Either keyword/concept Combine synonyms and similar concepts Expands search "Double quotes"  Specific phrase Wildcard* Any word ending variants (singular, plural, etc.) Example: nurs* = nurse, nurses, nursing, etc.

Controlled Vocabulary

Want to help make your search more accurate? Try using the controlled vocabulary, or main words or phrases that describe the main themes in an article, within databases. Controlled vocabulary is a standardized hierarchical system. For example, PubMed uses Medical Subject Headings or MeSH terms to “map” keywords to the controlled vocabulary. Not all databases use a controlled vocabulary, but many do. Embase’s controlled vocabulary is called Emtree, and CINAHL’s  controlled vocabulary is called CINAHL Headings. Consider focusing the controlled vocabulary as the major topic when using MeSH, Emtree, or CINAHL Headings. 

For Valentina’s question, there are MeSH terms for Adolescent, Diabetes Mellitus, Type 1, and Mobile Applications.

Example of  PubMed using MeSH MeSH helps focus your PubMed search

Talk with your librarians for more help with searching with controlled vocabularies. 

Every database uses filters to help you narrow your search. There are different filters in each database, but they tend to work in similar ways. Use filters to help you refine your search, rather than adding those keywords to the search. Filters include article/publication type, age, language, publication years, and species.

Using filters can help return the most accurate results for your search.

Article/publication types, such as randomized controlled trial, systematic reviews, can be used as filters.

Use an Age Filter, rather than adding “pediatric” or “geriatric” to your search.

Valentina uses the age filter for her question rather than as a keyword in the video below.

Example of a PubMed keyword search using filters PubMed is the most common search because it is the most widely available.

#4: Select relevant articles

Once you have completed your search, you’ll select articles that are relevant to your question. Some databases also include a “similar articles” feature which recommends other articles similar to the article you’re reviewing—this can also be a helpful tool.

When you’ve identified an article that appears relevant to your topic, use the “Snowballing” technique to find additional articles. Snowballing involves reviewing the reference lists of articles from your search. 

In other words, look at your key articles and review their reference list for additional key or seminal articles to aid in your search.

#5: Organize your results

As you begin to collect articles during your literature search, it is important to store them in an organized fashion. Most research databases include personalized accounts for storing selected references and search strategies. 

Reference managers are a great way to not only keep articles organized, but they also generate in-text citations and bibliographies when writing manuscripts, and provide a platform for sharing references with others working on your project.

A number of reference managers—such as Zotero , EndNote , RefWorks, Mendeley , and Papers are available. EndNote Basic (web-based) is freely available to U of U faculty, staff and students. If you need help with this process, contact a librarian to help you select the reference manager  that will best suit your needs.

Using these steps, you’re ready to start your literature search. It is important to remember that there is not a right or wrong way to do the search. Literature searches are an iterative process—it will take some time and negotiation to find what you are looking for. You can always change your approach, or the information resource you are using. The important thing is to just keep trying. And before you get frustrated or give up, contact a librarian . They are here to help!

This article originally appeared May 12, 2020. It was updated to reflect current practice on March 14, 2021.

Tallie Casucci

Barbara wilson.

You have a good idea about what you want to study, compare, understand or change. But where do you go from there? First, you need to be clear about exactly what it is you want to find out. In other words, what question are you attempting to answer? Librarian Tallie Casucci and nursing leaders Gigi Austria and Barb Wilson help us understand how to formulate searchable, answerable questions using the PICO(T) framework.

EBP, or evidence-based practice, is a term we encounter frequently in today’s health care environment. But what does it really mean for the health care provider? College of Nursing interim dean Barbara Wilson and Nurse manager Gigi Austria explain how to integrate EBP into all aspects of patient care.

Frequent and deliberate practice is critical to attaining procedural competency. Cheryl Yang, pediatric emergency medicine fellow, shares a framework for providing trainees with opportunities to learn, practice, and maintain procedural skills, while ensuring high standards for patient safety.

Subscribe to our newsletter

Receive the latest insights in health care equity, improvement, leadership, resilience, and more..

Contact the Accelerate Team

50 North Medical Drive   |   Salt Lake City, Utah 84132   |   801-587-2157

How to do a Literature Search: Introduction

• Introduction
• Choosing a database
• Choosing keywords
• Using keywords
• Author searching
• Managing your search/results

What is a literature review?

You may be asked to write a literature review as part of an undergraduate project or postgraduate dissertation.  A well-conducted literature review will showcase your ability to:

• Survey the literature and select the most important contributions on your topic
• Critically evaluate the literature to identify key developments, trends, issues, gaps in knowledge
• Present your findings in a clear and coherent manner

The structure of a literature review may vary according to your specific subject but it will normally include these three areas:

• Introduction : an overview of your topic explaining why it is important, putting it in the wider context and perhaps highlighting recent progress and future potential.  It may also explain the scope and the organisation of your review.
• Main body : a discussion of how research in the topic has progressed to date, critically evaluating the key studies and explaining their significance.  
• Conclusion : a summary of current knowledge, highlighting any gaps in current knowledge or practice and suggesting how these may be overcome in future research.

Having identified the topic of your review, the first step will be to undertake a literature search .  

What is a literature search?

Define your research question(s).

Before you login to a database to begin your search it's crucial that you analyse your topic, breaking it down into a number of research questions.

Take, for example, this topic:   Are biofuels the answer to falling oil reserves?

You  could  type this sentence into a database search box, but that is usually not helpful, as the sentence may not contain the most appropriate keywords.  Also this single sentence is unlikely to encompass everything that you want to find out.  You need to break down the topic into a number of separate questions and then look for the answers. For this example here are some of the questions you could ask:

• What is a biofuel?  
• How are they made?
• How much of our fuel is already from biofuel (market share)?
• Could we make enough to replace oil and/or gas?
• What are the advantages and disadvantages of using biofuels compared with oil and gas?
• Could we use biofuels for transport?
• What is UK government policy relating to biofuels?

You  may  find the answers to all of these questions using a single search engine such as Google Scholar, or a single Library database, but you are more likely to succeed if you match each question to a relevant source .

Introduction to literature searching

Library video (10 minutes)

• Next: Choosing a database >>
• Last Updated: Nov 22, 2022 9:56 AM
• URL: https://library.bath.ac.uk/literaturesearch

• Systematic Review
• Open access
• Published: 23 May 2024

Systematic literature review of real-world evidence for treatments in HR+/HER2- second-line LABC/mBC after first-line treatment with CDK4/6i

• Veronique Lambert   ORCID: orcid.org/0000-0002-6984-0038 1 ,
• Sarah Kane   ORCID: orcid.org/0009-0006-9341-4836 2   na1 ,
• Belal Howidi   ORCID: orcid.org/0000-0002-1166-7631 2   na1 ,
• Bao-Ngoc Nguyen   ORCID: orcid.org/0000-0001-6026-2270 2   na1 ,
• David Chandiwana   ORCID: orcid.org/0009-0002-3499-2565 3 ,
• Yan Wu   ORCID: orcid.org/0009-0008-3348-9232 1 ,
• Michelle Edwards   ORCID: orcid.org/0009-0001-4292-3140 3 &
• Imtiaz A. Samjoo   ORCID: orcid.org/0000-0003-1415-8055 2   na1  

BMC Cancer volume  24 , Article number:  631 ( 2024 ) Cite this article

320 Accesses

1 Altmetric

Metrics details

Cyclin-dependent kinase 4 and 6 inhibitors (CDK4/6i) combined with endocrine therapy (ET) are currently recommended by the National Comprehensive Cancer Network (NCCN) guidelines and the European Society for Medical Oncology (ESMO) guidelines as the first-line (1 L) treatment for patients with hormone receptor-positive, human epidermal growth factor receptor 2-negative, locally advanced/metastatic breast cancer (HR+/HER2- LABC/mBC). Although there are many treatment options, there is no clear standard of care for patients following 1 L CDK4/6i. Understanding the real-world effectiveness of subsequent therapies may help to identify an unmet need in this patient population. This systematic literature review qualitatively synthesized effectiveness and safety outcomes for treatments received in the real-world setting after 1 L CDK4/6i therapy in patients with HR+/ HER2- LABC/mBC.

MEDLINE®, Embase, and Cochrane were searched using the Ovid® platform for real-world evidence studies published between 2015 and 2022. Grey literature was searched to identify relevant conference abstracts published from 2019 to 2022. The review was conducted in accordance with PRISMA guidelines (PROSPERO registration: CRD42023383914). Data were qualitatively synthesized and weighted average median real-world progression-free survival (rwPFS) was calculated for NCCN/ESMO-recommended post-1 L CDK4/6i treatment regimens.

Twenty records (9 full-text articles and 11 conference abstracts) encompassing 18 unique studies met the eligibility criteria and reported outcomes for second-line (2 L) treatments after 1 L CDK4/6i; no studies reported disaggregated outcomes in the third-line setting or beyond. Sixteen studies included NCCN/ESMO guideline-recommended treatments with the majority evaluating endocrine-based therapy; five studies on single-agent ET, six studies on mammalian target of rapamycin inhibitors (mTORi) ± ET, and three studies with a mix of ET and/or mTORi. Chemotherapy outcomes were reported in 11 studies. The most assessed outcome was median rwPFS; the weighted average median rwPFS was calculated as 3.9 months (3.3-6.0 months) for single-agent ET, 3.6 months (2.5–4.9 months) for mTORi ± ET, 3.7 months for a mix of ET and/or mTORi (3.0–4.0 months), and 6.1 months (3.7–9.7 months) for chemotherapy. Very few studies reported other effectiveness outcomes and only two studies reported safety outcomes. Most studies had heterogeneity in patient- and disease-related characteristics.

Conclusions

The real-world effectiveness of current 2 L treatments post-1 L CDK4/6i are suboptimal, highlighting an unmet need for this patient population.

Peer Review reports

Introduction

Breast cancer (BC) is the most diagnosed form of cancer in women with an estimated 2.3 million new cases diagnosed worldwide each year [ 1 ]. BC is the second leading cause of cancer death, accounting for 685,000 deaths worldwide per year [ 2 ]. By 2040, the global burden associated with BC is expected to surpass three million new cases and one million deaths annually (due to population growth and aging) [ 3 ]. Numerous factors contribute to global disparities in BC-related mortality rates, including delayed diagnosis, resulting in a high number of BC cases that have progressed to locally advanced BC (LABC) or metastatic BC (mBC) [ 4 , 5 , 6 ]. In the United States (US), the five-year survival rate for patients who progress to mBC is three times lower (31%) than the overall five-year survival rate for all stages (91%) [ 6 , 7 ].

Hormone receptor (HR) positive (i.e., estrogen receptor and/or progesterone receptor positive) coupled with negative human epidermal growth factor 2 (HER2) expression is the most common subtype of BC, accounting for ∼ 60–70% of all BC cases [ 8 , 9 ]. Historically, endocrine therapy (ET) through estrogen receptor modulation and/or estrogen deprivation has been the standard of care for first-line (1 L) treatment of HR-positive/HER2-negative (HR+/HER2-) mBC [ 10 ]. However, with the approval of the cyclin-dependent kinase 4/6 inhibitor (CDK4/6i) palbociclib in combination with the aromatase inhibitor (AI) letrozole in 2015 by the US Food and Drug Administration (FDA), 1 L treatment practice patterns have evolved such that CDK4/6i (either in combination with AIs or with fulvestrant) are currently considered the standard of care [ 11 , 12 , 13 , 14 , 15 , 16 , 17 ]. Other CDK4/6i (ribociclib and abemaciclib) in combination with ET are approved for the treatment of HR+/HER2- LABC/mBC; 1 L use of ribociclib in combination with an AI was granted FDA approval in March 2017 for postmenopausal women (with expanded approval in July 2018 for pre/perimenopausal women and for use in 1 L with fulvestrant for patients with disease progression on ET as well as for postmenopausal women), and abemaciclib in combination with fulvestrant was granted FDA approval in September 2017 for patients with disease progression following ET and as monotherapy in cases where disease progression occurs following ET and prior chemotherapy in mBC (with expanded approval in February 2018 for use in 1 L in combination with an AI for postmenopausal women) [ 18 , 19 , 20 , 21 ].

Clinical trials investigating the addition of CDK4/6i to ET have demonstrated significant improvement in progression-free survival (PFS) and significant (ribociclib) or numerical (palbociclib and abemaciclib) improvement in overall survival (OS) compared to ET alone in patients with HR+/HER2- advanced or mBC, making this combination treatment the recommended option in the 1 L setting [ 22 , 23 , 24 , 25 , 26 , 27 ]. However, disease progression occurs in a significant portion of patients after 1 L CDK4/6i treatment [ 28 ] and the optimal treatment sequence after progression on CDK4/6i remains unclear [ 29 ]. At the time of this review (literature search conducted December 14, 2022), guidelines by the National Comprehensive Cancer Network (NCCN) and the European Society for Medical Oncology (ESMO) recommend various options for the treatment of HR+/HER2- advanced BC in the second-line (2 L) setting, including fulvestrant monotherapy, mammalian target of rapamycin inhibitors (mTORi; e.g., everolimus) ± ET, alpelisib + fulvestrant (if phosphatidylinositol-4,5-bisphosphate 3-kinase catalytic subunit alpha mutation positive [PIK3CA-m+]), poly-ADP ribose polymerase inhibitors (PARPi) including olaparib or talazoparib (if breast cancer gene/partner and localizer of BRCA2 positive [BRCA/PALB2m+]), and chemotherapy (in cases when a visceral crisis is present) [ 15 , 16 ]. CDK4/6i can also be used in 2 L [ 16 , 30 ]; however, limited data are available to support CDK4/6i rechallenge after its use in the 1 L setting [ 15 ]. Depending on treatments used in the 1 L and 2 L settings, treatment in the third-line setting is individualized based on the patient’s response to prior treatments, tumor load, duration of response, and patient preference [ 9 , 15 ]. Understanding subsequent treatments after 1 L CDK4/6i, and their associated effectiveness, is an important focus in BC research.

Treatment options for HR+/HER2- LABC/mBC continue to evolve, with ongoing research in both clinical trials and in the real-world setting. Real-world evidence (RWE) offers important insights into novel therapeutic regimens and the effectiveness of treatments for HR+/HER2- LABC/mBC. The effectiveness of the current treatment options following 1 L CDK4/6i therapy in the real-world setting highlights the unmet need in this patient population and may help to drive further research and drug development. In this study, we conducted a systematic literature review (SLR) to qualitatively summarize the effectiveness and safety of treatment regimens in the real-world setting after 1 L treatment with CDK4/6i in patients with HR+/HER2- LABC/mBC.

Literature search

An SLR was performed in accordance with the Cochrane Handbook for Systematic Reviews of Interventions [ 31 ] and reported in alignment with the Preferred Reporting Items for Systematic Literature Reviews and Meta-Analyses (PRISMA) statement [ 32 ] to identify all RWE studies assessing the effectiveness and safety of treatments used for patients with HR+/HER2- LABC/mBC following 1 L CDK4/6i therapy and received subsequent treatment in 2 L and beyond (2 L+). The Ovid® platform was used to search MEDLINE® (including Epub Ahead of Print and In-Process, In-Data-Review & Other Non-Indexed Citations), Ovid MEDLINE® Daily, Embase, Cochrane Central Register of Controlled Trials, and Cochrane Database of Systematic Reviews by an experienced medical information specialist. The MEDLINE® search strategy was peer-reviewed independently by a senior medical information specialist before execution using the Peer Review of Electronic Search Strategies (PRESS) checklist [ 33 ]. Searches were conducted on December 14, 2022. The review protocol was developed a priori and registered with the International Prospective Register of Systematic Review (PROSPERO; CRD42023383914) which outlined the population, intervention, comparator, outcome, and study design (PICOS) criteria and methodology used to conduct the review (Table  1 ).

Search strategies utilized a combination of controlled vocabulary (e.g., “HER2 Breast Cancer” or “HR Breast Cancer”) and keywords (e.g., “Retrospective studies”). Vocabulary and syntax were adjusted across databases. Published and validated filters were used to select for study design and were supplemented using additional medical subject headings (MeSH) terms and keywords to select for RWE and nonrandomized studies [ 34 ]. No language restrictions were included in the search strategy. Animal-only and opinion pieces were removed from the results. The search was limited to studies published between January 2015 and December 2022 to reflect the time at which FDA approval was granted for the first CDK4/6i agent (palbociclib) in combination with AI for the treatment of LABC/mBC [ 35 ]. Further search details are presented in Supplementary Material 1 .

Grey literature sources were also searched to identify relevant abstracts and posters published from January 2019 to December 2022 for prespecified relevant conferences including ESMO, San Antonio Breast Cancer Symposium (SABCS), American Society of Clinical Oncology (ASCO), the International Society for Pharmacoeconomics and Outcomes Research (ISPOR US), and the American Association for Cancer Research (AACR). A search of ClinicalTrials.gov was conducted to validate the findings from the database and grey literature searches.

Study selection, data extraction & weighted average calculation

Studies were screened for inclusion using DistillerSR Version 2.35 and 2.41 (DistillerSR Inc. 2021, Ottawa, Canada) by two independent reviewers based on the prespecified PICOS criteria (Table  1 ). A third reviewer was consulted to resolve any discrepancies during the screening process. Studies were included if they reported RWE on patients aged ≥ 18 years with HR+/HER2- LABC/mBC who received 1 L CDK4/6i treatment and received subsequent treatment in 2 L+. Studies were excluded if they reported the results of clinical trials (i.e., non-RWE), were published in any language other than English, and/or were published prior to 2015 (or prior to 2019 for conference abstracts and posters). For studies that met the eligibility criteria, data relating to study design and methodology, details of interventions, patient eligibility criteria and baseline characteristics, and outcome measures such as efficacy, safety, tolerability, and patient-reported outcomes (PROs), were extracted (as available) using a Microsoft Excel®-based data extraction form (Microsoft Corporation, WA, USA). Data extraction was performed by a single reviewer and was confirmed by a second reviewer. Multiple publications identified for the same RWE study, patient population, and setting that reported data for the same intervention were linked and extracted as a single publication. Weighted average median real-world progression-free survival (rwPFS) values were calculated by considering the contribution to the median rwPFS of each study proportional to its respective sample size. These weighted values were then used to compute the overall median rwPFS estimate.

Quality assessment

The Newcastle-Ottawa scale (NOS) for nonrandomized (cohort) studies was used to assess the risk of bias for published, full-text studies [ 36 ]. The NOS allocates a maximum of nine points for the least risk of bias across three domains: (1) Formation of study groups (four points), (2) Comparability between study groups (two points), (3) Outcome ascertainment (three points). NOS scores can be categorized in three groups: very high risk of bias (0 to 3 points), high risk of bias (4 to 6), and low risk of bias (7 to 9) [ 37 ]. Risk of bias assessment was performed by one reviewer and validated by a second independent reviewer to verify accuracy. Due to limited methodological data by which to assess study quality, risk of bias assessment was not performed on conference abstracts or posters. An amendment to the PROSPERO record (CRD42023383914) for this study was submitted in relation to the quality assessment method (specifying usage of the NOS).

The database search identified 3,377 records; after removal of duplicates, 2,759 were screened at the title and abstract stage of which 2,553 were excluded. Out of the 206 reports retrieved and assessed for eligibility, an additional 187 records were excluded after full-text review; most of these studies were excluded for having patients with mixed lines of CDK4/6i treatment (i.e., did not receive CDK4/6i exclusively in 1 L) (Fig.  1 and Table S1 ). The grey literature search identified 753 records which were assessed for eligibility; of which 752 were excluded mainly due to the population not meeting the eligibility criteria (Fig.  1 ). In total, the literature searches identified 20 records (9 published full-text articles and 11 conference abstracts/posters) representing 18 unique RWE studies that met the inclusion criteria. The NOS quality scores for the included full-text articles are provided in Table S2 . The scores ranged from four to six points (out of a total score of nine) and the median score was five, indicating that all the studies suffered from a high risk of bias [ 37 ].

Most studies were retrospective analyses of chart reviews or medical registries, and all studies were published between 2017 and 2022 (Table S3 ). Nearly half of the RWE studies (8 out of 18 studies) were conducted in the US [ 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 ], while the remaining studies included sites in Canada, China, Germany, Italy, Japan, and the United Kingdom [ 46 , 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 ]. Sample sizes ranged from as few as 4 to as many as 839 patients across included studies, with patient age ranging from 26 to 86 years old.

Although treatment characteristics in the 1 L setting were not the focus of the present review, these details are captured in Table S3 . Briefly, several RWE studies reported 1 L CDK4/6i use in combination with ET (8 out of 18 studies) or as monotherapy (2 out of 18 studies) (Table S3 ). Treatments used in combination with 1 L CDK4/6i included letrozole, fulvestrant, exemestane, and anastrozole. Where reported (4 out of 18 studies), palbociclib was the most common 1 L CDK4/6i treatment. Many studies (8 out of 18 studies) did not report which specific CDK4/6i treatment(s) were used in 1 L or if its administration was in combination or monotherapy.

Characteristics of treatments after 1 L CDK4/6i therapy

Across all studies included in this review, effectiveness and safety data were only available for treatments administered in the 2 L setting after 1 L CDK4/6i treatment. No studies were identified that reported outcomes for patients treated in the third-line setting or beyond after 1 L CDK4/6i treatment. All 18 studies reported effectiveness outcomes in 2 L, with only two of these studies also describing 2 L safety outcomes. The distribution of outcomes reported in these studies is provided in Table S4 . Studies varied in their reporting of outcomes for 2 L treatments; some studies reported outcomes for a group of 2 L treatments while others described independent outcomes for specific 2 L treatments (i.e., everolimus, fulvestrant, or chemotherapy agents such as eribulin mesylate) [ 42 , 45 , 50 , 54 , 55 ]. Due to the heterogeneity in treatment classes reported in these studies, this data was categorized (as described below) to align with the guidelines provided by NCCN and ESMO [ 15 , 16 ]. The treatment class categorizations for the purpose of this review are: single-agent ET (patients who exclusively received a single-agent ET after 1 L CDK4/6i treatment), mTORi ± ET (patients who exclusively received an mTORi with or without ET after 1 L CDK4/6i treatment), mix of ET and/or mTORi (patients who may have received only ET, only mTORi, and/or both treatments but the studies in this group lacked sufficient information to categorize these patients in the “single-agent ET” or “mTOR ± ET” categories), and chemotherapy (patients who exclusively received chemotherapy after 1 L CDK4/6i treatment). Despite ESMO and NCCN guidelines indicating that limited evidence exists to support rechallenge with CDK4/6i after 1 L CDK4/6i treatment [ 15 , 16 ], two studies reported outcomes for this treatment approach. Data for such patients were categorized as “ CDK4/6i ± ET ” as it was unclear how many patients receiving CDK4/6i rechallenge received concurrent ET. All other patient groups that lacked sufficient information or did not report outcome/safety data independently (i.e., grouped patients with mixed treatments) to categorize as one of the treatment classes described above were grouped as “ other ”.

The majority of studies reported effectiveness outcomes for endocrine-based therapy after 1 L CDK4/6i treatment; five studies for single-agent ET, six studies for mTORi ± ET, and three studies for a mix of ET and/or mTORi (Fig.  2 ). Eleven studies reported effectiveness outcomes for chemotherapy after 1 L CDK4/6i treatment, and only two studies reported effectiveness outcomes for CDK4/6i rechallenge ± ET. Eight studies that described effectiveness outcomes were grouped into the “other” category. Safety data was only reported in two studies: one study evaluating the chemotherapy agent eribulin mesylate and one evaluating the mTORi everolimus.

Effectiveness outcomes

Real-world progression-free survival

Median rwPFS was described in 13 studies (Tables  2 and Table S5 ). Across the 13 studies, the median rwPFS ranged from 2.5 months [ 49 ] to 17.3 months [ 39 ]. Out of the 13 studies reporting median rwPFS, 10 studies reported median rwPFS for a 2 L treatment recommended by ESMO and NCCN guidelines, which ranged from 2.5 months [ 49 ] to 9.7 months [ 45 ].

Weighted average median rwPFS was calculated for 2 L treatments recommended by both ESMO and NCCN guidelines (Fig.  3 ). The weighted average median rwPFS for single-agent ET was 3.9 months ( n  = 92 total patients) and was derived using data from two studies reporting median rwPFS values of 3.3 months ( n  = 70) [ 38 ] and 6.0 months ( n  = 22) [ 40 ]. For one study ( n  = 7) that reported outcomes for single agent ET, median rwPFS was not reached during the follow-up period; as such, this study was excluded from the weighted average median rwPFS calculation [ 49 ].

The weighted average median rwPFS for mTORi ± ET was 3.6 months ( n  = 128 total patients) and was derived based on data from 3 studies with median rwPFS ranging from 2.5 months ( n  = 4) [ 49 ] to 4.9 months ( n  = 25) [ 54 ] (Fig.  3 ). For patients who received a mix of ET and/or mTORi but could not be classified into the single-agent ET or mTORi ± ET treatment classes, the weighted average median rwPFS was calculated to be 3.7 months ( n  = 17 total patients). This was calculated based on data from two studies reporting median rwPFS values of 3.0 months ( n  = 5) [ 46 ] and 4.0 months ( n  = 12) [ 49 ]. Notably, one study of patients receiving ET and/or everolimus reported a median rwPFS duration of 3.0 months; however, this study was excluded from the weighted average median rwPFS calculation for the ET and/or mTORi class as the sample size was not reported [ 53 ].

The weighted average median rwPFS for chemotherapy was 6.1 months ( n  = 499 total patients), calculated using data from 7 studies reporting median rwPFS values ranging from 3.7 months ( n  = 249) [ 38 ] to 9.7 months ( n  = 121) [ 45 ] (Fig.  3 ). One study with a median rwPFS duration of 5.6 months was not included in the weighted average median rwPFS calculation as the study did not report the sample size [ 53 ]. A second study was excluded from the calculation since the reported median rwPFS was not reached during the study period ( n  = 7) [ 41 ].

Although 2 L CDK4/6i ± ET rechallenge lacks sufficient information to support recommendation by ESMO and NCCN guidelines, the limited data currently available for this treatment have shown promising results. Briefly, two studies reported median rwPFS for CDK4/6i ± ET with values of 8.3 months ( n  = 302) [ 38 ] and 17.3 months ( n  = 165) (Table  2 ) [ 39 ]. The remaining median rwPFS studies reported data for patients classified as “Other” (Table S5 ). The “Other” category included median rwPFS outcomes from seven studies, and included a myriad of treatments (e.g., ET, mTOR + ET, chemotherapy, CDK4/6i + ET, alpelisib + fulvestrant, chidamide + ET) for which disaggregated median rwPFS values were not reported.

Overall survival

Median OS for 2 L treatment was reported in only three studies (Table  2 ) [ 38 , 42 , 43 ]. Across the three studies, the 2 L median OS ranged from 5.2 months ( n  = 3) [ 43 ] to 35.7 months ( n  = 302) [ 38 ]. Due to the lack of OS data in most of the studies, weighted averages could not be calculated. No median OS data was reported for the single-agent ET treatment class whereas two studies reported median OS for the mTORi ± ET treatment class, ranging from 5.2 months ( n  = 3) [ 43 ] to 21.8 months ( n  = 54) [ 42 ]. One study reported 2 L median OS of 24.8 months for a single patient treated with chemotherapy [ 43 ]. The median OS data in the CDK4/6i ± ET rechallenge group was 35.7 months ( n  = 302) [ 38 ].

Patient mortality was reported in three studies [ 43 , 44 , 45 ]. No studies reported mortality for the single-agent ET treatment class and only one study reported this outcome for the mTORi ± ET treatment class, where 100% of patients died ( n  = 3) as a result of rapid disease progression [ 43 ]. For the chemotherapy class, one study reported mortality for one patient receiving 2 L capecitabine [ 43 ]. An additional study reported eight deaths (21.7%) following 1 L CDK4/6i treatment; however, this study did not disclose the 2 L treatments administered to these patients [ 44 ].

Other clinical endpoints

The studies included limited information on additional clinical endpoints; two studies reported on time-to-discontinuation (TTD), two reported on duration of response (DOR), and one each on time-to-next-treatment (TTNT), time-to-progression (TTP), objective response rate (ORR), clinical benefit rate (CBR), and stable disease (Tables  2 and Table S5 ).

Safety, tolerability, and patient-reported outcomes

Safety and tolerability data were reported in two studies [ 40 , 45 ]. One study investigating 2 L administration of the chemotherapy agent eribulin mesylate reported 27 patients (22.3%) with neutropenia, 3 patients (2.5%) with febrile neutropenia, 10 patients (8.3%) with peripheral neuropathy, and 14 patients (11.6%) with diarrhea [ 45 ]. Of these, neutropenia of grade 3–4 severity occurred in 9 patients (33.3%) [ 45 ]. A total of 55 patients (45.5%) discontinued eribulin mesylate treatment; 1 patient (0.83%) discontinued treatment due to adverse events [ 45 ]. Another study reported that 5 out of the 22 patients receiving the mTORi everolimus combined with ET in 2 L (22.7%) discontinued treatment due to toxicity [ 40 ]. PROs were not reported in any of the studies included in the SLR.

The objective of this study was to summarize the existing RWE on the effectiveness and safety of therapies for patients with HR+/HER2- LABC/mBC after 1 L CDK4/6i treatment. We identified 18 unique studies reporting specifically on 2 L treatment regimens after 1 L CDK4/6i treatment. The weighted average median rwPFS for NCCN- and ESMO- guideline recommended 2 L treatments ranged from 3.6 to 3.9 months for ET-based treatments and was 6.1 months when including chemotherapy-based regimens. Treatment selection following 1 L CDK4/6i therapy remains challenging primarily due to the suboptimal effectiveness or significant toxicities (e.g., chemotherapy) associated with currently available options [ 56 ]. These results highlight that currently available 2 L treatments for patients with HR+/HER2- LABC/mBC who have received 1 L CDK4/6i are suboptimal, as evidenced by the brief median rwPFS duration associated with ET-based treatments, or notable side effects and toxicity linked to chemotherapy. This conclusion is aligned with a recent review highlighting the limited effectiveness of treatment options for HR+/HER2- LABC/mBC patients post-CDK4/6i treatment [ 56 , 57 ]. Registrational trials which have also shed light on the short median PFS of 2–3 months achieved by ET (i.e., fulvestrant) after 1 L CDK4/6i therapy emphasize the need to develop improved treatment strategies aimed at prolonging the duration of effective ET-based treatment [ 56 ].

The results of this review reveal a paucity of additional real-world effectiveness and safety evidence after 1 L CDK4/6i treatment in HR+/HER2- LABC/mBC. OS and DOR were only reported in two studies while other clinical endpoints (i.e., TTD, TTNT, TTP, ORR, CBR, and stable disease) were only reported in one study each. Similarly, safety and tolerability data were only reported in two studies each, and PROs were not reported in any study. This hindered our ability to provide a comprehensive assessment of real-world treatment effectiveness and safety following 1 L CDK4/6i treatment. The limited evidence may be due to the relatively short period of time that has elapsed since CDK4/6i first received US FDA approval for 1 L treatment of HR+/HER2- LABC/mBC (2015) [ 35 ]. As such, almost half of our evidence was informed by conference abstracts. Similarly, no real-world studies were identified in our review that reported outcomes for treatments in the third- or later-lines of therapy after 1 L CDK4/6i treatment. The lack of data in this patient population highlights a significant gap which limits our understanding of the effectiveness and safety for patients receiving later lines of therapy. As more patients receive CDK4/6i therapy in the 1 L setting, the number of patients requiring subsequent lines of therapy will continue to grow. Addressing this data gap over time will be critical to improve outcomes for patients with HR+/HER2- LABC/mBC following 1 L CDK4/6i therapy.

There are several strengths of this study, including adherence to the guidelines outlined in the Cochrane Handbook to ensure a standardized and reliable approach to the SLR [ 58 ] and reporting of the SLR following PRISMA guidelines to ensure transparency and reproducibility [ 59 ]. Furthermore, the inclusion of only RWE studies allowed us to assess the effectiveness of current standard of care treatments outside of a controlled environment and enabled us to identify an unmet need in this patient population.

This study had some notable limitations, including the lack of safety and additional effectiveness outcomes reported. In addition, the dearth of studies reporting PROs is a limitation, as PROs provide valuable insight into the patient experience and are an important aspect of assessing the impact of 2 L treatments on patients’ quality of life. The studies included in this review also lacked consistent reporting of clinical characteristics (e.g., menopausal status, sites of metastasis, prior surgery) making it challenging to draw comprehensive conclusions or comparisons based on these factors across the studies. Taken together, there exists an important gap in our understanding of the long-term management of patients with HR+/HER2- LABC/mBC. Additionally, the effectiveness results reported in our evidence base were informed by small sample sizes; many of the included studies reported median rwPFS based on less than 30 patients [ 39 , 40 , 41 , 46 , 49 , 51 , 60 ], with two studies not reporting the sample size at all [ 47 , 53 ]. This may impact the generalizability and robustness of the results. Relatedly, the SLR database search was conducted in December 2022; as such, novel agents (e.g., elacestrant and capivasertib + fulvestrant) that have since received FDA approval for the treatment of HR+/HER2- LABC/mBC may impact current 2 L rwPFS outcomes [ 61 , 62 ]. Finally, relative to the number of peer-reviewed full-text articles, this SLR identified eight abstracts and one poster presentation, comprising half (50%) of the included unique studies. As conference abstracts are inherently limited by how much content that can be described due to word limit constraints, this likely had implications on the present synthesis whereby we identified a dearth of real-world effectiveness outcomes in patients with HR+/HER2- LABC/mBC treated with 1 L CDK4/6i therapy.

Future research in this area should aim to address the limitations of the current literature and provide a more comprehensive understanding of optimal sequencing of effective and safe treatment for patients following 1 L CDK4/6i therapy. Specifically, future studies should strive to report robust data related to effectiveness, safety, and PROs for patients receiving 2 L treatment after 1 L CDK4/6i therapy. Future studies should also aim to understand the mechanism underlying CDK4/6i resistance. Addressing these gaps in knowledge may improve the long-term real-world management of patients with HR+/HER2- LABC/mBC. A future update of this synthesis may serve to capture a wider breadth of full-text, peer-reviewed articles to gain a more robust understanding of the safety, effectiveness, and real-world treatment patterns for patients with HR+/HER2- LABC/mBC. This SLR underscores the necessity for ongoing investigation and the development of innovative therapeutic approaches to address these gaps and improve patient outcomes.

This SLR qualitatively summarized the existing real-world effectiveness data for patients with HR+/HER2- LABC/mBC after 1 L CDK4/6i treatment. Results of this study highlight the limited available data and the suboptimal effectiveness of treatments employed in the 2 L setting and underscore the unmet need in this patient population. Additional studies reporting effectiveness and safety outcomes, in addition to PROs, for this patient population are necessary and should be the focus of future research.

PRISMA flow diagram. *Two included conference abstracts reported the same information as already included full-text reports, hence both conference abstracts were not identified as unique. Abbreviations: 1 L = first-line; AACR = American Association of Cancer Research; ASCO = American Society of Clinical Oncology; CDK4/6i = cyclin-dependent kinase 4/6 inhibitor; ESMO = European Society for Medical Oncology; ISPOR = Professional Society for Health Economics and Outcomes Research; n = number of studies; NMA = network meta-analysis; pts = participants; SABCS = San Antonio Breast Cancer Symposium; SLR = systematic literature review.

Number of studies reporting effectiveness outcomes exclusively for each treatment class. *Studies that lack sufficient information on effectiveness outcomes to classify based on the treatment classes outlined in the legend above. Abbreviations: CDK4/6i = cyclin-dependent kinase 4/6 inhibitor; ET = endocrine therapy; mTORi = mammalian target of rapamycin inhibitor.

Weighted average median rwPFS for 2 L treatments (recommended in ESMO/NCCN guidelines) after 1 L CDK4/6i treatment. Circular dot represents weighted average median across studies. Horizontal bars represent the range of values reported in these studies. Abbreviations: CDK4/6i = cyclin-dependent kinase 4/6 inhibitor; ESMO = European Society for Medical Oncology; ET = endocrine therapy, mTORi = mammalian target of rapamycin inhibitor; n = number of patients; NCCN = National Comprehensive Cancer Network; rwPFS = real-world progression-free survival.

Data availability

All data generated or analyzed during this study are included in this published article [and its supplementary information files]. This study is registered with PROSPERO (CRD42023383914).

Abbreviations

Second-line

Second-line treatment setting and beyond

American Association of Cancer Research

Aromatase inhibitor

American Society of Clinical Oncology

• Breast cancer

breast cancer gene/partner and localizer of BRCA2 positive

Clinical benefit rate

Cyclin-dependent kinase 4/6 inhibitor

Complete response

Duration of response

European Society for Medical Oncology

Food and Drug Administration

Human epidermal growth factor receptor 2

Human epidermal growth factor receptor 2 negative

Hormone receptor

Hormone receptor positive

Professional Society for Health Economics and Outcomes Research

Locally advanced breast cancer

Metastatic breast cancer

Medical Literature Analysis and Retrieval System Online

Medical subject headings

Mammalian target of rapamycin inhibitor

National Comprehensive Cancer Network

Newcastle Ottawa Scale

Objective response rate

Poly-ADP ribose polymerase inhibitor

Progression-free survival

Population, Intervention, Comparator, Outcome, Study Design

Partial response

Preferred Reporting Items for Systematic Literature Reviews and Meta-Analyses

Patient-reported outcomes

• Real-world evidence

San Antonio Breast Cancer Symposium

• Systematic literature review

Time-to-discontinuation

Time-to-next-treatment

Time-to-progression

United States

Łukasiewicz S, Czeczelewski M, Forma A, Baj J, Sitarz R, Stanisławek A, Breast, Cancer—Epidemiology. Risk factors, classification, prognostic markers, and current treatment Strategies—An. Updated Rev Cancers. 2021;13(17):4287.

Google Scholar  

World Health Organization (WHO). Breast Cancer Facts Sheet [updated July 12 2023. https://www.who.int/news-room/fact-sheets/detail/breast-cancer .

Arnold M, Morgan E, Rumgay H, Mafra A, Singh D, Laversanne M, et al. Current and future burden of breast cancer: global statistics for 2020 and 2040. Breast. 2022;66:15–23.

Article   PubMed   PubMed Central   Google Scholar  

Wilkinson L, Gathani T. Understanding breast cancer as a global health concern. Br J Radiol. 2022;95(1130):20211033.

Article   PubMed   Google Scholar  

Giaquinto AN, Sung H, Miller KD, Kramer JL, Newman LA, Minihan A et al. Breast Cancer Statistics, 2022. CA: A Cancer Journal for Clinicians. 2022;72(6):524– 41.

National Cancer Institute (NIH). Cancer Stat Facts: Female Breast Cancer [updated 2020. https://seer.cancer.gov/statfacts/html/breast.html .

American Cancer Society. Key Statistics for Breast Cancer [ https://www.cancer.org/cancer/types/breast-cancer/about/how-common-is-breast-cancer.html .

Zagami P, Carey LA. Triple negative breast cancer: pitfalls and progress. npj Breast Cancer. 2022;8(1):95.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Matutino A, Joy AA, Brezden-Masley C, Chia S, Verma S. Hormone receptor-positive, HER2-negative metastatic breast cancer: redrawing the lines. Curr Oncol. 2018;25(Suppl 1):S131–41.

Lloyd MR, Wander SA, Hamilton E, Razavi P, Bardia A. Next-generation selective estrogen receptor degraders and other novel endocrine therapies for management of metastatic hormone receptor-positive breast cancer: current and emerging role. Ther Adv Med Oncol. 2022;14:17588359221113694.

Cardoso F, Senkus E, Costa A, Papadopoulos E, Aapro M, André F, et al. 4th ESO-ESMO International Consensus guidelines for advanced breast Cancer (ABC 4)†. Ann Oncol. 2018;29(8):1634–57.

Article   CAS   PubMed   Google Scholar  

US Food Drug Administration. Palbociclib (Ibrance) 2017 [updated March 31, 2017. https://www.fda.gov/drugs/resources-information-approved-drugs/palbociclib-ibrance .

US Food Drug Administration. FDA expands ribociclib indication in HR-positive, HER2-negative advanced or metastatic breast cancer 2018 [updated July 18. 2018. https://www.fda.gov/drugs/resources-information-approved-drugs/fda-expands-ribociclib-indication-hr-positive-her2-negative-advanced-or-metastatic-breast-cancer .

US Food Drug Administration. FDA approves abemaciclib for HR positive, HER2-negative breast cancer 2017 [updated Sept 28. 2017. https://www.fda.gov/drugs/resources-information-approved-drugs/fda-approves-abemaciclib-hr-positive-her2-negative-breast-cancer .

NCCN Clinical Practice Guidelines in Oncology (NCCN Guidelines®). Breast Cancer 2022 [ https://www.nccn.org/professionals/physician_gls/pdf/breast.pdf .

Gennari A, André F, Barrios CH, Cortés J, de Azambuja E, DeMichele A, et al. ESMO Clinical Practice Guideline for the diagnosis, staging and treatment of patients with metastatic breast cancer. Ann Oncol. 2021;32(12):1475–95.

Beaver JA, Amiri-Kordestani L, Charlab R, Chen W, Palmby T, Tilley A, et al. FDA approval: Palbociclib for the Treatment of Postmenopausal Patients with estrogen Receptor-Positive, HER2-Negative metastatic breast Cancer. Clin Cancer Res. 2015;21(21):4760–6.

US Food Drug Administration. Ribociclib (Kisqali) [ https://www.fda.gov/drugs/resources-information-approved-drugs/ribociclib-kisqali#:~:text=On%20March%2013%2C%202017%2C%20the,hormone%20receptor%20(HR)%2Dpositive%2C .

US Food Drug Administration. FDA approves new treatment for certain advanced or metastatic breast cancers [ https://www.fda.gov/news-events/press-announcements/fda-approves-new-treatment-certain-advanced-or-metastatic-breast-cancers .

US Food Drug Administration. FDA expands ribociclib indication in HR-positive, HER2-negative advanced or metastatic breast cancer. 2018 [ https://www.fda.gov/drugs/resources-information-approved-drugs/fda-expands-ribociclib-indication-hr-positive-her2-negative-advanced-or-metastatic-breast-cancer .

US Food Drug Administration. FDA approves abemaciclib as initial therapy for HR-positive, HER2-negative metastatic breast cancer [ https://www.fda.gov/drugs/resources-information-approved-drugs/fda-approves-abemaciclib-initial-therapy-hr-positive-her2-negative-metastatic-breast-cancer .

Turner NC, Slamon DJ, Ro J, Bondarenko I, Im S-A, Masuda N, et al. Overall survival with Palbociclib and fulvestrant in advanced breast Cancer. N Engl J Med. 2018;379(20):1926–36.

Slamon DJ, Neven P, Chia S, Fasching PA, De Laurentiis M, Im SA, et al. Phase III randomized study of Ribociclib and Fulvestrant in hormone Receptor-Positive, human epidermal growth factor receptor 2-Negative advanced breast Cancer: MONALEESA-3. J Clin Oncol. 2018;36(24):2465–72.

Goetz MP, Toi M, Campone M, Sohn J, Paluch-Shimon S, Huober J, et al. MONARCH 3: Abemaciclib as initial therapy for advanced breast Cancer. J Clin Oncol. 2017;35(32):3638–46.

Gopalan PK, Villegas AG, Cao C, Pinder-Schenck M, Chiappori A, Hou W, et al. CDK4/6 inhibition stabilizes disease in patients with p16-null non-small cell lung cancer and is synergistic with mTOR inhibition. Oncotarget. 2018;9(100):37352–66.

Watt AC, Goel S. Cellular mechanisms underlying response and resistance to CDK4/6 inhibitors in the treatment of hormone receptor-positive breast cancer. Breast Cancer Res. 2022;24(1):17.

Goetz M. MONARCH 3: final overall survival results of abemaciclib plus a nonsteroidal aromatase inhibitor as first-line therapy for HR+, HER2- advanced breast cancer. SABCS; 2023.

Munzone E, Pagan E, Bagnardi V, Montagna E, Cancello G, Dellapasqua S, et al. Systematic review and meta-analysis of post-progression outcomes in ER+/HER2– metastatic breast cancer after CDK4/6 inhibitors within randomized clinical trials. ESMO Open. 2021;6(6):100332.

Gennari A, André F, Barrios CH, Cortés J, de Azambuja E, DeMichele A, et al. ESMO Clinical Practice Guideline for the diagnosis, staging and treatment of patients with metastatic breast cancer. Annals of Oncology. 2021;32(12):1475-95.

European Society for Medical Oncology (ESMO). ESMO ​Metastatic Breast Cancer Living Guideline: ER-positive HER2-negative​ Breast Cancer​ [updated May 2023. https://www.esmo.org/living-guidelines/esmo-metastatic-breast-cancer-living-guideline/er-positive-her2-negative-breast-cancer .

Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Welch PM VA, editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.2 (updated February 2021). www.training.cochrane.org/handbook : Cochrane; 2021.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. PLoS Med. 2021;18(3):e1003583.

McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS peer review of electronic search strategies: 2015 Guideline Statement. J Clin Epidemiol. 2016;75:40–6.

Fraser C, Murray A, Burr J. Identifying observational studies of surgical interventions in MEDLINE and EMBASE. BMC Med Res Methodol. 2006;6(1):41.

US Food Drug Administration. Palbociclib (Ibrance). Silver Spring, MD: US Food and Drug Administration; 2017.

Book   Google Scholar  

GA Wells BS, D O’Connell J, Peterson V, Welch M, Losos PT. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses [ https://www.ohri.ca/programs/clinical_epidemiology/oxford.asp .

Lo CK-L, Mertz D, Loeb M. Newcastle-Ottawa Scale: comparing reviewers’ to authors’ assessments. BMC Med Res Methodol. 2014;14(1):45.

Martin JM, Handorf EA, Montero AJ, Goldstein LJ. Systemic therapies following progression on first-line CDK4/6-inhibitor treatment: analysis of real-world data. Oncologist. 2022;27(6):441–6.

Kalinsky KM, Kruse M, Smyth EN, Guimaraes CM, Gautam S, Nisbett AR et al. Abstract P1-18-37: Treatment patterns and outcomes associated with sequential and non-sequential use of CDK4 and 6i for HR+, HER2- MBC in the real world. Cancer Research. 2022;82(4_Supplement):P1-18-37-P1-18-37.

Choong GM, Liddell S, Ferre RAL, O’Sullivan CC, Ruddy KJ, Haddad TC, et al. Clinical management of metastatic hormone receptor-positive, HER2-negative breast cancer (MBC) after CDK 4/6 inhibitors: a retrospective single-institution study. Breast Cancer Res Treat. 2022;196(1):229–37.

Xi J, Oza A, Thomas S, Ademuyiwa F, Weilbaecher K, Suresh R, et al. Retrospective Analysis of Treatment Patterns and effectiveness of Palbociclib and subsequent regimens in metastatic breast Cancer. J Natl Compr Canc Netw. 2019;17(2):141–7.

Rozenblit M, Mun S, Soulos P, Adelson K, Pusztai L, Mougalian S. Patterns of treatment with everolimus exemestane in hormone receptor-positive HER2-negative metastatic breast cancer in the era of targeted therapy. Breast Cancer Res. 2021;23(1):14.

Bashour SI, Doostan I, Keyomarsi K, Valero V, Ueno NT, Brown PH, et al. Rapid breast Cancer Disease Progression following cyclin dependent kinase 4 and 6 inhibitor discontinuation. J Cancer. 2017;8(11):2004–9.

Giridhar KV, Choong GM, Leon-Ferre R, O’Sullivan CC, Ruddy K, Haddad T, et al. Abstract P6-18-09: clinical management of metastatic breast cancer (MBC) after CDK 4/6 inhibitors: a retrospective single-institution study. Cancer Res. 2019;79:P6–18.

Article   Google Scholar  

Mougalian SS, Feinberg BA, Wang E, Alexis K, Chatterjee D, Knoth RL, et al. Observational study of clinical outcomes of eribulin mesylate in metastatic breast cancer after cyclin-dependent kinase 4/6 inhibitor therapy. Future Oncol. 2019;15(34):3935–44.

Moscetti LML, Riggi L, Sperduti I, Piacentini FOC, Toss A, Barbieri E, Cortesi L, Canino FMA, Zoppoli G, Frassoldati A, Schirone A, Dominici MECF. SEQUENCE OF TREATMENTS AFTER CDK4/6 THERAPY IN ADVANCED BREAST CANCER (ABC), A GOIRC MULTICENTER RETRO/ PROSPECTIVE STUDY. PRELIMINARY RESULTS IN THE RETROSPECTIVE SERIES OF 116 PATIENTS. Tumori. 2022;108(4S):80.

Menichetti AZE, Giorgi CA, Bottosso M, Leporati R, Giarratano T, Barbieri C, Ligorio F, Mioranza E, Miglietta F, Lobefaro R, Faggioni G, Falci C, Vernaci G, Di Liso E, Girardi F, Griguolo G, Vernieri C, Guarneri V, Dieci MV. CDK 4/6 INHIBITORS FOR METASTATIC BREAST CANCER: A MULTICENTER REALWORLD STUDY. Tumori. 2022;108(4S):70.

Marschner NW, Harbeck N, Thill M, Stickeler E, Zaiss M, Nusch A, et al. 232P Second-line therapies of patients with early progression under CDK4/6-inhibitor in first-line– data from the registry platform OPAL. Annals of Oncology. 2022;33:S643-S4

Gousis C, Lowe KMH, Kapiris M. V. Angelis. Beyond First Line CDK4/6 Inhibitors (CDK4/6i) and Aromatase Inhibitors (AI) in Patients with Oestrogen Receptor Positive Metastatic Breast Cancer (ERD MBC): The Guy’s Cancer Centre Experience. Clinical Oncology2022. p. e178.

Endo Y, Yoshimura A, Sawaki M, Hattori M, Kotani H, Kataoka A, et al. Time to chemotherapy for patients with estrogen receptor-positive breast Cancer and cyclin-dependent kinase 4 and 6 inhibitor use. J Breast Cancer. 2022;25(4):296–306.

Li Y, Li W, Gong C, Zheng Y, Ouyang Q, Xie N, et al. A multicenter analysis of treatment patterns and clinical outcomes of subsequent therapies after progression on palbociclib in HR+/HER2- metastatic breast cancer. Ther Adv Med Oncol. 2021;13:17588359211022890.

Amaro CP, Batra A, Lupichuk S. First-line treatment with a cyclin-dependent kinase 4/6 inhibitor plus an aromatase inhibitor for metastatic breast Cancer in Alberta. Curr Oncol. 2021;28(3):2270–80.

Crocetti SPM, Tassone L, Marcantognini G, Bastianelli L, Della Mora A, Merloni F, Cantini L, Scortichini L, Agostinelli V, Ballatore Z, Savini A, Maccaroni E. Berardi R. What is the best therapeutic sequence for ER-Positive/HER2- Negative metastatic breast cancer in the era of CDK4/6 inhibitors? A single center experience. Tumori. 2020;106(2S).

Nichetti F, Marra A, Giorgi CA, Randon G, Scagnoli S, De Angelis C, et al. 337P Efficacy of everolimus plus exemestane in CDK 4/6 inhibitors-pretreated or naïve HR-positive/HER2-negative breast cancer patients: A secondary analysis of the EVERMET study. Annals of Oncology. 2020;31:S382

Luhn P, O’Hear C, Ton T, Sanglier T, Hsieh A, Oliveri D, et al. Abstract P4-13-08: time to treatment discontinuation of second-line fulvestrant monotherapy for HR+/HER2– metastatic breast cancer in the real-world setting. Cancer Res. 2019;79(4Supplement):P4–13.

Mittal A, Molto Valiente C, Tamimi F, Schlam I, Sammons S, Tolaney SM et al. Filling the gap after CDK4/6 inhibitors: Novel Endocrine and Biologic Treatment options for metastatic hormone receptor positive breast Cancer. Cancers (Basel). 2023;15(7).

Ashai N, Swain SM. Post-CDK 4/6 inhibitor therapy: current agents and novel targets. Cancers (Basel). 2023;15(6).

Higgins JPTTJ, Chandler J, Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022). www.training.cochrane.org/handbook : Cochrane; 2022.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71.

Serdar CC, Cihan M, Yücel D, Serdar MA. Sample size, power and effect size revisited: simplified and practical approaches in pre-clinical, clinical and laboratory studies. Biochem Med (Zagreb). 2021;31(1):010502.

US Food Drug Administration. FDA approves elacestrant for ER-positive, HER2-negative, ESR1-mutated advanced or metastatic breast cancer [updated January 27 2023. https://www.fda.gov/drugs/resources-information-approved-drugs/fda-approves-elacestrant-er-positive-her2-negative-esr1-mutated-advanced-or-metastatic-breast-cancer .

US Food Drug Administration. FDA approves capivasertib with fulvestrant for breast cancer [updated November 16 2023. https://www.fda.gov/drugs/resources-information-approved-drugs/fda-approves-capivasertib-fulvestrant-breast-cancer .

Download references

Acknowledgements

The authors would like to acknowledge Joanna Bielecki who developed, conducted, and documented the database searches.

This study was funded by Pfizer Inc. (New York, NY, USA) and Arvinas (New Haven, CT, USA).

Author information

Sarah Kane, Belal Howidi, Bao-Ngoc Nguyen and Imtiaz A. Samjoo contributed equally to this work.

Authors and Affiliations

Pfizer, 10017, New York, NY, USA

Veronique Lambert & Yan Wu

EVERSANA, Burlington, ON, Canada

Sarah Kane, Belal Howidi, Bao-Ngoc Nguyen & Imtiaz A. Samjoo

Arvinas, 06511, New Haven, CT, USA

David Chandiwana & Michelle Edwards

You can also search for this author in PubMed   Google Scholar

Contributions

VL, IAS, SK, BH, BN, DC, YW, and ME participated in the conception and design of the study. IAS, SK, BH and BN contributed to the literature review, data collection, analysis, and interpretation of the data. VL, IAS, SK, BH, BN, DC, YW, and ME contributed to the interpretation of the data and critically reviewed for the importance of intellectual content for the work. VL, IAS, SK, BH, BN, DC, YW, and ME were responsible for drafting or reviewing the manuscript and for providing final approval. VL, IAS, SK, BH, BN, DC, YW, and ME meet the International Committee of Medical Journal Editors (ICMJE) criteria for authorship for this article, take responsibility for the integrity of the work, and have given their approval for this version to be published.

Corresponding author

Correspondence to Imtiaz A. Samjoo .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors of this manuscript declare that the research presented was funded by Pfizer Inc. and Arvinas. While the support from Pfizer Inc. and Arvinas was instrumental in facilitating this research, the authors affirm that their interpretation of the data and the content of this manuscript were conducted independently and without bias to maintain the transparency and integrity of the research. IAS, SK, BH, and BN are employees of EVERSANA, Canada, which was a paid consultant to Pfizer in connection with the development of this manuscript.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Supplementary material 2, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Lambert, V., Kane, S., Howidi, B. et al. Systematic literature review of real-world evidence for treatments in HR+/HER2- second-line LABC/mBC after first-line treatment with CDK4/6i. BMC Cancer 24 , 631 (2024). https://doi.org/10.1186/s12885-024-12269-8

Download citation

Received : 26 January 2024

Accepted : 16 April 2024

Published : 23 May 2024

DOI : https://doi.org/10.1186/s12885-024-12269-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• First-line CDK4/6i

ISSN: 1471-2407

• Submission enquiries: [email protected]
• General enquiries: [email protected]

A systematic literature review of empirical research on ChatGPT in education

• Open access
• Published: 26 May 2024
• Volume 3 , article number  60 , ( 2024 )

Cite this article

You have full access to this open access article

• Yazid Albadarin   ORCID: orcid.org/0009-0005-8068-8902 1 ,
• Mohammed Saqr 1 ,
• Nicolas Pope 1 &
• Markku Tukiainen 1  

166 Accesses

Explore all metrics

Over the last four decades, studies have investigated the incorporation of Artificial Intelligence (AI) into education. A recent prominent AI-powered technology that has impacted the education sector is ChatGPT. This article provides a systematic review of 14 empirical studies incorporating ChatGPT into various educational settings, published in 2022 and before the 10th of April 2023—the date of conducting the search process. It carefully followed the essential steps outlined in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA 2020) guidelines, as well as Okoli’s (Okoli in Commun Assoc Inf Syst, 2015) steps for conducting a rigorous and transparent systematic review. In this review, we aimed to explore how students and teachers have utilized ChatGPT in various educational settings, as well as the primary findings of those studies. By employing Creswell’s (Creswell in Educational research: planning, conducting, and evaluating quantitative and qualitative research [Ebook], Pearson Education, London, 2015) coding techniques for data extraction and interpretation, we sought to gain insight into their initial attempts at ChatGPT incorporation into education. This approach also enabled us to extract insights and considerations that can facilitate its effective and responsible use in future educational contexts. The results of this review show that learners have utilized ChatGPT as a virtual intelligent assistant, where it offered instant feedback, on-demand answers, and explanations of complex topics. Additionally, learners have used it to enhance their writing and language skills by generating ideas, composing essays, summarizing, translating, paraphrasing texts, or checking grammar. Moreover, learners turned to it as an aiding tool to facilitate their directed and personalized learning by assisting in understanding concepts and homework, providing structured learning plans, and clarifying assignments and tasks. However, the results of specific studies (n = 3, 21.4%) show that overuse of ChatGPT may negatively impact innovative capacities and collaborative learning competencies among learners. Educators, on the other hand, have utilized ChatGPT to create lesson plans, generate quizzes, and provide additional resources, which helped them enhance their productivity and efficiency and promote different teaching methodologies. Despite these benefits, the majority of the reviewed studies recommend the importance of conducting structured training, support, and clear guidelines for both learners and educators to mitigate the drawbacks. This includes developing critical evaluation skills to assess the accuracy and relevance of information provided by ChatGPT, as well as strategies for integrating human interaction and collaboration into learning activities that involve AI tools. Furthermore, they also recommend ongoing research and proactive dialogue with policymakers, stakeholders, and educational practitioners to refine and enhance the use of AI in learning environments. This review could serve as an insightful resource for practitioners who seek to integrate ChatGPT into education and stimulate further research in the field.

Similar content being viewed by others

Students’ voices on generative AI: perceptions, benefits, and challenges in higher education

Examining science education in chatgpt: an exploratory study of generative artificial intelligence, evolution and revolution in artificial intelligence in education.

Avoid common mistakes on your manuscript.

1 Introduction

Educational technology, a rapidly evolving field, plays a crucial role in reshaping the landscape of teaching and learning [ 82 ]. One of the most transformative technological innovations of our era that has influenced the field of education is Artificial Intelligence (AI) [ 50 ]. Over the last four decades, AI in education (AIEd) has gained remarkable attention for its potential to make significant advancements in learning, instructional methods, and administrative tasks within educational settings [ 11 ]. In particular, a large language model (LLM), a type of AI algorithm that applies artificial neural networks (ANNs) and uses massively large data sets to understand, summarize, generate, and predict new content that is almost difficult to differentiate from human creations [ 79 ], has opened up novel possibilities for enhancing various aspects of education, from content creation to personalized instruction [ 35 ]. Chatbots that leverage the capabilities of LLMs to understand and generate human-like responses have also presented the capacity to enhance student learning and educational outcomes by engaging students, offering timely support, and fostering interactive learning experiences [ 46 ].

The ongoing and remarkable technological advancements in chatbots have made their use more convenient, increasingly natural and effortless, and have expanded their potential for deployment across various domains [ 70 ]. One prominent example of chatbot applications is the Chat Generative Pre-Trained Transformer, known as ChatGPT, which was introduced by OpenAI, a leading AI research lab, on November 30th, 2022. ChatGPT employs a variety of deep learning techniques to generate human-like text, with a particular focus on recurrent neural networks (RNNs). Long short-term memory (LSTM) allows it to grasp the context of the text being processed and retain information from previous inputs. Also, the transformer architecture, a neural network architecture based on the self-attention mechanism, allows it to analyze specific parts of the input, thereby enabling it to produce more natural-sounding and coherent output. Additionally, the unsupervised generative pre-training and the fine-tuning methods allow ChatGPT to generate more relevant and accurate text for specific tasks [ 31 , 62 ]. Furthermore, reinforcement learning from human feedback (RLHF), a machine learning approach that combines reinforcement learning techniques with human-provided feedback, has helped improve ChatGPT’s model by accelerating the learning process and making it significantly more efficient.

This cutting-edge natural language processing (NLP) tool is widely recognized as one of today's most advanced LLMs-based chatbots [ 70 ], allowing users to ask questions and receive detailed, coherent, systematic, personalized, convincing, and informative human-like responses [ 55 ], even within complex and ambiguous contexts [ 63 , 77 ]. ChatGPT is considered the fastest-growing technology in history: in just three months following its public launch, it amassed an estimated 120 million monthly active users [ 16 ] with an estimated 13 million daily queries [ 49 ], surpassing all other applications [ 64 ]. This remarkable growth can be attributed to the unique features and user-friendly interface that ChatGPT offers. Its intuitive design allows users to interact seamlessly with the technology, making it accessible to a diverse range of individuals, regardless of their technical expertise [ 78 ]. Additionally, its exceptional performance results from a combination of advanced algorithms, continuous enhancements, and extensive training on a diverse dataset that includes various text sources such as books, articles, websites, and online forums [ 63 ], have contributed to a more engaging and satisfying user experience [ 62 ]. These factors collectively explain its remarkable global growth and set it apart from predecessors like Bard, Bing Chat, ERNIE, and others.

In this context, several studies have explored the technological advancements of chatbots. One noteworthy recent research effort, conducted by Schöbel et al. [ 70 ], stands out for its comprehensive analysis of more than 5,000 studies on communication agents. This study offered a comprehensive overview of the historical progression and future prospects of communication agents, including ChatGPT. Moreover, other studies have focused on making comparisons, particularly between ChatGPT and alternative chatbots like Bard, Bing Chat, ERNIE, LaMDA, BlenderBot, and various others. For example, O’Leary [ 53 ] compared two chatbots, LaMDA and BlenderBot, with ChatGPT and revealed that ChatGPT outperformed both. This superiority arises from ChatGPT’s capacity to handle a wider range of questions and generate slightly varied perspectives within specific contexts. Similarly, ChatGPT exhibited an impressive ability to formulate interpretable responses that were easily understood when compared with Google's feature snippet [ 34 ]. Additionally, ChatGPT was compared to other LLMs-based chatbots, including Bard and BERT, as well as ERNIE. The findings indicated that ChatGPT exhibited strong performance in the given tasks, often outperforming the other models [ 59 ].

Furthermore, in the education context, a comprehensive study systematically compared a range of the most promising chatbots, including Bard, Bing Chat, ChatGPT, and Ernie across a multidisciplinary test that required higher-order thinking. The study revealed that ChatGPT achieved the highest score, surpassing Bing Chat and Bard [ 64 ]. Similarly, a comparative analysis was conducted to compare ChatGPT with Bard in answering a set of 30 mathematical questions and logic problems, grouped into two question sets. Set (A) is unavailable online, while Set (B) is available online. The results revealed ChatGPT's superiority in Set (A) over Bard. Nevertheless, Bard's advantage emerged in Set (B) due to its capacity to access the internet directly and retrieve answers, a capability that ChatGPT does not possess [ 57 ]. However, through these varied assessments, ChatGPT consistently highlights its exceptional prowess compared to various alternatives in the ever-evolving chatbot technology.

The widespread adoption of chatbots, especially ChatGPT, by millions of students and educators, has sparked extensive discussions regarding its incorporation into the education sector [ 64 ]. Accordingly, many scholars have contributed to the discourse, expressing both optimism and pessimism regarding the incorporation of ChatGPT into education. For example, ChatGPT has been highlighted for its capabilities in enriching the learning and teaching experience through its ability to support different learning approaches, including adaptive learning, personalized learning, and self-directed learning [ 58 , 60 , 91 ]), deliver summative and formative feedback to students and provide real-time responses to questions, increase the accessibility of information [ 22 , 40 , 43 ], foster students’ performance, engagement and motivation [ 14 , 44 , 58 ], and enhance teaching practices [ 17 , 18 , 64 , 74 ].

On the other hand, concerns have been also raised regarding its potential negative effects on learning and teaching. These include the dissemination of false information and references [ 12 , 23 , 61 , 85 ], biased reinforcement [ 47 , 50 ], compromised academic integrity [ 18 , 40 , 66 , 74 ], and the potential decline in students' skills [ 43 , 61 , 64 , 74 ]. As a result, ChatGPT has been banned in multiple countries, including Russia, China, Venezuela, Belarus, and Iran, as well as in various educational institutions in India, Italy, Western Australia, France, and the United States [ 52 , 90 ].

Clearly, the advent of chatbots, especially ChatGPT, has provoked significant controversy due to their potential impact on learning and teaching. This indicates the necessity for further exploration to gain a deeper understanding of this technology and carefully evaluate its potential benefits, limitations, challenges, and threats to education [ 79 ]. Therefore, conducting a systematic literature review will provide valuable insights into the potential prospects and obstacles linked to its incorporation into education. This systematic literature review will primarily focus on ChatGPT, driven by the aforementioned key factors outlined above.

However, the existing literature lacks a systematic literature review of empirical studies. Thus, this systematic literature review aims to address this gap by synthesizing the existing empirical studies conducted on chatbots, particularly ChatGPT, in the field of education, highlighting how ChatGPT has been utilized in educational settings, and identifying any existing gaps. This review may be particularly useful for researchers in the field and educators who are contemplating the integration of ChatGPT or any chatbot into education. The following research questions will guide this study:

What are students' and teachers' initial attempts at utilizing ChatGPT in education?

What are the main findings derived from empirical studies that have incorporated ChatGPT into learning and teaching?

2 Methodology

To conduct this study, the authors followed the essential steps of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA 2020) and Okoli’s [ 54 ] steps for conducting a systematic review. These included identifying the study’s purpose, drafting a protocol, applying a practical screening process, searching the literature, extracting relevant data, evaluating the quality of the included studies, synthesizing the studies, and ultimately writing the review. The subsequent section provides an extensive explanation of how these steps were carried out in this study.

2.1 Identify the purpose

Given the widespread adoption of ChatGPT by students and teachers for various educational purposes, often without a thorough understanding of responsible and effective use or a clear recognition of its potential impact on learning and teaching, the authors recognized the need for further exploration of ChatGPT's impact on education in this early stage. Therefore, they have chosen to conduct a systematic literature review of existing empirical studies that incorporate ChatGPT into educational settings. Despite the limited number of empirical studies due to the novelty of the topic, their goal is to gain a deeper understanding of this technology and proactively evaluate its potential benefits, limitations, challenges, and threats to education. This effort could help to understand initial reactions and attempts at incorporating ChatGPT into education and bring out insights and considerations that can inform the future development of education.

2.2 Draft the protocol

The next step is formulating the protocol. This protocol serves to outline the study process in a rigorous and transparent manner, mitigating researcher bias in study selection and data extraction [ 88 ]. The protocol will include the following steps: generating the research question, predefining a literature search strategy, identifying search locations, establishing selection criteria, assessing the studies, developing a data extraction strategy, and creating a timeline.

2.3 Apply practical screen

The screening step aims to accurately filter the articles resulting from the searching step and select the empirical studies that have incorporated ChatGPT into educational contexts, which will guide us in answering the research questions and achieving the objectives of this study. To ensure the rigorous execution of this step, our inclusion and exclusion criteria were determined based on the authors' experience and informed by previous successful systematic reviews [ 21 ]. Table 1 summarizes the inclusion and exclusion criteria for study selection.

2.4 Literature search

We conducted a thorough literature search to identify articles that explored, examined, and addressed the use of ChatGPT in Educational contexts. We utilized two research databases: Dimensions.ai, which provides access to a large number of research publications, and lens.org, which offers access to over 300 million articles, patents, and other research outputs from diverse sources. Additionally, we included three databases, Scopus, Web of Knowledge, and ERIC, which contain relevant research on the topic that addresses our research questions. To browse and identify relevant articles, we used the following search formula: ("ChatGPT" AND "Education"), which included the Boolean operator "AND" to get more specific results. The subject area in the Scopus and ERIC databases were narrowed to "ChatGPT" and "Education" keywords, and in the WoS database was limited to the "Education" category. The search was conducted between the 3rd and 10th of April 2023, which resulted in 276 articles from all selected databases (111 articles from Dimensions.ai, 65 from Scopus, 28 from Web of Science, 14 from ERIC, and 58 from Lens.org). These articles were imported into the Rayyan web-based system for analysis. The duplicates were identified automatically by the system. Subsequently, the first author manually reviewed the duplicated articles ensured that they had the same content, and then removed them, leaving us with 135 unique articles. Afterward, the titles, abstracts, and keywords of the first 40 manuscripts were scanned and reviewed by the first author and were discussed with the second and third authors to resolve any disagreements. Subsequently, the first author proceeded with the filtering process for all articles and carefully applied the inclusion and exclusion criteria as presented in Table  1 . Articles that met any one of the exclusion criteria were eliminated, resulting in 26 articles. Afterward, the authors met to carefully scan and discuss them. The authors agreed to eliminate any empirical studies solely focused on checking ChatGPT capabilities, as these studies do not guide us in addressing the research questions and achieving the study's objectives. This resulted in 14 articles eligible for analysis.

2.5 Quality appraisal

The examination and evaluation of the quality of the extracted articles is a vital step [ 9 ]. Therefore, the extracted articles were carefully evaluated for quality using Fink’s [ 24 ] standards, which emphasize the necessity for detailed descriptions of methodology, results, conclusions, strengths, and limitations. The process began with a thorough assessment of each study's design, data collection, and analysis methods to ensure their appropriateness and comprehensive execution. The clarity, consistency, and logical progression from data to results and conclusions were also critically examined. Potential biases and recognized limitations within the studies were also scrutinized. Ultimately, two articles were excluded for failing to meet Fink’s criteria, particularly in providing sufficient detail on methodology, results, conclusions, strengths, or limitations. The review process is illustrated in Fig.  1 .

The study selection process

2.6 Data extraction

The next step is data extraction, the process of capturing the key information and categories from the included studies. To improve efficiency, reduce variation among authors, and minimize errors in data analysis, the coding categories were constructed using Creswell's [ 15 ] coding techniques for data extraction and interpretation. The coding process involves three sequential steps. The initial stage encompasses open coding , where the researcher examines the data, generates codes to describe and categorize it, and gains a deeper understanding without preconceived ideas. Following open coding is axial coding , where the interrelationships between codes from open coding are analyzed to establish more comprehensive categories or themes. The process concludes with selective coding , refining and integrating categories or themes to identify core concepts emerging from the data. The first coder performed the coding process, then engaged in discussions with the second and third authors to finalize the coding categories for the first five articles. The first coder then proceeded to code all studies and engaged again in discussions with the other authors to ensure the finalization of the coding process. After a comprehensive analysis and capturing of the key information from the included studies, the data extraction and interpretation process yielded several themes. These themes have been categorized and are presented in Table  2 . It is important to note that open coding results were removed from Table  2 for aesthetic reasons, as it included many generic aspects, such as words, short phrases, or sentences mentioned in the studies.

2.7 Synthesize studies

In this stage, we will gather, discuss, and analyze the key findings that emerged from the selected studies. The synthesis stage is considered a transition from an author-centric to a concept-centric focus, enabling us to map all the provided information to achieve the most effective evaluation of the data [ 87 ]. Initially, the authors extracted data that included general information about the selected studies, including the author(s)' names, study titles, years of publication, educational levels, research methodologies, sample sizes, participants, main aims or objectives, raw data sources, and analysis methods. Following that, all key information and significant results from the selected studies were compiled using Creswell’s [ 15 ] coding techniques for data extraction and interpretation to identify core concepts and themes emerging from the data, focusing on those that directly contributed to our research questions and objectives, such as the initial utilization of ChatGPT in learning and teaching, learners' and educators' familiarity with ChatGPT, and the main findings of each study. Finally, the data related to each selected study were extracted into an Excel spreadsheet for data processing. The Excel spreadsheet was reviewed by the authors, including a series of discussions to ensure the finalization of this process and prepare it for further analysis. Afterward, the final result being analyzed and presented in various types of charts and graphs. Table 4 presents the extracted data from the selected studies, with each study labeled with a capital 'S' followed by a number.

This section consists of two main parts. The first part provides a descriptive analysis of the data compiled from the reviewed studies. The second part presents the answers to the research questions and the main findings of these studies.

3.1 Part 1: descriptive analysis

This section will provide a descriptive analysis of the reviewed studies, including educational levels and fields, participants distribution, country contribution, research methodologies, study sample size, study population, publication year, list of journals, familiarity with ChatGPT, source of data, and the main aims and objectives of the studies. Table 4 presents a comprehensive overview of the extracted data from the selected studies.

3.1.1 The number of the reviewed studies and publication years

The total number of the reviewed studies was 14. All studies were empirical studies and published in different journals focusing on Education and Technology. One study was published in 2022 [S1], while the remaining were published in 2023 [S2]-[S14]. Table 3 illustrates the year of publication, the names of the journals, and the number of reviewed studies published in each journal for the studies reviewed.

3.1.2 Educational levels and fields

The majority of the reviewed studies, 11 studies, were conducted in higher education institutions [S1]-[S10] and [S13]. Two studies did not specify the educational level of the population [S12] and [S14], while one study focused on elementary education [S11]. However, the reviewed studies covered various fields of education. Three studies focused on Arts and Humanities Education [S8], [S11], and [S14], specifically English Education. Two studies focused on Engineering Education, with one in Computer Engineering [S2] and the other in Construction Education [S3]. Two studies focused on Mathematics Education [S5] and [S12]. One study focused on Social Science Education [S13]. One study focused on Early Education [S4]. One study focused on Journalism Education [S9]. Finally, three studies did not specify the field of education [S1], [S6], and [S7]. Figure  2 represents the educational levels in the reviewed studies, while Fig.  3 represents the context of the reviewed studies.

Educational levels in the reviewed studies

Context of the reviewed studies

3.1.3 Participants distribution and countries contribution

The reviewed studies have been conducted across different geographic regions, providing a diverse representation of the studies. The majority of the studies, 10 in total, [S1]-[S3], [S5]-[S9], [S11], and [S14], primarily focused on participants from single countries such as Pakistan, the United Arab Emirates, China, Indonesia, Poland, Saudi Arabia, South Korea, Spain, Tajikistan, and the United States. In contrast, four studies, [S4], [S10], [S12], and [S13], involved participants from multiple countries, including China and the United States [S4], China, the United Kingdom, and the United States [S10], the United Arab Emirates, Oman, Saudi Arabia, and Jordan [S12], Turkey, Sweden, Canada, and Australia [ 13 ]. Figures  4 and 5 illustrate the distribution of participants, whether from single or multiple countries, and the contribution of each country in the reviewed studies, respectively.

The reviewed studies conducted in single or multiple countries

The Contribution of each country in the studies

3.1.4 Study population and sample size

Four study populations were included: university students, university teachers, university teachers and students, and elementary school teachers. Six studies involved university students [S2], [S3], [S5] and [S6]-[S8]. Three studies focused on university teachers [S1], [S4], and [S6], while one study specifically targeted elementary school teachers [S11]. Additionally, four studies included both university teachers and students [S10] and [ 12 , 13 , 14 ], and among them, study [S13] specifically included postgraduate students. In terms of the sample size of the reviewed studies, nine studies included a small sample size of less than 50 participants [S1], [S3], [S6], [S8], and [S10]-[S13]. Three studies had 50–100 participants [S2], [S9], and [S14]. Only one study had more than 100 participants [S7]. It is worth mentioning that study [S4] adopted a mixed methods approach, including 10 participants for qualitative analysis and 110 participants for quantitative analysis.

3.1.5 Participants’ familiarity with using ChatGPT

The reviewed studies recruited a diverse range of participants with varying levels of familiarity with ChatGPT. Five studies [S2], [S4], [S6], [S8], and [S12] involved participants already familiar with ChatGPT, while eight studies [S1], [S3], [S5], [S7], [S9], [S10], [S13] and [S14] included individuals with differing levels of familiarity. Notably, one study [S11] had participants who were entirely unfamiliar with ChatGPT. It is important to note that four studies [S3], [S5], [S9], and [S11] provided training or guidance to their participants before conducting their studies, while ten studies [S1], [S2], [S4], [S6]-[S8], [S10], and [S12]-[S14] did not provide training due to the participants' existing familiarity with ChatGPT.

3.1.6 Research methodology approaches and source(S) of data

The reviewed studies adopted various research methodology approaches. Seven studies adopted qualitative research methodology [S1], [S4], [S6], [S8], [S10], [S11], and [S12], while three studies adopted quantitative research methodology [S3], [S7], and [S14], and four studies employed mixed-methods, which involved a combination of both the strengths of qualitative and quantitative methods [S2], [S5], [S9], and [S13].

In terms of the source(s) of data, the reviewed studies obtained their data from various sources, such as interviews, questionnaires, and pre-and post-tests. Six studies relied on interviews as their primary source of data collection [S1], [S4], [S6], [S10], [S11], and [S12], four studies relied on questionnaires [S2], [S7], [S13], and [S14], two studies combined the use of pre-and post-tests and questionnaires for data collection [S3] and [S9], while two studies combined the use of questionnaires and interviews to obtain the data [S5] and [S8]. It is important to note that six of the reviewed studies were quasi-experimental [S3], [S5], [S8], [S9], [S12], and [S14], while the remaining ones were experimental studies [S1], [S2], [S4], [S6], [S7], [S10], [S11], and [S13]. Figures  6 and 7 illustrate the research methodologies and the source (s) of data used in the reviewed studies, respectively.

Research methodologies in the reviewed studies

Source of data in the reviewed studies

3.1.7 The aim and objectives of the studies

The reviewed studies encompassed a diverse set of aims, with several of them incorporating multiple primary objectives. Six studies [S3], [S6], [S7], [S8], [S11], and [S12] examined the integration of ChatGPT in educational contexts, and four studies [S4], [S5], [S13], and [S14] investigated the various implications of its use in education, while three studies [S2], [S9], and [S10] aimed to explore both its integration and implications in education. Additionally, seven studies explicitly explored attitudes and perceptions of students [S2] and [S3], educators [S1] and [S6], or both [S10], [S12], and [S13] regarding the utilization of ChatGPT in educational settings.

3.2 Part 2: research questions and main findings of the reviewed studies

This part will present the answers to the research questions and the main findings of the reviewed studies, classified into two main categories (learning and teaching) according to AI Education classification by [ 36 ]. Figure  8 summarizes the main findings of the reviewed studies in a visually informative diagram. Table 4 provides a detailed list of the key information extracted from the selected studies that led to generating these themes.

The main findings in the reviewed studies

4 Students' initial attempts at utilizing ChatGPT in learning and main findings from students' perspective

4.1 virtual intelligent assistant.

Nine studies demonstrated that ChatGPT has been utilized by students as an intelligent assistant to enhance and support their learning. Students employed it for various purposes, such as answering on-demand questions [S2]-[S5], [S8], [S10], and [S12], providing valuable information and learning resources [S2]-[S5], [S6], and [S8], as well as receiving immediate feedback [S2], [S4], [S9], [S10], and [S12]. In this regard, students generally were confident in the accuracy of ChatGPT's responses, considering them relevant, reliable, and detailed [S3], [S4], [S5], and [S8]. However, some students indicated the need for improvement, as they found that answers are not always accurate [S2], and that misleading information may have been provided or that it may not always align with their expectations [S6] and [S10]. It was also observed by the students that the accuracy of ChatGPT is dependent on several factors, including the quality and specificity of the user's input, the complexity of the question or topic, and the scope and relevance of its training data [S12]. Many students felt that ChatGPT's answers were not always accurate and most of them believed that it requires good background knowledge to work with.

4.2 Writing and language proficiency assistant

Six of the reviewed studies highlighted that ChatGPT has been utilized by students as a valuable assistant tool to improve their academic writing skills and language proficiency. Among these studies, three mainly focused on English education, demonstrating that students showed sufficient mastery in using ChatGPT for generating ideas, summarizing, paraphrasing texts, and completing writing essays [S8], [S11], and [S14]. Furthermore, ChatGPT helped them in writing by making students active investigators rather than passive knowledge recipients and facilitated the development of their writing skills [S11] and [S14]. Similarly, ChatGPT allowed students to generate unique ideas and perspectives, leading to deeper analysis and reflection on their journalism writing [S9]. In terms of language proficiency, ChatGPT allowed participants to translate content into their home languages, making it more accessible and relevant to their context [S4]. It also enabled them to request changes in linguistic tones or flavors [S8]. Moreover, participants used it to check grammar or as a dictionary [S11].

4.3 Valuable resource for learning approaches

Five studies demonstrated that students used ChatGPT as a valuable complementary resource for self-directed learning. It provided learning resources and guidance on diverse educational topics and created a supportive home learning environment [S2] and [S4]. Moreover, it offered step-by-step guidance to grasp concepts at their own pace and enhance their understanding [S5], streamlined task and project completion carried out independently [S7], provided comprehensive and easy-to-understand explanations on various subjects [S10], and assisted in studying geometry operations, thereby empowering them to explore geometry operations at their own pace [S12]. Three studies showed that students used ChatGPT as a valuable learning resource for personalized learning. It delivered age-appropriate conversations and tailored teaching based on a child's interests [S4], acted as a personalized learning assistant, adapted to their needs and pace, which assisted them in understanding mathematical concepts [S12], and enabled personalized learning experiences in social sciences by adapting to students' needs and learning styles [S13]. On the other hand, it is important to note that, according to one study [S5], students suggested that using ChatGPT may negatively affect collaborative learning competencies between students.

4.4 Enhancing students' competencies

Six of the reviewed studies have shown that ChatGPT is a valuable tool for improving a wide range of skills among students. Two studies have provided evidence that ChatGPT led to improvements in students' critical thinking, reasoning skills, and hazard recognition competencies through engaging them in interactive conversations or activities and providing responses related to their disciplines in journalism [S5] and construction education [S9]. Furthermore, two studies focused on mathematical education have shown the positive impact of ChatGPT on students' problem-solving abilities in unraveling problem-solving questions [S12] and enhancing the students' understanding of the problem-solving process [S5]. Lastly, one study indicated that ChatGPT effectively contributed to the enhancement of conversational social skills [S4].

4.5 Supporting students' academic success

Seven of the reviewed studies highlighted that students found ChatGPT to be beneficial for learning as it enhanced learning efficiency and improved the learning experience. It has been observed to improve students' efficiency in computer engineering studies by providing well-structured responses and good explanations [S2]. Additionally, students found it extremely useful for hazard reporting [S3], and it also enhanced their efficiency in solving mathematics problems and capabilities [S5] and [S12]. Furthermore, by finding information, generating ideas, translating texts, and providing alternative questions, ChatGPT aided students in deepening their understanding of various subjects [S6]. It contributed to an increase in students' overall productivity [S7] and improved efficiency in composing written tasks [S8]. Regarding learning experiences, ChatGPT was instrumental in assisting students in identifying hazards that they might have otherwise overlooked [S3]. It also improved students' learning experiences in solving mathematics problems and developing abilities [S5] and [S12]. Moreover, it increased students' successful completion of important tasks in their studies [S7], particularly those involving average difficulty writing tasks [S8]. Additionally, ChatGPT increased the chances of educational success by providing students with baseline knowledge on various topics [S10].

5 Teachers' initial attempts at utilizing ChatGPT in teaching and main findings from teachers' perspective

5.1 valuable resource for teaching.

The reviewed studies showed that teachers have employed ChatGPT to recommend, modify, and generate diverse, creative, organized, and engaging educational contents, teaching materials, and testing resources more rapidly [S4], [S6], [S10] and [S11]. Additionally, teachers experienced increased productivity as ChatGPT facilitated quick and accurate responses to questions, fact-checking, and information searches [S1]. It also proved valuable in constructing new knowledge [S6] and providing timely answers to students' questions in classrooms [S11]. Moreover, ChatGPT enhanced teachers' efficiency by generating new ideas for activities and preplanning activities for their students [S4] and [S6], including interactive language game partners [S11].

5.2 Improving productivity and efficiency

The reviewed studies showed that participants' productivity and work efficiency have been significantly enhanced by using ChatGPT as it enabled them to allocate more time to other tasks and reduce their overall workloads [S6], [S10], [S11], [S13], and [S14]. However, three studies [S1], [S4], and [S11], indicated a negative perception and attitude among teachers toward using ChatGPT. This negativity stemmed from a lack of necessary skills to use it effectively [S1], a limited familiarity with it [S4], and occasional inaccuracies in the content provided by it [S10].

5.3 Catalyzing new teaching methodologies

Five of the reviewed studies highlighted that educators found the necessity of redefining their teaching profession with the assistance of ChatGPT [S11], developing new effective learning strategies [S4], and adapting teaching strategies and methodologies to ensure the development of essential skills for future engineers [S5]. They also emphasized the importance of adopting new educational philosophies and approaches that can evolve with the introduction of ChatGPT into the classroom [S12]. Furthermore, updating curricula to focus on improving human-specific features, such as emotional intelligence, creativity, and philosophical perspectives [S13], was found to be essential.

5.4 Effective utilization of CHATGPT in teaching

According to the reviewed studies, effective utilization of ChatGPT in education requires providing teachers with well-structured training, support, and adequate background on how to use ChatGPT responsibly [S1], [S3], [S11], and [S12]. Establishing clear rules and regulations regarding its usage is essential to ensure it positively impacts the teaching and learning processes, including students' skills [S1], [S4], [S5], [S8], [S9], and [S11]-[S14]. Moreover, conducting further research and engaging in discussions with policymakers and stakeholders is indeed crucial for the successful integration of ChatGPT in education and to maximize the benefits for both educators and students [S1], [S6]-[S10], and [S12]-[S14].

6 Discussion

The purpose of this review is to conduct a systematic review of empirical studies that have explored the utilization of ChatGPT, one of today’s most advanced LLM-based chatbots, in education. The findings of the reviewed studies showed several ways of ChatGPT utilization in different learning and teaching practices as well as it provided insights and considerations that can facilitate its effective and responsible use in future educational contexts. The results of the reviewed studies came from diverse fields of education, which helped us avoid a biased review that is limited to a specific field. Similarly, the reviewed studies have been conducted across different geographic regions. This kind of variety in geographic representation enriched the findings of this review.

In response to RQ1 , "What are students' and teachers' initial attempts at utilizing ChatGPT in education?", the findings from this review provide comprehensive insights. Chatbots, including ChatGPT, play a crucial role in supporting student learning, enhancing their learning experiences, and facilitating diverse learning approaches [ 42 , 43 ]. This review found that this tool, ChatGPT, has been instrumental in enhancing students' learning experiences by serving as a virtual intelligent assistant, providing immediate feedback, on-demand answers, and engaging in educational conversations. Additionally, students have benefited from ChatGPT’s ability to generate ideas, compose essays, and perform tasks like summarizing, translating, paraphrasing texts, or checking grammar, thereby enhancing their writing and language competencies. Furthermore, students have turned to ChatGPT for assistance in understanding concepts and homework, providing structured learning plans, and clarifying assignments and tasks, which fosters a supportive home learning environment, allowing them to take responsibility for their own learning and cultivate the skills and approaches essential for supportive home learning environment [ 26 , 27 , 28 ]. This finding aligns with the study of Saqr et al. [ 68 , 69 ] who highlighted that, when students actively engage in their own learning process, it yields additional advantages, such as heightened motivation, enhanced achievement, and the cultivation of enthusiasm, turning them into advocates for their own learning.

Moreover, students have utilized ChatGPT for tailored teaching and step-by-step guidance on diverse educational topics, streamlining task and project completion, and generating and recommending educational content. This personalization enhances the learning environment, leading to increased academic success. This finding aligns with other recent studies [ 26 , 27 , 28 , 60 , 66 ] which revealed that ChatGPT has the potential to offer personalized learning experiences and support an effective learning process by providing students with customized feedback and explanations tailored to their needs and abilities. Ultimately, fostering students' performance, engagement, and motivation, leading to increase students' academic success [ 14 , 44 , 58 ]. This ultimate outcome is in line with the findings of Saqr et al. [ 68 , 69 ], which emphasized that learning strategies are important catalysts of students' learning, as students who utilize effective learning strategies are more likely to have better academic achievement.

Teachers, too, have capitalized on ChatGPT's capabilities to enhance productivity and efficiency, using it for creating lesson plans, generating quizzes, providing additional resources, generating and preplanning new ideas for activities, and aiding in answering students’ questions. This adoption of technology introduces new opportunities to support teaching and learning practices, enhancing teacher productivity. This finding aligns with those of Day [ 17 ], De Castro [ 18 ], and Su and Yang [ 74 ] as well as with those of Valtonen et al. [ 82 ], who revealed that emerging technological advancements have opened up novel opportunities and means to support teaching and learning practices, and enhance teachers’ productivity.

In response to RQ2 , "What are the main findings derived from empirical studies that have incorporated ChatGPT into learning and teaching?", the findings from this review provide profound insights and raise significant concerns. Starting with the insights, chatbots, including ChatGPT, have demonstrated the potential to reshape and revolutionize education, creating new, novel opportunities for enhancing the learning process and outcomes [ 83 ], facilitating different learning approaches, and offering a range of pedagogical benefits [ 19 , 43 , 72 ]. In this context, this review found that ChatGPT could open avenues for educators to adopt or develop new effective learning and teaching strategies that can evolve with the introduction of ChatGPT into the classroom. Nonetheless, there is an evident lack of research understanding regarding the potential impact of generative machine learning models within diverse educational settings [ 83 ]. This necessitates teachers to attain a high level of proficiency in incorporating chatbots, such as ChatGPT, into their classrooms to create inventive, well-structured, and captivating learning strategies. In the same vein, the review also found that teachers without the requisite skills to utilize ChatGPT realized that it did not contribute positively to their work and could potentially have adverse effects [ 37 ]. This concern could lead to inequity of access to the benefits of chatbots, including ChatGPT, as individuals who lack the necessary expertise may not be able to harness their full potential, resulting in disparities in educational outcomes and opportunities. Therefore, immediate action is needed to address these potential issues. A potential solution is offering training, support, and competency development for teachers to ensure that all of them can leverage chatbots, including ChatGPT, effectively and equitably in their educational practices [ 5 , 28 , 80 ], which could enhance accessibility and inclusivity, and potentially result in innovative outcomes [ 82 , 83 ].

Additionally, chatbots, including ChatGPT, have the potential to significantly impact students' thinking abilities, including retention, reasoning, analysis skills [ 19 , 45 ], and foster innovation and creativity capabilities [ 83 ]. This review found that ChatGPT could contribute to improving a wide range of skills among students. However, it found that frequent use of ChatGPT may result in a decrease in innovative capacities, collaborative skills and cognitive capacities, and students' motivation to attend classes, as well as could lead to reduced higher-order thinking skills among students [ 22 , 29 ]. Therefore, immediate action is needed to carefully examine the long-term impact of chatbots such as ChatGPT, on learning outcomes as well as to explore its incorporation into educational settings as a supportive tool without compromising students' cognitive development and critical thinking abilities. In the same vein, the review also found that it is challenging to draw a consistent conclusion regarding the potential of ChatGPT to aid self-directed learning approach. This finding aligns with the recent study of Baskara [ 8 ]. Therefore, further research is needed to explore the potential of ChatGPT for self-directed learning. One potential solution involves utilizing learning analytics as a novel approach to examine various aspects of students' learning and support them in their individual endeavors [ 32 ]. This approach can bridge this gap by facilitating an in-depth analysis of how learners engage with ChatGPT, identifying trends in self-directed learning behavior, and assessing its influence on their outcomes.

Turning to the significant concerns, on the other hand, a fundamental challenge with LLM-based chatbots, including ChatGPT, is the accuracy and quality of the provided information and responses, as they provide false information as truth—a phenomenon often referred to as "hallucination" [ 3 , 49 ]. In this context, this review found that the provided information was not entirely satisfactory. Consequently, the utilization of chatbots presents potential concerns, such as generating and providing inaccurate or misleading information, especially for students who utilize it to support their learning. This finding aligns with other findings [ 6 , 30 , 35 , 40 ] which revealed that incorporating chatbots such as ChatGPT, into education presents challenges related to its accuracy and reliability due to its training on a large corpus of data, which may contain inaccuracies and the way users formulate or ask ChatGPT. Therefore, immediate action is needed to address these potential issues. One possible solution is to equip students with the necessary skills and competencies, which include a background understanding of how to use it effectively and the ability to assess and evaluate the information it generates, as the accuracy and the quality of the provided information depend on the input, its complexity, the topic, and the relevance of its training data [ 28 , 49 , 86 ]. However, it's also essential to examine how learners can be educated about how these models operate, the data used in their training, and how to recognize their limitations, challenges, and issues [ 79 ].

Furthermore, chatbots present a substantial challenge concerning maintaining academic integrity [ 20 , 56 ] and copyright violations [ 83 ], which are significant concerns in education. The review found that the potential misuse of ChatGPT might foster cheating, facilitate plagiarism, and threaten academic integrity. This issue is also affirmed by the research conducted by Basic et al. [ 7 ], who presented evidence that students who utilized ChatGPT in their writing assignments had more plagiarism cases than those who did not. These findings align with the conclusions drawn by Cotton et al. [ 13 ], Hisan and Amri [ 33 ] and Sullivan et al. [ 75 ], who revealed that the integration of chatbots such as ChatGPT into education poses a significant challenge to the preservation of academic integrity. Moreover, chatbots, including ChatGPT, have increased the difficulty in identifying plagiarism [ 47 , 67 , 76 ]. The findings from previous studies [ 1 , 84 ] indicate that AI-generated text often went undetected by plagiarism software, such as Turnitin. However, Turnitin and other similar plagiarism detection tools, such as ZeroGPT, GPTZero, and Copyleaks, have since evolved, incorporating enhanced techniques to detect AI-generated text, despite the possibility of false positives, as noted in different studies that have found these tools still not yet fully ready to accurately and reliably identify AI-generated text [ 10 , 51 ], and new novel detection methods may need to be created and implemented for AI-generated text detection [ 4 ]. This potential issue could lead to another concern, which is the difficulty of accurately evaluating student performance when they utilize chatbots such as ChatGPT assistance in their assignments. Consequently, the most LLM-driven chatbots present a substantial challenge to traditional assessments [ 64 ]. The findings from previous studies indicate the importance of rethinking, improving, and redesigning innovative assessment methods in the era of chatbots [ 14 , 20 , 64 , 75 ]. These methods should prioritize the process of evaluating students' ability to apply knowledge to complex cases and demonstrate comprehension, rather than solely focusing on the final product for assessment. Therefore, immediate action is needed to address these potential issues. One possible solution would be the development of clear guidelines, regulatory policies, and pedagogical guidance. These measures would help regulate the proper and ethical utilization of chatbots, such as ChatGPT, and must be established before their introduction to students [ 35 , 38 , 39 , 41 , 89 ].

In summary, our review has delved into the utilization of ChatGPT, a prominent example of chatbots, in education, addressing the question of how ChatGPT has been utilized in education. However, there remain significant gaps, which necessitate further research to shed light on this area.

7 Conclusions

This systematic review has shed light on the varied initial attempts at incorporating ChatGPT into education by both learners and educators, while also offering insights and considerations that can facilitate its effective and responsible use in future educational contexts. From the analysis of 14 selected studies, the review revealed the dual-edged impact of ChatGPT in educational settings. On the positive side, ChatGPT significantly aided the learning process in various ways. Learners have used it as a virtual intelligent assistant, benefiting from its ability to provide immediate feedback, on-demand answers, and easy access to educational resources. Additionally, it was clear that learners have used it to enhance their writing and language skills, engaging in practices such as generating ideas, composing essays, and performing tasks like summarizing, translating, paraphrasing texts, or checking grammar. Importantly, other learners have utilized it in supporting and facilitating their directed and personalized learning on a broad range of educational topics, assisting in understanding concepts and homework, providing structured learning plans, and clarifying assignments and tasks. Educators, on the other hand, found ChatGPT beneficial for enhancing productivity and efficiency. They used it for creating lesson plans, generating quizzes, providing additional resources, and answers learners' questions, which saved time and allowed for more dynamic and engaging teaching strategies and methodologies.

However, the review also pointed out negative impacts. The results revealed that overuse of ChatGPT could decrease innovative capacities and collaborative learning among learners. Specifically, relying too much on ChatGPT for quick answers can inhibit learners' critical thinking and problem-solving skills. Learners might not engage deeply with the material or consider multiple solutions to a problem. This tendency was particularly evident in group projects, where learners preferred consulting ChatGPT individually for solutions over brainstorming and collaborating with peers, which negatively affected their teamwork abilities. On a broader level, integrating ChatGPT into education has also raised several concerns, including the potential for providing inaccurate or misleading information, issues of inequity in access, challenges related to academic integrity, and the possibility of misusing the technology.

Accordingly, this review emphasizes the urgency of developing clear rules, policies, and regulations to ensure ChatGPT's effective and responsible use in educational settings, alongside other chatbots, by both learners and educators. This requires providing well-structured training to educate them on responsible usage and understanding its limitations, along with offering sufficient background information. Moreover, it highlights the importance of rethinking, improving, and redesigning innovative teaching and assessment methods in the era of ChatGPT. Furthermore, conducting further research and engaging in discussions with policymakers and stakeholders are essential steps to maximize the benefits for both educators and learners and ensure academic integrity.

It is important to acknowledge that this review has certain limitations. Firstly, the limited inclusion of reviewed studies can be attributed to several reasons, including the novelty of the technology, as new technologies often face initial skepticism and cautious adoption; the lack of clear guidelines or best practices for leveraging this technology for educational purposes; and institutional or governmental policies affecting the utilization of this technology in educational contexts. These factors, in turn, have affected the number of studies available for review. Secondly, the utilization of the original version of ChatGPT, based on GPT-3 or GPT-3.5, implies that new studies utilizing the updated version, GPT-4 may lead to different findings. Therefore, conducting follow-up systematic reviews is essential once more empirical studies on ChatGPT are published. Additionally, long-term studies are necessary to thoroughly examine and assess the impact of ChatGPT on various educational practices.

Despite these limitations, this systematic review has highlighted the transformative potential of ChatGPT in education, revealing its diverse utilization by learners and educators alike and summarized the benefits of incorporating it into education, as well as the forefront critical concerns and challenges that must be addressed to facilitate its effective and responsible use in future educational contexts. This review could serve as an insightful resource for practitioners who seek to integrate ChatGPT into education and stimulate further research in the field.

Data availability

The data supporting our findings are available upon request.

Abbreviations

• Artificial intelligence

AI in education

Large language model

Artificial neural networks

Chat Generative Pre-Trained Transformer

Recurrent neural networks

Long short-term memory

Reinforcement learning from human feedback

Natural language processing

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

AlAfnan MA, Dishari S, Jovic M, Lomidze K. ChatGPT as an educational tool: opportunities, challenges, and recommendations for communication, business writing, and composition courses. J Artif Intell Technol. 2023. https://doi.org/10.37965/jait.2023.0184 .

Article   Google Scholar  

Ali JKM, Shamsan MAA, Hezam TA, Mohammed AAQ. Impact of ChatGPT on learning motivation. J Engl Stud Arabia Felix. 2023;2(1):41–9. https://doi.org/10.56540/jesaf.v2i1.51 .

Alkaissi H, McFarlane SI. Artificial hallucinations in ChatGPT: implications in scientific writing. Cureus. 2023. https://doi.org/10.7759/cureus.35179 .

Anderson N, Belavý DL, Perle SM, Hendricks S, Hespanhol L, Verhagen E, Memon AR. AI did not write this manuscript, or did it? Can we trick the AI text detector into generated texts? The potential future of ChatGPT and AI in sports & exercise medicine manuscript generation. BMJ Open Sport Exerc Med. 2023;9(1): e001568. https://doi.org/10.1136/bmjsem-2023-001568 .

Ausat AMA, Massang B, Efendi M, Nofirman N, Riady Y. Can chat GPT replace the role of the teacher in the classroom: a fundamental analysis. J Educ. 2023;5(4):16100–6.

Google Scholar  

Baidoo-Anu D, Ansah L. Education in the Era of generative artificial intelligence (AI): understanding the potential benefits of ChatGPT in promoting teaching and learning. Soc Sci Res Netw. 2023. https://doi.org/10.2139/ssrn.4337484 .

Basic Z, Banovac A, Kruzic I, Jerkovic I. Better by you, better than me, chatgpt3 as writing assistance in students essays. 2023. arXiv preprint arXiv:2302.04536 .‏

Baskara FR. The promises and pitfalls of using chat GPT for self-determined learning in higher education: an argumentative review. Prosiding Seminar Nasional Fakultas Tarbiyah dan Ilmu Keguruan IAIM Sinjai. 2023;2:95–101. https://doi.org/10.47435/sentikjar.v2i0.1825 .

Behera RK, Bala PK, Dhir A. The emerging role of cognitive computing in healthcare: a systematic literature review. Int J Med Inform. 2019;129:154–66. https://doi.org/10.1016/j.ijmedinf.2019.04.024 .

Chaka C. Detecting AI content in responses generated by ChatGPT, YouChat, and Chatsonic: the case of five AI content detection tools. J Appl Learn Teach. 2023. https://doi.org/10.37074/jalt.2023.6.2.12 .

Chiu TKF, Xia Q, Zhou X, Chai CS, Cheng M. Systematic literature review on opportunities, challenges, and future research recommendations of artificial intelligence in education. Comput Educ Artif Intell. 2023;4:100118. https://doi.org/10.1016/j.caeai.2022.100118 .

Choi EPH, Lee JJ, Ho M, Kwok JYY, Lok KYW. Chatting or cheating? The impacts of ChatGPT and other artificial intelligence language models on nurse education. Nurse Educ Today. 2023;125:105796. https://doi.org/10.1016/j.nedt.2023.105796 .

Cotton D, Cotton PA, Shipway JR. Chatting and cheating: ensuring academic integrity in the era of ChatGPT. Innov Educ Teach Int. 2023. https://doi.org/10.1080/14703297.2023.2190148 .

Crawford J, Cowling M, Allen K. Leadership is needed for ethical ChatGPT: Character, assessment, and learning using artificial intelligence (AI). J Univ Teach Learn Pract. 2023. https://doi.org/10.53761/1.20.3.02 .

Creswell JW. Educational research: planning, conducting, and evaluating quantitative and qualitative research [Ebook]. 4th ed. London: Pearson Education; 2015.

Curry D. ChatGPT Revenue and Usage Statistics (2023)—Business of Apps. 2023. https://www.businessofapps.com/data/chatgpt-statistics/

Day T. A preliminary investigation of fake peer-reviewed citations and references generated by ChatGPT. Prof Geogr. 2023. https://doi.org/10.1080/00330124.2023.2190373 .

De Castro CA. A Discussion about the Impact of ChatGPT in education: benefits and concerns. J Bus Theor Pract. 2023;11(2):p28. https://doi.org/10.22158/jbtp.v11n2p28 .

Deng X, Yu Z. A meta-analysis and systematic review of the effect of Chatbot technology use in sustainable education. Sustainability. 2023;15(4):2940. https://doi.org/10.3390/su15042940 .

Eke DO. ChatGPT and the rise of generative AI: threat to academic integrity? J Responsib Technol. 2023;13:100060. https://doi.org/10.1016/j.jrt.2023.100060 .

Elmoazen R, Saqr M, Tedre M, Hirsto L. A systematic literature review of empirical research on epistemic network analysis in education. IEEE Access. 2022;10:17330–48. https://doi.org/10.1109/access.2022.3149812 .

Farrokhnia M, Banihashem SK, Noroozi O, Wals AEJ. A SWOT analysis of ChatGPT: implications for educational practice and research. Innov Educ Teach Int. 2023. https://doi.org/10.1080/14703297.2023.2195846 .

Fergus S, Botha M, Ostovar M. Evaluating academic answers generated using ChatGPT. J Chem Educ. 2023;100(4):1672–5. https://doi.org/10.1021/acs.jchemed.3c00087 .

Fink A. Conducting research literature reviews: from the Internet to Paper. Incorporated: SAGE Publications; 2010.

Firaina R, Sulisworo D. Exploring the usage of ChatGPT in higher education: frequency and impact on productivity. Buletin Edukasi Indonesia (BEI). 2023;2(01):39–46. https://doi.org/10.56741/bei.v2i01.310 .

Firat, M. (2023). How chat GPT can transform autodidactic experiences and open education.  Department of Distance Education, Open Education Faculty, Anadolu Unive .‏ https://orcid.org/0000-0001-8707-5918

Firat M. What ChatGPT means for universities: perceptions of scholars and students. J Appl Learn Teach. 2023. https://doi.org/10.37074/jalt.2023.6.1.22 .

Fuchs K. Exploring the opportunities and challenges of NLP models in higher education: is Chat GPT a blessing or a curse? Front Educ. 2023. https://doi.org/10.3389/feduc.2023.1166682 .

García-Peñalvo FJ. La percepción de la inteligencia artificial en contextos educativos tras el lanzamiento de ChatGPT: disrupción o pánico. Educ Knowl Soc. 2023;24: e31279. https://doi.org/10.14201/eks.31279 .

Gilson A, Safranek CW, Huang T, Socrates V, Chi L, Taylor A, Chartash D. How does ChatGPT perform on the United States medical Licensing examination? The implications of large language models for medical education and knowledge assessment. JMIR Med Educ. 2023;9: e45312. https://doi.org/10.2196/45312 .

Hashana AJ, Brundha P, Ayoobkhan MUA, Fazila S. Deep Learning in ChatGPT—A Survey. In   2023 7th international conference on trends in electronics and informatics (ICOEI) . 2023. (pp. 1001–1005). IEEE. https://doi.org/10.1109/icoei56765.2023.10125852

Hirsto L, Saqr M, López-Pernas S, Valtonen T. (2022). A systematic narrative review of learning analytics research in K-12 and schools.  Proceedings . https://ceur-ws.org/Vol-3383/FLAIEC22_paper_9536.pdf

Hisan UK, Amri MM. ChatGPT and medical education: a double-edged sword. J Pedag Educ Sci. 2023;2(01):71–89. https://doi.org/10.13140/RG.2.2.31280.23043/1 .

Hopkins AM, Logan JM, Kichenadasse G, Sorich MJ. Artificial intelligence chatbots will revolutionize how cancer patients access information: ChatGPT represents a paradigm-shift. JNCI Cancer Spectr. 2023. https://doi.org/10.1093/jncics/pkad010 .

Househ M, AlSaad R, Alhuwail D, Ahmed A, Healy MG, Latifi S, Sheikh J. Large Language models in medical education: opportunities, challenges, and future directions. JMIR Med Educ. 2023;9: e48291. https://doi.org/10.2196/48291 .

Ilkka T. The impact of artificial intelligence on learning, teaching, and education. Minist de Educ. 2018. https://doi.org/10.2760/12297 .

Iqbal N, Ahmed H, Azhar KA. Exploring teachers’ attitudes towards using CHATGPT. Globa J Manag Adm Sci. 2022;3(4):97–111. https://doi.org/10.46568/gjmas.v3i4.163 .

Irfan M, Murray L, Ali S. Integration of Artificial intelligence in academia: a case study of critical teaching and learning in Higher education. Globa Soc Sci Rev. 2023;8(1):352–64. https://doi.org/10.31703/gssr.2023(viii-i).32 .

Jeon JH, Lee S. Large language models in education: a focus on the complementary relationship between human teachers and ChatGPT. Educ Inf Technol. 2023. https://doi.org/10.1007/s10639-023-11834-1 .

Khan RA, Jawaid M, Khan AR, Sajjad M. ChatGPT—Reshaping medical education and clinical management. Pak J Med Sci. 2023. https://doi.org/10.12669/pjms.39.2.7653 .

King MR. A conversation on artificial intelligence, Chatbots, and plagiarism in higher education. Cell Mol Bioeng. 2023;16(1):1–2. https://doi.org/10.1007/s12195-022-00754-8 .

Kooli C. Chatbots in education and research: a critical examination of ethical implications and solutions. Sustainability. 2023;15(7):5614. https://doi.org/10.3390/su15075614 .

Kuhail MA, Alturki N, Alramlawi S, Alhejori K. Interacting with educational chatbots: a systematic review. Educ Inf Technol. 2022;28(1):973–1018. https://doi.org/10.1007/s10639-022-11177-3 .

Lee H. The rise of ChatGPT: exploring its potential in medical education. Anat Sci Educ. 2023. https://doi.org/10.1002/ase.2270 .

Li L, Subbareddy R, Raghavendra CG. AI intelligence Chatbot to improve students learning in the higher education platform. J Interconnect Netw. 2022. https://doi.org/10.1142/s0219265921430325 .

Limna P. A Review of Artificial Intelligence (AI) in Education during the Digital Era. 2022. https://ssrn.com/abstract=4160798

Lo CK. What is the impact of ChatGPT on education? A rapid review of the literature. Educ Sci. 2023;13(4):410. https://doi.org/10.3390/educsci13040410 .

Luo W, He H, Liu J, Berson IR, Berson MJ, Zhou Y, Li H. Aladdin’s genie or pandora’s box For early childhood education? Experts chat on the roles, challenges, and developments of ChatGPT. Early Educ Dev. 2023. https://doi.org/10.1080/10409289.2023.2214181 .

Meyer JG, Urbanowicz RJ, Martin P, O’Connor K, Li R, Peng P, Moore JH. ChatGPT and large language models in academia: opportunities and challenges. Biodata Min. 2023. https://doi.org/10.1186/s13040-023-00339-9 .

Mhlanga D. Open AI in education, the responsible and ethical use of ChatGPT towards lifelong learning. Soc Sci Res Netw. 2023. https://doi.org/10.2139/ssrn.4354422 .

Neumann, M., Rauschenberger, M., & Schön, E. M. (2023). “We Need To Talk About ChatGPT”: The Future of AI and Higher Education.‏ https://doi.org/10.1109/seeng59157.2023.00010

Nolan B. Here are the schools and colleges that have banned the use of ChatGPT over plagiarism and misinformation fears. Business Insider . 2023. https://www.businessinsider.com

O’Leary DE. An analysis of three chatbots: BlenderBot, ChatGPT and LaMDA. Int J Intell Syst Account, Financ Manag. 2023;30(1):41–54. https://doi.org/10.1002/isaf.1531 .

Okoli C. A guide to conducting a standalone systematic literature review. Commun Assoc Inf Syst. 2015. https://doi.org/10.17705/1cais.03743 .

OpenAI. (2023). https://openai.com/blog/chatgpt

Perkins M. Academic integrity considerations of AI large language models in the post-pandemic era: ChatGPT and beyond. J Univ Teach Learn Pract. 2023. https://doi.org/10.53761/1.20.02.07 .

Plevris V, Papazafeiropoulos G, Rios AJ. Chatbots put to the test in math and logic problems: A preliminary comparison and assessment of ChatGPT-3.5, ChatGPT-4, and Google Bard. arXiv (Cornell University) . 2023. https://doi.org/10.48550/arxiv.2305.18618

Rahman MM, Watanobe Y (2023) ChatGPT for education and research: opportunities, threats, and strategies. Appl Sci 13(9):5783. https://doi.org/10.3390/app13095783

Ram B, Verma P. Artificial intelligence AI-based Chatbot study of ChatGPT, google AI bard and baidu AI. World J Adv Eng Technol Sci. 2023;8(1):258–61. https://doi.org/10.30574/wjaets.2023.8.1.0045 .

Rasul T, Nair S, Kalendra D, Robin M, de Oliveira Santini F, Ladeira WJ, Heathcote L. The role of ChatGPT in higher education: benefits, challenges, and future research directions. J Appl Learn Teach. 2023. https://doi.org/10.37074/jalt.2023.6.1.29 .

Ratnam M, Sharm B, Tomer A. ChatGPT: educational artificial intelligence. Int J Adv Trends Comput Sci Eng. 2023;12(2):84–91. https://doi.org/10.30534/ijatcse/2023/091222023 .

Ray PP. ChatGPT: a comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet Things Cyber-Phys Syst. 2023;3:121–54. https://doi.org/10.1016/j.iotcps.2023.04.003 .

Roumeliotis KI, Tselikas ND. ChatGPT and Open-AI models: a preliminary review. Future Internet. 2023;15(6):192. https://doi.org/10.3390/fi15060192 .

Rudolph J, Tan S, Tan S. War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education. J Appl Learn Teach. 2023. https://doi.org/10.37074/jalt.2023.6.1.23 .

Ruiz LMS, Moll-López S, Nuñez-Pérez A, Moraño J, Vega-Fleitas E. ChatGPT challenges blended learning methodologies in engineering education: a case study in mathematics. Appl Sci. 2023;13(10):6039. https://doi.org/10.3390/app13106039 .

Sallam M, Salim NA, Barakat M, Al-Tammemi AB. ChatGPT applications in medical, dental, pharmacy, and public health education: a descriptive study highlighting the advantages and limitations. Narra J. 2023;3(1): e103. https://doi.org/10.52225/narra.v3i1.103 .

Salvagno M, Taccone FS, Gerli AG. Can artificial intelligence help for scientific writing? Crit Care. 2023. https://doi.org/10.1186/s13054-023-04380-2 .

Saqr M, López-Pernas S, Helske S, Hrastinski S. The longitudinal association between engagement and achievement varies by time, students’ profiles, and achievement state: a full program study. Comput Educ. 2023;199:104787. https://doi.org/10.1016/j.compedu.2023.104787 .

Saqr M, Matcha W, Uzir N, Jovanović J, Gašević D, López-Pernas S. Transferring effective learning strategies across learning contexts matters: a study in problem-based learning. Australas J Educ Technol. 2023;39(3):9.

Schöbel S, Schmitt A, Benner D, Saqr M, Janson A, Leimeister JM. Charting the evolution and future of conversational agents: a research agenda along five waves and new frontiers. Inf Syst Front. 2023. https://doi.org/10.1007/s10796-023-10375-9 .

Shoufan A. Exploring students’ perceptions of CHATGPT: thematic analysis and follow-up survey. IEEE Access. 2023. https://doi.org/10.1109/access.2023.3268224 .

Sonderegger S, Seufert S. Chatbot-mediated learning: conceptual framework for the design of Chatbot use cases in education. Gallen: Institute for Educational Management and Technologies, University of St; 2022. https://doi.org/10.5220/0010999200003182 .

Book   Google Scholar  

Strzelecki A. To use or not to use ChatGPT in higher education? A study of students’ acceptance and use of technology. Interact Learn Environ. 2023. https://doi.org/10.1080/10494820.2023.2209881 .

Su J, Yang W. Unlocking the power of ChatGPT: a framework for applying generative AI in education. ECNU Rev Educ. 2023. https://doi.org/10.1177/20965311231168423 .

Sullivan M, Kelly A, McLaughlan P. ChatGPT in higher education: Considerations for academic integrity and student learning. J ApplLearn Teach. 2023;6(1):1–10. https://doi.org/10.37074/jalt.2023.6.1.17 .

Szabo A. ChatGPT is a breakthrough in science and education but fails a test in sports and exercise psychology. Balt J Sport Health Sci. 2023;1(128):25–40. https://doi.org/10.33607/bjshs.v127i4.1233 .

Taecharungroj V. “What can ChatGPT do?” analyzing early reactions to the innovative AI chatbot on Twitter. Big Data Cognit Comput. 2023;7(1):35. https://doi.org/10.3390/bdcc7010035 .

Tam S, Said RB. User preferences for ChatGPT-powered conversational interfaces versus traditional methods. Biomed Eng Soc. 2023. https://doi.org/10.58496/mjcsc/2023/004 .

Tedre M, Kahila J, Vartiainen H. (2023). Exploration on how co-designing with AI facilitates critical evaluation of ethics of AI in craft education. In: Langran E, Christensen P, Sanson J (Eds).  Proceedings of Society for Information Technology and Teacher Education International Conference . 2023. pp. 2289–2296.

Tlili A, Shehata B, Adarkwah MA, Bozkurt A, Hickey DT, Huang R, Agyemang B. What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education. Smart Learn Environ. 2023. https://doi.org/10.1186/s40561-023-00237-x .

Uddin SMJ, Albert A, Ovid A, Alsharef A. Leveraging CHATGPT to aid construction hazard recognition and support safety education and training. Sustainability. 2023;15(9):7121. https://doi.org/10.3390/su15097121 .

Valtonen T, López-Pernas S, Saqr M, Vartiainen H, Sointu E, Tedre M. The nature and building blocks of educational technology research. Comput Hum Behav. 2022;128:107123. https://doi.org/10.1016/j.chb.2021.107123 .

Vartiainen H, Tedre M. Using artificial intelligence in craft education: crafting with text-to-image generative models. Digit Creat. 2023;34(1):1–21. https://doi.org/10.1080/14626268.2023.2174557 .

Ventayen RJM. OpenAI ChatGPT generated results: similarity index of artificial intelligence-based contents. Soc Sci Res Netw. 2023. https://doi.org/10.2139/ssrn.4332664 .

Wagner MW, Ertl-Wagner BB. Accuracy of information and references using ChatGPT-3 for retrieval of clinical radiological information. Can Assoc Radiol J. 2023. https://doi.org/10.1177/08465371231171125 .

Wardat Y, Tashtoush MA, AlAli R, Jarrah AM. ChatGPT: a revolutionary tool for teaching and learning mathematics. Eurasia J Math, Sci Technol Educ. 2023;19(7):em2286. https://doi.org/10.29333/ejmste/13272 .

Webster J, Watson RT. Analyzing the past to prepare for the future: writing a literature review. Manag Inf Syst Quart. 2002;26(2):3.

Xiao Y, Watson ME. Guidance on conducting a systematic literature review. J Plan Educ Res. 2017;39(1):93–112. https://doi.org/10.1177/0739456x17723971 .

Yan D. Impact of ChatGPT on learners in a L2 writing practicum: an exploratory investigation. Educ Inf Technol. 2023. https://doi.org/10.1007/s10639-023-11742-4 .

Yu H. Reflection on whether Chat GPT should be banned by academia from the perspective of education and teaching. Front Psychol. 2023;14:1181712. https://doi.org/10.3389/fpsyg.2023.1181712 .

Zhu C, Sun M, Luo J, Li T, Wang M. How to harness the potential of ChatGPT in education? Knowl Manag ELearn. 2023;15(2):133–52. https://doi.org/10.34105/j.kmel.2023.15.008 .

Download references

The paper is co-funded by the Academy of Finland (Suomen Akatemia) Research Council for Natural Sciences and Engineering for the project Towards precision education: Idiographic learning analytics (TOPEILA), Decision Number 350560.

Author information

Authors and affiliations.

School of Computing, University of Eastern Finland, 80100, Joensuu, Finland

Yazid Albadarin, Mohammed Saqr, Nicolas Pope & Markku Tukiainen

You can also search for this author in PubMed   Google Scholar

Contributions

YA contributed to the literature search, data analysis, discussion, and conclusion. Additionally, YA contributed to the manuscript’s writing, editing, and finalization. MS contributed to the study’s design, conceptualization, acquisition of funding, project administration, allocation of resources, supervision, validation, literature search, and analysis of results. Furthermore, MS contributed to the manuscript's writing, revising, and approving it in its finalized state. NP contributed to the results, and discussions, and provided supervision. NP also contributed to the writing process, revisions, and the final approval of the manuscript in its finalized state. MT contributed to the study's conceptualization, resource management, supervision, writing, revising the manuscript, and approving it.

Corresponding author

Correspondence to Yazid Albadarin .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

See Table  4

The process of synthesizing the data presented in Table  4 involved identifying the relevant studies through a search process of databases (ERIC, Scopus, Web of Knowledge, Dimensions.ai, and lens.org) using specific keywords "ChatGPT" and "education". Following this, inclusion/exclusion criteria were applied, and data extraction was performed using Creswell's [ 15 ] coding techniques to capture key information and identify common themes across the included studies.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Albadarin, Y., Saqr, M., Pope, N. et al. A systematic literature review of empirical research on ChatGPT in education. Discov Educ 3 , 60 (2024). https://doi.org/10.1007/s44217-024-00138-2

Download citation

Received : 22 October 2023

Accepted : 10 May 2024

Published : 26 May 2024

DOI : https://doi.org/10.1007/s44217-024-00138-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Large language models
• Educational technology
• Systematic review

Advertisement

• Find a journal
• Publish with us
• Track your research