Skip to main content

Systematic reviews: Study selection and appraisal

By Paul Levett

Study selection: PRISMA Item 9

First level screening - title and abstract review

At the initial screening stage read just the title and abstract of the candidate studies and make a decision to include or exclude the study from your review.  

For small reviews of a few studies (e.g. <100)

The research team should agree on the inclusion and exclusion criteria for studies you wish to review and put together a study screening form.  To help identify your inclusion/exclusion criteria, revisit the PICOS of interest you came up with for your search strategy and gain agreement/approval from your colleagues or supervisor. The screening form may look similar to Table 3 of Brown et al (2013). You may write down your decision to include or exclude an article on an Excel spreadsheet like this one, or if you have a small number of records you may choose to print out one copy for each record, although printing will be impractical for larger numbers of records.  Screen each potentially useful article identified in the literature search as follows:

  1. Read the title and abstract (where available) and apply the inclusion/exclusion criteria from the screening form.
  2. Make a decision on whether or not to include the study in the review.
  3. Record the decision and reasons for inclusion/exclusion on the study screening form or spreadsheet. You will summarise the reasons for exclusion on the PRISMA flow diagram - see Study Selection PRISMA item # 17 below.

For large reviews of many studies (e.g. >100) - in case you need to partially automate the screening process

There are two web-based software applications that can help with screening and tracking your selection decisions:

  1. Covidence (free trial account for one review then tiered pricing). The software provides a decision dashboard and annotation tool, and the ability to create forms for screening, for evaluating the risk of bias, and for extracting data. Covidence is used by Cochrane review teams as their first level screening, risk of bias assessment, and data extraction tool, the resulting study characteristics and decision data can be exported to RevMan (free for academic use) or Excel. 
  2. Abstrackr (free, Beta, open-source). Abstrackr comprises two components; a web-based annotation tool that allows participants in a review to collaboratively screen citations for relevance, and machine learning technologies that semi-automate the screening process. The web-based annotation tool allows project leads to import the citations that are to be screened for a review from either RefMan or Pubmed. Participants can then join the project and begin screening; the tool maintains a digital paper trail of all screening decisions.  The machine learning technology permits reviewers to screen roughly half of the set of citations imported for a given review, and then let the software automatically exclude a (hopefully large) portion of the remaining citations; the reviewers will then only need to screen the articles classified as relevant by the software.  A recent article evaluating the use of Abstrackr in the systematic review process is Rathbone, J., Hoffmann, T., & Glasziou, P. (2015). Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers. Systematic Reviews, 480. doi:10.1186/s13643-015-0067-6

Second level screening - full text review

Having excluded candidate studies that did not meet your inclusion/exclusion criteria you should have a smaller number of potentially relevant studies. Read and critically appraise the full text of each study you selected at the first pass screening stage to determine whether you wish to include them in your discussion and analysis. Specifically each study must be evaluated based on the following criteria:

Does this study address a clearly focused question?
Did the study use valid methods to address this question?
Are the valid results of this study important?
Are these valid, important results applicable to my patient or population?

If the answer to any of these questions is “no”, you may wish to read no further and exclude the study, or you may decide to include the study to inform your discussion but not include the results in your analysis. 

To help with this process you may wish to download and apply one of the example Critical Appraisal Worksheets from the Center for Evidence Based Medicine to each study - choose a worksheet that matches the type of study:
Systematic Review article Critical Appraisal Sheet
Diagnosis study Critical Appraisal Sheet
Prognosis study Critical Appraisal Sheet
Therapy / Randomized Controlled Trial Critical Appraisal Sheet

Make a decision on whether or not to include the study in your review, and write your decision and reasons for inclusion/exclusion at this second level/full text review stage on the study screening form. You will summarise the reasons for exclusion on the PRISMA flow diagram - see Study Selection PRISMA item # 17 below.

Reporting your screening decisions

In the final report in the methods section the PRISMA checklist Item 9 study selection will be reported as:

  • How studies were screened e.g. by reading title & abstract, and how they were crtically appraised e.g. by applying a standardised appraisal form appropriate for that study type - see above.
  • What sort of studies were excluded e.g. letters, conference abstracts, etc.
  • Who reviewed/appraised the studies
  • What the process was for resolving disagreements e.g. reporting the level of inter-rater agreement, how often arbitration about selection was required, & what efforts were made to resolve disagreement e.g. were original authors contacted

Study selection PRISMA Item 17

Researchers must keep the screening forms to create a summary descriptive flow diagram of study selection.

In the final report in the results section the PRISMA checklist Item 17 study selection should be reported as follows:

  • Record the number of studies screened, assessed for eligibility and included in the review, with reasons for exclusions both in the text and in form of a PRISMA flow diagram of study selection e.g. similar to Fig 2 of Liberati et al. (2009). There is a PRISMA flow diagram generator at http://old_theta.phm.utoronto.ca/tools/prisma

The Himmelfarb Health Sciences Library
Questions? Ask us.
Creative Commons License
All LibGuides by Himmelfarb Health Sciences Library are licensed under a
Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

The George Washington University