The UT Southwestern Librarians provide two levels of Evidence Synthesis/Systematic Review (ES/SR) support.
| GROUP | PRIMARY RESPONSIBILITY | NOTES |
|---|---|---|
| Librarian | ✔ | Librarian forwards search results for each database as EndNote ready files (.ris) and WORD documents (upon request) as part of the fee-based service. |
| ES/SR Team | ✔ | The ES/SR Team de-duplicates using the preferred software after records from all of the databases have been imported. |
While a bibliographic database contains unique references, there can be overlap between the different databases. When performing exhaustive searches across multiple databases for ES/SR, it is not uncommon to retrieve multiple citations for the same article. After the searches have been performed, the ES/SR Team can screen for duplicate citations. De-duplicating will save reviewers time and work before continuing to the next task of screening titles and abstracts (Mitchell 2022).
Citation or reference management software (e.g., EndNote, Mendeley, RefWorks, Zotero) can be used to de-duplicate search results. Systematic review software programs (e.g., Covidence, DistillerSR, Rayyan) were developed to facilitate and streamline the creation of systematic reviews and other syntheses and include de-duplication functionality. UT Southwestern subscribes to EndNote (citation management software) and Covidence (systematic review software).
In Task 6, the Librarian forwards the search results from the multiple databases to the ES/SR Team as EndNote ready files (.ris) and WORD documents (upon request). The ES/SR Team uses the results from the Librarian's searches and the ES/SR Team's searches to begin the PRISMA Flow Diagram.
The EndNote ready files allow the ES/SR Team to import the results into EndNote or Covidence. A combination of the selected software's de-duplication algorithm and manual review can be used to eliminate citation duplicates.
By default, EndNote identifies duplicate references in the EndNote library as references of the same reference type with matching Author, Year, and Title fields. The ES/SR Team can customize the criteria that EndNote uses to deduplicate citations. To access these settings:
For detailed instructions and training video, please see the following link(s):
Covidence will check for duplicates in the first set of references that are imported and also in the order in which they are imported. The first unique occurrence of a reference becomes a primary reference and all subsequent instances are duplicates. Covidence then checks within each subsequent import and also against all previous imports for duplicates.
Detailed instructions and training videos are available at the following links:
| GROUP | PRIMARY RESPONSIBILITY | NOTES |
|---|---|---|
| ES/SR Team | ✔ |
To identify the relevant studies, ES/SR searches aim to be comprehensive and seek to maximize recall (sensitivity) while balancing precision (specificity). It is not uncommon for these searches to retrieve thousands of studies that need to be reviewed. Sampson, et. al., found the average percent of studies retrieved making it into a final systematic review is around 3%, with a range of 3-6 articles meeting inclusion criteria per 100 articles examined. (Sampson, Tetzlaff, & Urquhart, 2011)
While time consuming, screening the title and abstracts for their relevance for inclusion is one of the most important tasks in the process. This task involves careful – sometimes subjective – judgments and thorough documentation. If the study inclusion criteria outlined in the ES/SR protocol are too narrow, critical data may be missed. If the inclusion criteria outlined in the SR protocol are too broad, irrelevant studies may overburden the process. (IOM (Institute of Medicine), 2011; Mitchell and Ennis, 2022; Page, 2021)
Typically the screening or study selection process is a multi-stage, iterative process and includes:
Two ES/SR Team members must independently examine the titles and abstracts in the de-duplicated results (Task 6) versus the inclusion criteria to remove obvious irrelevant results. If a citation lacks an abstract and only has the title, the citation should be included in the full-text review. A third ES/SR team member will resolve any differences in their decisions. The ES/SR Team should generally be over-inclusive at this step.
The screening steps can be time and labor intensive and logistically challenging. EndNote, Covidence, and Excel can be used to help organize the screening process. Systematic review software programs (Covidence, DistillerSR, and Rayyan) were developed to facilitate and streamline the creation of systematic reviews and other syntheses. UT Southwestern subscribes to Covidence. For more information, see Screening Tools (for Tasks 7 and 9).
| GROUP | PRIMARY RESPONSIBILITY | NOTES |
|---|---|---|
| ES/SR Team | ✔ |
Search results are forwarded to the ES/SR Team as EndNote ready files in the .ris format. The .ris files can be either imported into EndNote or Covidence. The ES/SR Team can de-duplicate the results (Task 6) and screen the title/abstract (Task 7) using either software program.
The ES/SR Team can obtain full text links within Covidence using LibKey Nomad.
Important Note about Uploading Full Text Articles
UT Southwestern reviewers are cautioned to NOT upload the full text articles obtained through UT Southwestern Library subscriptions or document delivery (Interlibrary Loan) to Covidence due to licensing and copyright restrictions. Institutional licenses generally prohibit retransmission/dissemination of licensed resources to non-authorized users. It is strongly recommended that UT Southwestern SR/ES team members download the articles using the doi/LibKey Nomad in Covidence.
For those citations where full text is not available, the SR/ES Team can:
UT Southwestern team members are prohibited from retransmitting/sharing licensed resources with non-UT Southwestern SR/ES team members. Non-UT Southwestern team members working on a SR/ES must obtain full text from their own Library collection or Interlibrary Loan.
For additional information, see:
| GROUP | PRIMARY RESPONSIBILITY | NOTES |
|---|---|---|
| ES/SR Team | ✔ |
Two ES/SR Team members must independently examine the retrieved full text of the potentially relevant citations and evaluate it against the inclusion/eligibility criteria. Research suggests it is beneficial for one reviewer to be a context expert and the second reviewer be a novice in the field. The reasoning is to minimize the content expert's beliefs and opinions which could bias the screening. (Mitchell and Ennis, 2022)
The ES/SR Team may need to correspond with the researchers to clarify whether the study should be included. A third ES/SR team member will resolve any differences in their decisions.
Please note the following:
The screening steps can be time- and labor-intensive, as well as logistically challenging. In addition to using EndNote and Excel to help organize the screening process, systematic review software programs (Covidence, DistillerSR, and Rayyan) were developed to facilitate and streamline the creation of systematic reviews and other syntheses. UT Southwestern subscribes to Covidence.
For more information, see Screening Tools (for Tasks 7 and 9).
| GROUP | PRIMARY RESPONSIBILITY | NOTES |
|---|---|---|
| ES/SR Team | ✔ |
Citation searching – also known as "handsearching" or "snowballing" – is frequently recommended by systematic review manuals and is undertaken by the ES/SR Team after full-text screening (Task 9). It supplements searching bibliographic databases to identify studies for an evidence synthesis/systematic review.
Assuming the retained citations are likely to have similar content, these reference lists are reviewed for relevant studies cited in them (backward search) or articles that have cited the retained full-text articles (forward search). Also, the reference lists of published related evidence syntheses/systematic reviews should be reviewed. Horsley, et. al., performed a systematic review to examine the effectiveness of checking reference lists as a method to supplement electronic searching. The data suggests that in situations where researchers have difficulty locating information, checking the reference lists of included articles may be an important way to reduce the risk of missing relevant information. (Briscoe, Bethel, & Rogers, 2020; Higgins JPT, 2020; Horsley, Dingwall, & Sampson, 2011; Muka et al., 2020)
Citation searching can also include the page-by-page examination of the entire contents of a print or online journal issue or conference proceedings to identify all eligible reports of trials. When reporting, it is important to specify whether the full text of a journal or conference proceedings has been searched electronically or using the print version. Some journals omit sections of the print version – for example, letters – from the electronic version, and some include supplementary information such as extra articles in the electronic format only. (Higgins JPT, 2020)
Articles identified from citation searching will need to undergo the de-duplicating and screening steps (Tasks 6 - 9).
| GROUP | PRIMARY RESPONSIBILITY | NOTES |
|---|---|---|
| ES/SR Team | ✔ |
Critical appraisal systematically examines factors, such as:
It is important for the ES/SR Team to report the quality of the evidence of the included studies. The quality encompasses the study's methodological quality (i.e., how the study was conducted) and reporting quality and reproducibility (how it was described). Poor methodological and reporting quality may introduce bias and questionable conclusions (Muka et al., 2020).
The following table includes definitions of some types of non-reporting biases (Higgins JPT, 2020). For more information on bias and risk of bias assessment, see Chapters 7, 8, 13, and 25 in Cochrane Handbook for Systematic Reviews of Interventions and Chapter 12 in Piecing Together Systematic Reviews and Other Evidence Syntheses.
| TYPE | DEFINITION |
|---|---|
| Publication bias | The publication or non-publication of research findings, depending on the nature and direction of the results. |
| Time-lag bias | The rapid or delayed publication of research findings, depending on the nature and direction of the results. |
| Language bias | The publication of research findings in a particular language, depending on the nature and direction of the results. |
| Citation bias | The citation or non-citation of research findings, depending on the nature and direction of the results. |
| Multiple (duplicate) publication bias | The multiple or singular publication of research findings, depending on the nature and direction of the results. |
| Location bias | The selective reporting of some outcomes or analyses, but not others, depending on the nature and direction of the results. |
The Cochrane Collaboration's tool, Risk-of-Bias Tool (RoB2), and the Newcastle-Ottawa Scale are commonly used to evaluate the risk of bias in randomized controlled trials and prospective observational studies, respectively (Muka, 2020).
Systematic review software programs (Covidence, DistillerSR, and Rayyan) were developed to facilitate and streamline the creation of systematic reviews and other syntheses. UT Southwestern subscribes to Covidence. To learn more, refer to the following links:
The screening steps can be time- and labor-intensive and logistically challenging. Systematic review software programs (Covidence, DistillerSR, and Rayyan) were developed to facilitate and streamline the creation of systematic reviews and other syntheses. UT Southwestern subscribes to Covidence.
Disclaimer
Your use of Covidence is completely voluntary and at your own discretion. By your use of Covidence, you accept that your use of it is governed exclusively by the terms of use of Covidence available here: https://www.covidence.org/terms/.
First time users need to register; existing accounts can proceed to Covidence.
The following flowchart summarizes the Covidence screening process.
While the Library does not provide training, Covidence provides excellent video tutorials and instructions detailing the screening process from title/abstract screening to full-text review to resolving conflicts. For more information, see:
Cochrane Handbook for Systematic Reviews of Interventions, 2nd ed. (Wiley Online Library)
by
Finding What Works in Health Care: Standards for Systematic Reviews (NCBI Bookshelf)
by
Piecing Together Systematic Reviews and Other Evidence Syntheses (ProQuest Ebook Central)
by
