Doing your systematic review: managing data and reporting

Doing your systematic
review: managing data
and reporting
Session 3/3
#systematicreview
@LivUniKnowHow | @LivUniLRiG

Ruaraidh Hill PhD MSc FHEA
Lecturer in evidence synthesis
Angela Boland MSc PhD PGCert (LTHE)
Director – Liverpool Reviews & Implementation Group
linktr.ee/RuaraidhHill

Your comfort & safety
§ Slides ✓
§ Recording ✓
§ Chat ✓
§ Tools see links in chat
§ References see chat & final slides

This session
Introduction & overview
Plan and manage:
§ search results
§ data extraction
§ quality assessment
§ reporting of your review

WeMap
LRiG – Health Data Science, Population Health
Liverpool Reviews and Implementation Group (LRiG) – 20+ years
at the core of evidence-based guidance development at NICE.
LRiG delivers a range of high-quality Health Technology
Assessments and capacity development initiatives.
Capacity development – support for individuals and teams to use
evidence to develop practice, services and research.
@LivUniLRiG | liverpool.ac.uk/LRiG

Our book – #&❤’d
www.tinyURL.com/DoingaReview2

Our book – your questions answered
www.tinyURL.com/DoingaReview2

Systematic review – in 3 numbers
3 | 4 | 10
essentials activities steps

Systematic review – essentials and key
activities
Question
Explicit
method*
Findings
§ Finding
§ Describing
§ Appraisal
§ Synthesis
Identifying
relevant
research
(finding)
Describing
relevant
research
(describing)
Critically
appraising
research
reports in a
systematic
manner
(appraisal)
Bringing
together
the findings
into a
coherent
statement
(synthesis)

Systematic review – our 10-step road
map

Systematic review approaches – for
different questions
§ To present a concept
Descriptive
§ Explore preferences about what should happen
Normative
§ Investigate a relationship between two or more
variables
Observational/relational
§ Investigate the effect of one or more independent
variables on one or more outcome variables
Causal
§ Explore factors that cause a condition, event or
process
Theoretical

Managing search results
◉◎◎◎

Selection - what is…?
Included
records
Excluded
records

Selection - what is…?
Selecting evidence from your search results in
2 stages:
§ Screen the titles and abstracts of search results against your
inclusion and exclusion criteria
§ Select full-text papers for inclusion in your review, noting
reason for exclusion
Keeping records:
Record and report the results of your search and process of
selecting evidence

Key steps
1. De-duplicate references
2. Develop and pilot your screening and selection tool
3. Screen all titles and abstracts identified via searches against
your inclusion and exclusion criteria (Stage 1)
4. Obtain the full-text papers of all potentially eligible references
5. Use your selection tool to help you identify full-text papers for
inclusion in review (Stage 2) – and record primary reason for
exclusion at full text stage

Key steps – tools & decision points
Create protocol
Search for
studies
Create database
of studies to
screen
Screen on
title/abstract
INCLUDE?
Reduced database
Screen on
full report
INCLUDE?
‘INCLUDED’
database
Data extraction
Quality
assessment
Synthesis
Flow of reference through a review. Adapted (L Jones)
from Figure 5.3, page 92
Brunton J & Thomas J. Information management in reviews. In: Gough D,
Oliver S, Thomas J (eds). An Introduction to Systematic Reviews. London;
SAGE Publications Ltd

Examples
Screening tool
Criteria for first screen – using title & abstract, a rapid assessment to exclude or mark for retrieval of
full text
Question frame Options Examples/notes
Can I not exclude this record?
Does the record appear to be:
music therapy
AND
eating disorders
AND
experimental design
OR
case series (see notes)
Refer to the Screening
Tool with inclusion and
exclusions criteria.
Yes – code as: Include
(for consideration of
the full text at a further
stage of review)
If you have included
but have strong doubt
about its suitability
mark as ‘Maybe’
instead. Add a note
briefly describing your
area of doubt.
Also see
If considering for more than 2 min,
do not exclude and move to next
record
If the record appears to describe a
case study or case series - then code
as ‘Include’ and [Label] ‘Case series’
at at ti.abs. (A decision will be made
on inclusion at the next stage of the
review.)
If the record is a review (of any kind)
or an Expert piece - then code as
‘Include’ at ti.abs (for identification
of other primary research and
analysis framework purposes) and
use [Label] facility, ‘Systematic
review’, ‘Review’ or an ‘Expert
piece’.
If the record is a book chapter and
may report primary data, then. then
code as ‘Include’ at ti.abs (for
identification of other primary
research and analysis framework
purposes) and use [Label] facility,
‘Book’. (A decision will be made on
inclusion at the next stage of the
review.)
Can I exclude this record? Yes – code as: Exclude Exclusion include:
Population – not eating disorders
such as dementia-related eating
difficulties anatomical, coordination,
§ Written, agreed and tested
guide to the process and
criteria for screening
§ Consistent with what you
stated in your protocol

Review systems – help with finding and
describing
https://libraryguides.mcgill.ca/rayyan/gettingstarted | https://www.rayyan.ai

Review systems – for managing and
recording evidence screening & selection
§ Office applications
§ Word, Excel, Access, Pages,
Numbers, Docs, Sheets… etc
§ Bibliographic management
§ EndNote, Mendeley… etc
§ Reviewing systems
§ RevMan
§ Covidence*
§ EPPI-Reviewer*
§ SR Distiller*
§ sysrev*
§ Rayyan
§ Open source…
* Subscription with some trial and educational
use options

PRISMA flow diagram
§ Summary of numbers of
records (search results)
considered at key stages of
selecting evidence
§ Keep copies of searches,
screening and full text
decisions to allow PRISMA to
be reported
PRISMA 2009 Flow Diagram
Records identified through
database searching
(n = )
Screening
Included
Eligibility
Identification
Additional records identified
through other sources
(n = )
Records after duplicates removed
(n = )
Records screened
(n = )
Records excluded
(n = )
Full-text articles assessed
for eligibility
(n = )
Full-text articles excluded,
with reasons
(n = )
Studies included in
qualitative synthesis
(n = )
Studies included in
quantitative synthesis
(meta-analysis)
(n = )

PRISMA flow diagram
§ Summary of numbers of
records (search results)
considered at key stages of
selecting evidence
§ Keep copies of searches,
screening and full text
decisions to allow PRISMA to
be reported

Tips
Keep extensive
records and back
these up
Consider software to
assist you
Test your screening
and inc/exclusion
criteria

What is…?
• A data extraction involves identify, extracting and recording
relevant data to answer your review question
§ Data extraction forms record relevant data from a particular
study – therefore you need a form for each study.
§ Data extraction tables are tables which allow you to store
extracted data from one or all of your studies in one place.
§ Data tables (or evidence tables) are tables that describe and
summarise your extracted data in your report/ thesis.

Key steps
1. Identify the data that you want to extract
2. Build and pilot your data extraction form or data extraction
table(s)
3. Extract relevant data – quality assurance/ error check
(preferably another reviewer does this)
4. Complete the data tables
5. Report your data

Examples – extraction, data management

Examples – extraction, data management
§ mini tip: search for power query in Excel to separate data into fields

Examples – data tables
Bibliographi c reference
Interventio n Comparato r Method of allocation
Intervention in detail (who, where, when) Methods use to minimize confounders
Number of
participants
Participant
characteristics
Length of
follow-up
Methods of
analysis
Outcomes/
Results
Limitations
Additional
comments
… … … … … … …
Power
information
Method of
recruitment
Information on
representative-
ness
Loss to
follow-up
ITT or
completer
Adjustments
for baseline
differences
Objective/
subjective
Time points
Health
inequalities
impact
Identified by
authors
Identified by
developers
Evidence
gaps
Further
research
identified
§ NICE Guidelines Maunual Appendix H

Examples – data tables
§ NICE Guidelines Maunual Appendix H

Tips
§ As a minimum record first author and publication year.
§ Consider unique ID for records (e.g EndNote ref) and studies. Identify
publications linked to studies.
§ Plan what you will extract.
§ Think: what do I need to answer my review question?
§ Consider coded responses (e.g. RCT/ non-RCT or UK/ non-UK)
§ Include ‘yes’, ‘no’, ‘not reported’ and ‘unclear’ options
§ Keep numerical data and information (units) separate, if you can
§ Consider keeping to a ‘1 datum’ per cell rule ([n] [N] [%]…)
§ Include room for notes

Quality assessment
◎◎◉◎

What is…?
§ The degree to which a study employs measures to minimize
bias and error in its design, conduct and analysis.
(Khan et al., 2003, p. 39)
§ Also consider applicability when considering the evidence.

Key steps
1. Note the methodological design(s) of your included studies – different
checklists are used depending on study design
2. Identify the quality assessment tool(s) available for your review
3. Choose the appropriate quality assessment tool(s)
4. Carry out quality assessment using the appropriate tool(s)
5. Tabulate and summarise the results of your quality assessment
6. Think about how the results of your quality assessment exercise might
impact on the conclusions and recommendations of your systematic
review

nice.org.uk/process/pmg20/
Key steps –
what design?

Examples
CASP Checklist: 10 questions to help you make sense of a Qualitative research
How to use this appraisal tool: Three broad issues need to be considered when appraising a
qualitative study:
Are the results of the study valid? (Section A)
What are the results? (Section B)
Will the results help locally? (Section C)
The 10 questions on the following pages are designed to help you think about these issues
systematically. The first two questions are screening questions and can be answered quickly.
If the answer to both is “yes”, it is worth proceeding with the remaining questions. There is
some degree of overlap between the questions, you are asked to record a “yes”, “no” or
“can’t tell” to most of the questions. A number of italicised prompts are given after each
question. These are designed to remind you why the question is important. Record your
reasons for your answers in the spaces provided.
About: These checklists were designed to be used as educational pedagogic tools, as part of a
workshop setting, therefore we do not suggest a scoring system. The core CASP checklists
(randomised controlled trial & systematic review) were based on JAMA 'Users’ guides to the
medical literature 1994 (adapted from Guyatt GH, Sackett DL, and Cook DJ), and piloted with
health care practitioners.
For each new checklist, a group of experts were assembled to develop and pilot the checklist
and the workshop format with which it would be used. Over the years overall adjustments
have been made to the format, but a recent survey of checklist users reiterated that the basic
format continues to be useful and appropriate.
Referencing: we recommend using the Harvard style citation, i.e.: Critical Appraisal Skills
Programme (2018). CASP (insert name of checklist i.e. Qualitative) Checklist. [online] Available
at: URL. Accessed: Date Accessed.
©CASP this work is licensed under the Creative Commons Attribution – Non-Commercial-
Share A like. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-
sa/3.0/ www.casp-uk.net
Critical Appraisal Skills Programme (CASP) part of Oxford Centre for Triple Value Healthcare Ltd www.casp-uk.net

Tips
More than completing a cross word puzzle, but some there are
some common approaches…
44

Tips – approach
Take a measured approach, combining:
§ Looking broadly
§ Using tools
§ Make notes and highlight
§ Use tables or forms
§ Record reasons for quality judgement, if not obvious
§ Second - preferably independent - assessment
§ Implications for your view of the evidence
§ Implications for practice
45

Tips – tool development & choice
§ Some tools or questions are based on rigorous quantitative assessment,
testing and review
§ Many are based on (expert) consensus
§ Some tools have been created (rather like disease specific outcomes) for
use in only limited areas of study
§ There is an expanse of QA tools – choosing can be the issue, rather
finding a tool
Choice:
§ Use in publications in the field | context studied
§ Society, network or national endorsement
§ Methods manuals, protocols or language
46

What is….?
Discussion of findings and evidence quality and conclusions
§ Provide a critical interpretation of the results of your review in
relation to the review question that you set out to answer.
§ Explore uncertainties and their implications on your findings.

Key steps
First, start with a brief overview of review question and
methodology.
Then, answer the following questions:
§ What were the main findings of my review?
§ How do my findings fit with previous research?
§ What are the strengths and limitations of the included studies?
§ What are the strengths and limitations of the review?
§ Can the findings be generalised?
§ What are the implications of the review?
§ What conclusions can be drawn from the review?

Tip – evidence statements
The NICE ‘manual’ says…
A clear, self-contained summary, that includes:
Evidence source (study type and references) and their quality
Content of the intervention including population and setting
Strength of evidence
Appropriateness of the study design to answer the question and quality,
quantity and consistency of evidence
Outcomes including the direction size of effect
Applicability to the question, target population and setting
50

Tip – evidence statements
In brief…
A clear, self-contained summary:
§ Evidence sources
§ Intervention
§ including population and setting
§ Strength of evidence
§ Outcomes
§ Applicability
§ to question, target population and setting
51

Evidence statement 3: Effectiveness of interventions aiming to
improve TB identification among homeless people
• 3·1 Weak evidence from Bernard et al., 2012 [+] showed that screening
homeless people by Mobile X-ray Unit (MXU) improved screening
coverage and reduced TB transmission among homeless people using
shelters but also among non-shelter users.
• Seven studies identified by the NICE review reported on the use of
incentives, two studies focussed on homeless people (Citron et al., 1995
[+] and Pilote et al., 1996 [++]).24,25
• 3·2 Moderate evidence from two studies (Citron et al., 1995 [+] and Pilote
et al., 1996 [++]) showed that the screening uptake improved among
homeless people when a monetary incentive was given.

Tip – PRISMA checklist
Use PRISMA
systematic review
reporting standard
as a guide to
conducting and
checklist for
reporting your
review
Address or discuss
where you have not
met a PRISMA item
www.prisma-statement.org/PRISMAStatement/
PRISMA 2020 Checklist
Section and
Topic
Item
#
Checklist item
Location
where item
is reported
TITLE
Title 1 Identify the report as a systematic review.
ABSTRACT
Abstract 2 See the PRISMA 2020 for Abstracts checklist.
INTRODUCTION
Rationale 3 Describe the rationale for the review in the context of existing knowledge.
Objectives 4 Provide an explicit statement of the objective(s) or question(s) the review addresses.
METHODS
Eligibility criteria 5 Specify the inclusion and exclusion criteria for the review and how studies were grouped for the syntheses.
Information
sources
6 Specify all databases, registers, websites, organisations, reference lists and other sources searched or consulted to identify studies. Specify the
date when each source was last searched or consulted.
Search strategy 7 Present the full search strategies for all databases, registers and websites, including any filters and limits used.
Selection process 8 Specify the methods used to decide whether a study met the inclusion criteria of the review, including how many reviewers screened each record
and each report retrieved, whether they worked independently, and if applicable, details of automation tools used in the process.
Data collection
process
9 Specify the methods used to collect data from reports, including how many reviewers collected data from each report, whether they worked
independently, any processes for obtaining or confirming data from study investigators, and if applicable, details of automation tools used in the
process.
Data items 10a List and define all outcomes for which data were sought. Specify whether all results that were compatible with each outcome domain in each
study were sought (e.g. for all measures, time points, analyses), and if not, the methods used to decide which results to collect.
10b List and define all other variables for which data were sought (e.g. participant and intervention characteristics, funding sources). Describe any
assumptions made about any missing or unclear information.
Study risk of bias
assessment
11 Specify the methods used to assess risk of bias in the included studies, including details of the tool(s) used, how many reviewers assessed each
study and whether they worked independently, and if applicable, details of automation tools used in the process.
Effect measures 12 Specify for each outcome the effect measure(s) (e.g. risk ratio, mean difference) used in the synthesis or presentation of results.
Synthesis
methods
13a Describe the processes used to decide which studies were eligible for each synthesis (e.g. tabulating the study intervention characteristics and
comparing against the planned groups for each synthesis (item #5)).
13b Describe any methods required to prepare the data for presentation or synthesis, such as handling of missing summary statistics, or data
conversions.
13c Describe any methods used to tabulate or visually display results of individual studies and syntheses.
13d Describe any methods used to synthesize results and provide a rationale for the choice(s). If meta-analysis was performed, describe the
model(s), method(s) to identify the presence and extent of statistical heterogeneity, and software package(s) used.
13e Describe any methods used to explore possible causes of heterogeneity among study results (e.g. subgroup analysis, meta-regression).
13f Describe any sensitivity analyses conducted to assess robustness of the synthesized results.
Reporting bias
assessment
14 Describe any methods used to assess risk of bias due to missing results in a synthesis (arising from reporting biases).
Certainty
assessment
15 Describe any methods used to assess certainty (or confidence) in the body of evidence for an outcome.

Summary – this session
Plan and manage:
§search results
§data extraction
§quality assessment
§reporting of your review
Identifying
relevant
research
(finding)
Describing
relevant
research
(describing)
Critically
appraising
research
reports in a
systematic
manner
(appraisal)
Bringing
together
the findings
into a
coherent
statement
(synthesis)

Next steps
§ Your questions
§ References
§ This presentation
§ Finding tools…

Finding tools – tools we use
TinyURL.com/KnowHowSRtools

Finding tools – SR toolbox
systematicreviewtools.com

Finding tools – methods guides
nice.org.uk/process/pmg20/
training.cochrane.org/handbook/
tinyURL.com/DoingaReview2

Finding tools – quality assessment
QA groups and networks
§ GRADE working group
§ http://www.gradeworkinggroup.org
§ The Cochrane Collaboration
§ http://training.cochrane.org/handbook
§ CASP
§ http://www.casp-uk.net/casp-tools-
checklists
Review and guideline producers
§ Centre for Reviews & Dissemination
§ https://www.york.ac.uk/crd/guidance/
§ NICE
§ https://www.nice.org.uk/article/PMG20/
§ Appendix H
§ Effective Public Health Practice Project
§ http://www.ephpp.ca/aboutus.html
Topic groups
§ Societies/Colleges
§ Journals
§ Research groups

Finding tools – reporting standards
equator-
network.org

Doing evidence?
training.cochrane.
org/essentials

References
Key steps and top tips based on:
Boland, A., Cherry, G. and Dickson,
R. eds., 2017. Doing a systematic
review: A student's guide. Sage.
NICE, 2014. Developing NICE guidelines: the
manual. NICE, Manchester
§ https://www.nice.org.uk/process/pmg20/chapter/reviewing-
research-evidence#summarising-evidence
§ https://www.nice.org.uk/process/pmg20/chapter/writing-the-
guideline#interpreting-the-evidence-to-make-
recommendations
§ https://www.nice.org.uk/process/pmg20/resources/appendix-
h-appraisal-checklists-evidence-tables-grade-and-economic-
profiles-pdf-8779777885

Doing your systematic review: managing data and reporting

More Related Content

What's hot (20)

Similar to Doing your systematic review: managing data and reporting (20)

More from University of Liverpool Library (20)

Recently uploaded (20)

Doing your systematic review: managing data and reporting