SlideShare a Scribd company logo

1 - Systematic Literature Reviews: introduction and methods

Vittorio Scarano, profile picture
Vittorio Scarano

The document provides an introduction to systematic literature reviews (SLRs), outlining their importance, processes, and methodologies in research. It describes the three main phases of conducting an SLR: planning, conducting, and reporting, emphasizing the need for a structured approach to gather and assess literature accurately. Additionally, it discusses the evolution of SLRs, their relevance across various fields, and key components of a review protocol.

Education
Related topics:
Computer Science Basics
1 of 78
Downloaded 14 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Most read
21
Most read
22
Most read
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 1 / 78 1 - Systematic literature reviews: introduction and methods Vittorio Scarano Dipartimento di Informatica, Universit`a di Salerno (Italy) vitsca@unisa.it PhD in Computer Science, University of Salerno (Italy) January 14th, 2020
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 2 / 78 LESSON PLAN 1 Preamble to the two seminars 2 Why Systematic Literature Reviews? 3 Systematic Literature Reviews 4 The entire process 5 Phase 1 - Planning 6 Phase 2 - Conducting 7 Phase 3 - Reporting 8 An example 9 Conclusions and comments
1 - Systematic Literature Reviews: introduction and methods
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 4 / 78 MAP 1 Preamble to the two seminars 2 Why Systematic Literature Reviews? 3 Systematic Literature Reviews 4 The entire process 5 Phase 1 - Planning 6 Phase 2 - Conducting 7 Phase 3 - Reporting 8 An example 9 Conclusions and comments
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 5 / 78 A CIRCLE THAT CONTAINS HUMAN KNOWLEDGE. . . From “The illustrated guide to a Ph.D.” by Matt Might, CS Dept. of Univ. of Utah
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 6 / 78 BY END OF ELEMENTARY SCHOOL . . . From “The illustrated guide to a Ph.D.” by Matt Might, CS Dept. of Univ. of Utah
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 7 / 78 BY END OF HIGH SCHOOL From “The illustrated guide to a Ph.D.” by Matt Might, CS Dept. of Univ. of Utah
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 8 / 78 A BACHELOR: A SPECIALTY From “The illustrated guide to a Ph.D.” by Matt Might, CS Dept. of Univ. of Utah
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 9 / 78 A MASTER DEEPENS THE SPECIALTY From “The illustrated guide to a Ph.D.” by Matt Might, CS Dept. of Univ. of Utah
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 10 / 78 READING PAPER TAKES TO THE EDGE From “The illustrated guide to a Ph.D.” by Matt Might, CS Dept. of Univ. of Utah
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 11 / 78 THEN, YOU FOCUS . . . From “The illustrated guide to a Ph.D.” by Matt Might, CS Dept. of Univ. of Utah
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 12 / 78 . . . PUSHING THE BOUNDARY From “The illustrated guide to a Ph.D.” by Matt Might, CS Dept. of Univ. of Utah
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 13 / 78 . . . UNTIL THE BOUNDARY GIVES AWAY! From “The illustrated guide to a Ph.D.” by Matt Might, CS Dept. of Univ. of Utah
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 14 / 78 HOW THE WORLD LOOKS NOW! From “The illustrated guide to a Ph.D.” by Matt Might, CS Dept. of Univ. of Utah
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 15 / 78 BUT, PLEASE, DON’T FORGET THE BIGGER PICTURE! From “The illustrated guide to a Ph.D.” by Matt Might, CS Dept. of Univ. of Utah
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 16 / 78 THE MOTIVATIONS TO THESE 2 SEMINARS • Doing literature review is not optional (obviously! ) • It must follow some guidelines, rules (a process). . . • . . . and use advanced tools and techniques • Objectives: completeness, soundness, usefulness, reusability
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 17 / 78 THE ORGANIZATION OF THE SEMINARS • Two parts • First seminar on Systematic Literature Review: • principles • the process • some examples • Second seminar on Tools and Instruments to help on bibliographic search and SLRs
1 - Systematic Literature Reviews: introduction and methods
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 19 / 78 MAP 1 Preamble to the two seminars 2 Why Systematic Literature Reviews? 3 Systematic Literature Reviews 4 The entire process 5 Phase 1 - Planning 6 Phase 2 - Conducting 7 Phase 3 - Reporting 8 An example 9 Conclusions and comments
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 20 / 78 FIRST, WHAT IS A SYSTEMATIC LITERATURE REVIEW Definition of Systematic Literature Review (SLR) A Systematic Literature Review is a means of identifying, evaluating and interpreting all available research relevant to a particular research question, or topic area, or phenomenon of interest. • Original guidelines proposed by Barbara Kitchenham (2004) and, then from B.Kitchenham et. al (2004) for “Evidence-based Software Engineering”. • An evolution of experiences and process used in biomedical and healthcare context • Further evolved and used in several contexts (among them, whole CS, but also Social Science, Business and Economics)
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 21 / 78 LITERATURE REVIEW/SURVEYS • What we really love to find: the most recent the better! • A study of all the available research work and results in a given subject • Possibly, with some critical approach to distinguish between well established results and still open issues • If it includes also ongoing work and future challenges, bingo! • Often called “secondary sources” as opposed to the primary, i.e. the original papers, where experiments/proofs are reported. • Producing a literature review is part of graduate/post-graduate work: a chapter of your PhD thesis will be a Literature Review of the topics you are dealing with • ⇒ a foundation to build novelty upon!
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 22 / 78 REASONS FOR A SYSTEMATIC LITERATURE REVIEW • A standard path: from informal reviews and survey, moving on systematic searches of the literature • Important to summarize evidence concerning a methodology/technology • Helpful to identify gaps in current research (hint on further investigation needed!) • Crucial to provide a framework/background to position new research activities
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 23 / 78 NOT ONLY FOR PHD STUDENTS DEVOTED TO ACADEMIA! • The ability to study and gain knowledge on current, state-of-the-art, technologies and methodologies is crucial for ICT • Not only in academia, a long path of continuous knowledge update is waiting you out there! • “New topics in CS are going to pop up like popcorn in the pan” (A.Omicini) • Crucial to be able to ensure that 1 you move quickly to the result (well documented process) 2 nothing significant was left out 3 you can easily share knowledge within the organisation
1 - Systematic Literature Reviews: introduction and methods
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 25 / 78 MAP 1 Preamble to the two seminars 2 Why Systematic Literature Reviews? 3 Systematic Literature Reviews 4 The entire process 5 Phase 1 - Planning 6 Phase 2 - Conducting 7 Phase 3 - Reporting 8 An example 9 Conclusions and comments
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 26 / 78 THE BIRTH OF SYSTEMATIC LITERATURE REVIEW • It is first developed in the healthcare domain • The need for a well-founded methodological approach to literature results emerges in medical research in the 90s • It gets popular in the 2000, and then formalised in terms of the so called “Cochrane Reviews” • Barbara Kitchenham, first, proposes it as a tool for the “Evidence-based Software Engineering” in 2004 (with a TR) and later adopted by SE and, by generalization, to many subfields of Computer Science
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 27 / 78 IMPORTANCE OF A SYSTEMATIC LITERATURE REVIEW • Literature review must be thorough • Literature review must be fair • A predefined search strategy (protocol) help the assessment of the completeness • Every effort to identify research that does not support their preferred search hypothesis
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 28 / 78 PLUS AND MINUS OF SYSTEMATIC LITERATURE REVIEW • Requires considerably more effort • Information about effects of a phenomenon across a wide range of methods • The phenomenon can be assessed as robust and transferable • Combines qualitative with quantitative methods (on metadata) • Agreed-upon, sound (⇒ publishable) result
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 29 / 78 FEATURES OF A SLR • Definition of a review protocol that specifies the research question being addressed and the methods that will be used to perform the review. • Definition of a documented, search strategy that aims to detect as much of the relevant literature as possible. • Documentation is critical to assess its rigour and completeness. • It require explicit inclusion and exclusion criteria to assess each potential primary study. • Definition of the information to be analyzed from each primary study, including quality criteria • A systematic review is a prerequisite for quantitative meta-analysis
1 - Systematic Literature Reviews: introduction and methods
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 31 / 78 MAP 1 Preamble to the two seminars 2 Why Systematic Literature Reviews? 3 Systematic Literature Reviews 4 The entire process 5 Phase 1 - Planning 6 Phase 2 - Conducting 7 Phase 3 - Reporting 8 An example 9 Conclusions and comments
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 32 / 78 THE REVIEW PROCESS • A systematic review involves several discrete activities. • The process is meant as a standardisation of several existing guidelines for systematic reviews • The advantage of a standard, well-defined, process to be followed • Three main phases (subdivided in stages): 1 Planning the Review, 2 Conducting the Review, 3 Reporting the Review. • Although apparently sequential, many of the stages involve iteration. • Activities initiated in the Planning (i.e., the protocol development stage) can be later refined when the review takes place.
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 33 / 78 THE STAGES • The stages associated with Planning the review are: 1 Identification of the need for a review 2 Development of a review protocol. • The stages associated with Conducting the review are: 1 Identification of research 2 Selection of primary studies 3 Study quality assessment 4 Data extraction & monitoring 5 Data synthesis. • Reporting the review is a single stage activity
1 - Systematic Literature Reviews: introduction and methods
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 35 / 78 MAP 1 Preamble to the two seminars 2 Why Systematic Literature Reviews? 3 Systematic Literature Reviews 4 The entire process 5 Phase 1 - Planning 6 Phase 2 - Conducting 7 Phase 3 - Reporting 8 An example 9 Conclusions and comments
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 36 / 78 THE STAGES OF THE PLANNING PHASE • The stages associated with Planning the review are: 1 Identification of the need for a review 2 Development of a review protocol.
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 37 / 78 1.1 THE NEED FOR A SYSTEMATIC REVIEW • It arises from the requirement of researchers to summarise all existing information about some phenomenon in a thorough and unbiased manner. • The purpose can be to draw more general conclusion about some phenomenon than is possible from individual studies, • Act as a prelude to further research activities • Researchers should ensure that a systematic review is necessary. • identify and review any existing systematic reviews of the phenomenon of interest against appropriate evaluation criteria.
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 38 / 78 HOW TO CHECK EXISTING REVIEWS • What are the review’s objectives? • What sources were searched to identify primary studies? Were there any restrictions? Missing sources? • What were the inclusion/exclusion criteria and how were they applied? • What criteria were used to assess the quality of primary studies and how were they applied? • How were the data extracted from the primary studies? • How were the data synthesised? How were differences between studies investigated? How were the data combined? Was it reasonable to combine the studies? Do the conclusions flow from the evidence?
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 39 / 78 1.2 THE DEVELOPMENT OF A REVIEW PROTOCOL • A review protocol specifies the methods that will be used to undertake a specific systematic review. • A pre-defined protocol is necessary to reduce the possibility researcher bias: fairness. • Without a protocol, it is possible that the selection of individual studies or the analysis may be driven by researcher expectations. • Critical point: in some disciplines (i.e. medicine) review protocols are usually submitted to peer review.
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 40 / 78 COMPONENTS OF A PROTOCOL • Background: rationale for the survey. • Research questions that the review is intended answer. • The search strategy for primary studies: search terms and resources to be searched • Study selection (include/exclude) criteria and procedures. • Study quality assessment checklists and procedures. • Data extraction strategy to define how the information required from each primary study would be obtained • in case of required manipulation, the protocol should make it explicit • Synthesis of the extracted data • Project timetable
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 41 / 78 RESEARCH QUESTIONS • A specific problem (P) is tackled using some specific constraints, methods and/or approaches (C) to develop a system, application or algorithm (S). • Goal: what existing solutions are available, how they compare, what the strength of the evidence is and what implications these solutions have. • RQ1: What are the existing solutions to P? • RQ2: How does the different solutions found by addressing RQ1 compare to each other with respect to C? • RQ3: What is the strength of the evidence in support of the different solutions? • RQ4: What implications will these findings have when creating S?
1 - Systematic Literature Reviews: introduction and methods
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 43 / 78 MAP 1 Preamble to the two seminars 2 Why Systematic Literature Reviews? 3 Systematic Literature Reviews 4 The entire process 5 Phase 1 - Planning 6 Phase 2 - Conducting 7 Phase 3 - Reporting 8 An example 9 Conclusions and comments
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 44 / 78 THE STAGES OF THE CONDUCTING PHASE • The stages associated with Conducting the review are: 1 2.1 Identification of research 2 2.2 Selection of primary studies 3 2.3 Study quality assessment 4 2.4 Data extraction & monitoring 5 2.5 Data synthesis.
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 45 / 78 2.1 IDENTIFICATION OF RESEARCH • The aim of a systematic review is to find as many primary studies relating to the research question as possible using an unbiased search strategy. • Determine and follow a search strategy (in collaboration with librarian) • 1. Preliminary searches aimed at both identifying existing systematic reviews and assessing the volume of potentially relevant studies. • 2. Trial searchers using various combinations of search terms derived from the research question (from many sources) • 3. Reviews of research results • 4. Consultations with experts in the field
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 46 / 78 SEARCH TERMS • Break down the question into individual facets • Draw up a list of synonyms, abbreviations, and alternative spellings. • Pick other terms can be obtained by considering subject headings used in journals and data bases. • Sophisticated search strings can then be constructed using Boolean AND’s and OR’s.
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 47 / 78 INFORMATION SOURCES • ACM digital library, • IEEE Xplore, • ISI web of knowledge, • ScienceDirect, • CiteSeer, • SpringerLink • DBLP • Wiley Inter Science • . . . • For domain specific sources, experts in the domain will normally know which are the best.
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 48 / 78 SOME ADDITIONAL FEATURES OF THIS STAGE • Avoid publication bias: • positive results are more likely to be published than negative results • Recent results: • scan repository of unpublished-preprint results (arXiv, Google Scholar, . . . ) • Document the search (also with the search query): • it must be possible to assess the thoroughness of the search • Useful to be helped with bibliographic management tools
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 49 / 78 2.2 STUDY SELECTION • Choose the study selection criteria to identify those primary studies that provide direct evidence about the research questions. • Som generally used criteria for removal: duplicates, same study in different sources (conference ⇒ journal), after/before a certain date • Avoid language bias • Avoid “clique” bias (well-known institutions, labs, researchers)
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 50 / 78 2.3 STUDY QUALITY ASSESSMENT • Choose the study selection criteria to identify those primary studies that provide direct evidence about the research questions. • Som generally used criteria for removal: duplicates, same study in different sources (conference ⇒ journal), after/before a certain date • The protocol should define exactly which inclusion and quality criteria are employed • Avoid language bias • Avoid “clique” bias (well-known institutions, labs, researchers) • Example of inclusion criteria: “The study’s main concern is P”, “The study focuses on C”,“ The study describes an S”
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 51 / 78 THREE-STEPS SCREENING • Title and Abstract: apply the inclusion criteria • Introduction and conclusions: apply the inclusion criteria • Full text: apply the inclusion criteria and the quality criteria
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 52 / 78 QUALITY CRITERIA • They are designed to answer “What is the strength of the evidence in support of the different solutions?” • Examples are: • QC 1: Is there is a clear statement of the aim of the research? • QC 2: Is the study is put into context of other studies and research? • QC 3: Are system or algorithmic design decisions justified? • QC 4: Is the test data set reproducible? • QC 5: Is the study algorithm reproducible? • QC 6: Is the experimental procedure throughly explained and reproducible? • QC 7: Is it clearly stated in the study which other algorithms the study’s algorithm(s) have been compared with? • QC 8: Are the performance metrics used in the study explained and justified? • QC 9: Are the test results thoroughly analysed? • QC 10: Does the test evidence support the findings presented? • Example: score each QC in 0, 1/2, 1 and add them up for global rank
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 53 / 78 2.4 DATA EXTRACTION • Objective: to design data extraction forms to accurately record the information researchers obtain from the primary studies • It will define a set of numerical values that should be extracted for each study (e.g. number of subjects, performance parameters, . . . ) • To ensure consistency among multiple researcher, some papers should be reviewed by all researchers so that inter-researcher consistency can be assessed.
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 54 / 78 2.5 DATA SYNTHESIS • Data synthesis involves collating and summarising the results of the included primary studies. • Synthesis can be descriptive (non-quantitative) • It is sometimes possible to complement a descriptive synthesis with a quantitative summary • It is important to identify whether results from studies are consistent one with another (i.e. homogeneous) or inconsistent (e.g. heterogeneous) in a tabulated/graphical form
1 - Systematic Literature Reviews: introduction and methods
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 56 / 78 MAP 1 Preamble to the two seminars 2 Why Systematic Literature Reviews? 3 Systematic Literature Reviews 4 The entire process 5 Phase 1 - Planning 6 Phase 2 - Conducting 7 Phase 3 - Reporting 8 An example 9 Conclusions and comments
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 57 / 78 STRUCTURE OF A REPORT OF SLR - 1 • Title • The title should be short but informative. It should be based on the question being asked. In journal papers, it should indicate that the study is a systematic review. • Abstract • Context/Background: the importance of the research questions addressed • Objectives: The questions addressed • Methods: Data Sources, Study selection, Quality Assessment and Data extraction • Results: Main finding • Conclusions: Implications for practice and future research
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 58 / 78 STRUCTURE OF A REPORT OF SLR - 2 • Background: Justification of the need for the review. Summary of previous reviews • Review questions: Each review question should be specified • Review Methods: • Data sources and search strategy • Study selection • Study quality assessment • Data extraction • Data synthesis • Included and excluded studies • Results: Findings (description of primary studies and results of any quantitative summaries)
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 59 / 78 STRUCTURE OF A REPORT OF SLR - 3 • Discussion: • Principal findings • Strengths and Weaknesses • Meaning of findings (applicability/generalisability) • Conclusions: recommendations (practical implementations) and future research • Results: Findings (description of primary studies and results of any quantitative summaries) • Acknowledgments • Conflict of Interest • References • Appendices
1 - Systematic Literature Reviews: introduction and methods
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 61 / 78 MAP 1 Preamble to the two seminars 2 Why Systematic Literature Reviews? 3 Systematic Literature Reviews 4 The entire process 5 Phase 1 - Planning 6 Phase 2 - Conducting 7 Phase 3 - Reporting 8 An example 9 Conclusions and comments
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 62 / 78 A PAPER AS AN EXAMPLE • Artem Voronkov, Leonardo Horn Iwaya, Leonardo A. Martucci, and Stefan Lindskog. 2017. “Systematic Literature Review on Usability of Firewall Configuration”. ACM Comput. Surv. 50, 6, Article 87 (December 2017) 35 pages. • A nice survey on Firewall usability • Published on one of the major sources for Surveys
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 63 / 78 THE ABSTRACT
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 64 / 78 THE ABSTRACT WITH ANNOTATIONS
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 65 / 78 SOURCES
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 66 / 78 RESEARCH QUESTIONS
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 67 / 78 INCLUSION CRITERIA
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 68 / 78 INDEX TERMS: TWO QUERIES
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 69 / 78 THE RESULTING QUERIES
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 70 / 78 QUALITY CRITERIA
1 - Systematic Literature Reviews: introduction and methods
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 72 / 78 MAP 1 Preamble to the two seminars 2 Why Systematic Literature Reviews? 3 Systematic Literature Reviews 4 The entire process 5 Phase 1 - Planning 6 Phase 2 - Conducting 7 Phase 3 - Reporting 8 An example 9 Conclusions and comments
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 73 / 78 COMPARISON OF SLR VS LITERATURE SURVEY • A SLR defines a review protocol that specifies the (1) research question being addressed and (2) the methods that will be used to perform the review • A SLR defines the search strategy to help readers to determine its thoroughness and fairness (and also repeatability, up to a certain extent) • searches on digital libraries cannot be replicated • A SLR specifies quality criteria to evaluate each primary study.
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 74 / 78 PERSONAL CONSIDERATIONS • Important to follow a process when documenting literature surveys • Secondary sources are important: citations and publication venues are available out there! • Useful tools and instruments can facilitate the effort • If not publishable, SLRs make, at least, a great chapter of a PhD Thesis: thoroughness, fairness and replicability
1 - Systematic Literature Reviews: introduction and methods
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 76 / 78 READING MATERIAL AND CREDITS • Material and references available at www.isislab.it under “Research” • Slides are available on Slideshare under Open licence Common Creative - Attribution 4.0 International (CC BY 4.0)
Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 77 / 78 QUESTIONS? COMMENTS? ANYTHING?
1 - Systematic Literature Reviews: introduction and methods

More Related Content

PDF
2 - Systematic Literature Reviews: tools
Vittorio Scarano
 
PDF
Systematic Literature Review & Mapping
Education in the Knowledge Society PhD
 
PDF
Introduction to Systematic Literature Review method
Norsaremah Salleh
 
PDF
Systematic Literature Review
Tiago Silva da Silva
 
PPT
Writing research proposal
Kaimrc_Rss_Jd
 
PPTX
Research tools and techniques
Sujith Jayaprakash
 
PDF
Predicting student performance using aggregated data sources
Olugbenga Wilson Adejo
 
PPTX
Citation analysis
Masoud Mohammadi
 
2 - Systematic Literature Reviews: tools
Vittorio Scarano
 
Systematic Literature Review & Mapping
Education in the Knowledge Society PhD
 
Introduction to Systematic Literature Review method
Norsaremah Salleh
 
Systematic Literature Review
Tiago Silva da Silva
 
Writing research proposal
Kaimrc_Rss_Jd
 
Research tools and techniques
Sujith Jayaprakash
 
Predicting student performance using aggregated data sources
Olugbenga Wilson Adejo
 
Citation analysis
Masoud Mohammadi
 

What's hot (20)

PPTX
Research Methodology-02: Quality Indices
Bhasker Vijaykumar Bhatt
 
PPTX
How to write a good postgraduate research proposal
BC Chew
 
PPTX
Preparing a manuscript
lemberger
 
PPTX
Data Exploration.pptx
PerumalPitchandi
 
PPTX
Scholarly Communication 101
Claire Sewell
 
PDF
Scientometric Analysis
sumitbanshal
 
PPT
Data indexing presentation
gmbmanikandan
 
PDF
Synopsis - (How to write a effective Synopsis?)
Umapati Baragi
 
PPTX
Introduction to research methodology
Praveen Minz
 
PPTX
Introduction to Databases
National College of Art & Design Library
 
PDF
Bibliometrics - an overview
claudia cavicchi
 
PPTX
presentation on Sampling
PANKAJ KUMAR JENA
 
PDF
Systematic Literature Reviews and Systematic Mapping Studies
alessio_ferrari
 
PPT
4 Literature Search Techniques 2 Strategic Searching
richard kemp
 
PPT
journal and impact factor
Working as a Lecturer
 
PDF
LaTeX-Presentation
Komala Anamalamudi
 
PDF
Naive Bayes
CloudxLab
 
PPTX
Methods of data collection
Don Bosco College Sulthan Bathery
 
PPTX
SYNOPSIS WRITING
Prof. Mridul Panditrao
 
PPT
Format Of Synopsis
Abdul Rahman Khawaja
 
Research Methodology-02: Quality Indices
Bhasker Vijaykumar Bhatt
 
How to write a good postgraduate research proposal
BC Chew
 
Preparing a manuscript
lemberger
 
Data Exploration.pptx
PerumalPitchandi
 
Scholarly Communication 101
Claire Sewell
 
Scientometric Analysis
sumitbanshal
 
Data indexing presentation
gmbmanikandan
 
Synopsis - (How to write a effective Synopsis?)
Umapati Baragi
 
Introduction to research methodology
Praveen Minz
 
Introduction to Databases
National College of Art & Design Library
 
Bibliometrics - an overview
claudia cavicchi
 
presentation on Sampling
PANKAJ KUMAR JENA
 
Systematic Literature Reviews and Systematic Mapping Studies
alessio_ferrari
 
4 Literature Search Techniques 2 Strategic Searching
richard kemp
 
journal and impact factor
Working as a Lecturer
 
LaTeX-Presentation
Komala Anamalamudi
 
Naive Bayes
CloudxLab
 
Methods of data collection
Don Bosco College Sulthan Bathery
 
SYNOPSIS WRITING
Prof. Mridul Panditrao
 
Format Of Synopsis
Abdul Rahman Khawaja
 
Ad

Similar to 1 - Systematic Literature Reviews: introduction and methods (20)

PPTX
How to conduct systematic literature review
Kashif Hussain
 
PPTX
Literature Review (Review of Related Literature - Research Methodology)
Dilip Barad
 
PDF
presentationlitreview-2ND COT PR2.pdf
AnalieCabanlit1
 
PPTX
Literature review for a dissertation: a step-by-step guide
Olga Koz, DM, MLS
 
PDF
SLR.pdf
NargesShahHoseini
 
PDF
SLR.pdf
NargesShahHoseini
 
PDF
Approaching Literature Review For Academic Purposes The Literature Review Ch...
Sandra Valenzuela
 
PPTX
lecture 3(Literature Review).pptx........
CharuNangia
 
PPT
Presentationlitreview 130824122504-phpapp02
Brian O'Connell
 
PPTX
Introduction to systematic literature review
ILRI
 
PPT
Literature Review
BC Chew
 
PPTX
LITERATURE REVIEW- HOW TO MAKE YOURS COUNT?
Florence Suganya
 
PDF
Sgd literature review-1
Sanjeev Deshmukh
 
PPT
Writing a literature review by Tim Lang
Learning Development Centre
 
PPT
Systematic Literature Reviews : Concise Overview
youkayaslam
 
PDF
systematic review info.pdf
raazia gul
 
PDF
Workshop Slides on Research Proposal and Procedure 190415
Hiram Ting
 
PPTX
Systematic literature review technique.pptx
TANMAY DAS GUPTA
 
PPTX
A lecture on a Literature review (1).pptx
MDHOSSAINAHMED2
 
PPTX
How to do Literature Review For Dissertations and Research Paper
HomeworkHelpExperts
 
How to conduct systematic literature review
Kashif Hussain
 
Literature Review (Review of Related Literature - Research Methodology)
Dilip Barad
 
presentationlitreview-2ND COT PR2.pdf
AnalieCabanlit1
 
Literature review for a dissertation: a step-by-step guide
Olga Koz, DM, MLS
 
SLR.pdf
NargesShahHoseini
 
SLR.pdf
NargesShahHoseini
 
Approaching Literature Review For Academic Purposes The Literature Review Ch...
Sandra Valenzuela
 
lecture 3(Literature Review).pptx........
CharuNangia
 
Presentationlitreview 130824122504-phpapp02
Brian O'Connell
 
Introduction to systematic literature review
ILRI
 
Literature Review
BC Chew
 
LITERATURE REVIEW- HOW TO MAKE YOURS COUNT?
Florence Suganya
 
Sgd literature review-1
Sanjeev Deshmukh
 
Writing a literature review by Tim Lang
Learning Development Centre
 
Systematic Literature Reviews : Concise Overview
youkayaslam
 
systematic review info.pdf
raazia gul
 
Workshop Slides on Research Proposal and Procedure 190415
Hiram Ting
 
Systematic literature review technique.pptx
TANMAY DAS GUPTA
 
A lecture on a Literature review (1).pptx
MDHOSSAINAHMED2
 
How to do Literature Review For Dissertations and Research Paper
HomeworkHelpExperts
 
Ad

More from Vittorio Scarano (12)

PDF
Impostors and Computer Science
Vittorio Scarano
 
PDF
Offloading Computation to the Edge
Vittorio Scarano
 
PDF
Storylet: prima giornata di formazione
Vittorio Scarano
 
PDF
Hetor per la Rete dei Licei Artistici della Campania
Vittorio Scarano
 
PDF
Solid: an introduction
Vittorio Scarano
 
PDF
Introduzione agli Open Data
Vittorio Scarano
 
PPTX
Progetto Hetor: La Campania da riscoprire
Vittorio Scarano
 
PDF
Scratch da 0 a 100 Km/h in 3 ore, scarse!
Vittorio Scarano
 
PDF
Learning by doing: il pensiero computazionale
Vittorio Scarano
 
PDF
La presenza su LinkedIn (ma non solo!)
Vittorio Scarano
 
PDF
Come "costruire" il curriculum di un Informatico
Vittorio Scarano
 
PDF
Come presentare se stessi ed il proprio lavoro
Vittorio Scarano
 
Impostors and Computer Science
Vittorio Scarano
 
Offloading Computation to the Edge
Vittorio Scarano
 
Storylet: prima giornata di formazione
Vittorio Scarano
 
Hetor per la Rete dei Licei Artistici della Campania
Vittorio Scarano
 
Solid: an introduction
Vittorio Scarano
 
Introduzione agli Open Data
Vittorio Scarano
 
Progetto Hetor: La Campania da riscoprire
Vittorio Scarano
 
Scratch da 0 a 100 Km/h in 3 ore, scarse!
Vittorio Scarano
 
Learning by doing: il pensiero computazionale
Vittorio Scarano
 
La presenza su LinkedIn (ma non solo!)
Vittorio Scarano
 
Come "costruire" il curriculum di un Informatico
Vittorio Scarano
 
Come presentare se stessi ed il proprio lavoro
Vittorio Scarano
 

Recently uploaded (20)

PDF
STATICS OF THE RIGID BODIES Hibbelers.pdf
pascualasturiaskaye4
 
PDF
Complications of Minimal Access Surgery at WLH
mohitsuren827
 
PPTX
Tissue processing ( HISTOPATHOLOGICAL TECHNIQUE
mathiasmelwyn7
 
PDF
Microbial disease of the cardiovascular and lymphatic systems
KeyaSharma
 
PDF
OBE - B.A.(HON'S) IN INTERIOR ARCHITECTURE -Ar.MOHIUDDIN.pdf
ArchitectAFMMohiuddi
 
PDF
Paper A Mock Exam 9_ Attempt review.pdf.
SameeraNaveed2
 
PPTX
school management -TNTEU- B.Ed., Semester II Unit 1.pptx
NihumathunnisaH
 
PPTX
1st Inaugural Professorial Lecture held on 19th February 2020 (Governance and...
kawisojames10
 
PDF
2.FourierTransform-ShortQuestionswithAnswers.pdf
Raji Murugan
 
PDF
Anesthesia in Laparoscopic Surgery in India
mohitsuren827
 
PPTX
History, Philosophy and sociology of education (1).pptx
tmdth1
 
PDF
The Lost Whites of Pakistan by Jahanzaib Mughal.pdf
jahanzaibmughal6
 
PPTX
Lesson notes of climatology university.
sgladness67
 
PDF
Trump Administration's workforce development strategy
Mebane Rash
 
PDF
ChatGPT for Dummies - Pam Baker Ccesa007.pdf
Demetrio Ccesa Rayme
 
PDF
GENETICS IN BIOLOGY IN SECONDARY LEVEL FORM 3
ElishamichaelSamwel
 
PDF
LDMMIA Reiki Yoga Finals Review Spring Summer
LDMMia Reiki Lectures
 
PDF
01-Introduction-to-Information-Management.pdf
PrinceIbarra
 
PPTX
PPT- ENG7_QUARTER1_LESSON1_WEEK1. IMAGERY -DESCRIPTIONS pptx.pptx
KMDee
 
PDF
Computing-Curriculum for Schools in Ghana
abigailanane203
 
STATICS OF THE RIGID BODIES Hibbelers.pdf
pascualasturiaskaye4
 
Complications of Minimal Access Surgery at WLH
mohitsuren827
 
Tissue processing ( HISTOPATHOLOGICAL TECHNIQUE
mathiasmelwyn7
 
Microbial disease of the cardiovascular and lymphatic systems
KeyaSharma
 
OBE - B.A.(HON'S) IN INTERIOR ARCHITECTURE -Ar.MOHIUDDIN.pdf
ArchitectAFMMohiuddi
 
Paper A Mock Exam 9_ Attempt review.pdf.
SameeraNaveed2
 
school management -TNTEU- B.Ed., Semester II Unit 1.pptx
NihumathunnisaH
 
1st Inaugural Professorial Lecture held on 19th February 2020 (Governance and...
kawisojames10
 
2.FourierTransform-ShortQuestionswithAnswers.pdf
Raji Murugan
 
Anesthesia in Laparoscopic Surgery in India
mohitsuren827
 
History, Philosophy and sociology of education (1).pptx
tmdth1
 
The Lost Whites of Pakistan by Jahanzaib Mughal.pdf
jahanzaibmughal6
 
Lesson notes of climatology university.
sgladness67
 
Trump Administration's workforce development strategy
Mebane Rash
 
ChatGPT for Dummies - Pam Baker Ccesa007.pdf
Demetrio Ccesa Rayme
 
GENETICS IN BIOLOGY IN SECONDARY LEVEL FORM 3
ElishamichaelSamwel
 
LDMMIA Reiki Yoga Finals Review Spring Summer
LDMMia Reiki Lectures
 
01-Introduction-to-Information-Management.pdf
PrinceIbarra
 
PPT- ENG7_QUARTER1_LESSON1_WEEK1. IMAGERY -DESCRIPTIONS pptx.pptx
KMDee
 
Computing-Curriculum for Schools in Ghana
abigailanane203
 

1 - Systematic Literature Reviews: introduction and methods

  • 1. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 1 / 78 1 - Systematic literature reviews: introduction and methods Vittorio Scarano Dipartimento di Informatica, Universit`a di Salerno (Italy) vitsca@unisa.it PhD in Computer Science, University of Salerno (Italy) January 14th, 2020
  • 2. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 2 / 78 LESSON PLAN 1 Preamble to the two seminars 2 Why Systematic Literature Reviews? 3 Systematic Literature Reviews 4 The entire process 5 Phase 1 - Planning 6 Phase 2 - Conducting 7 Phase 3 - Reporting 8 An example 9 Conclusions and comments
  • 4. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 4 / 78 MAP 1 Preamble to the two seminars 2 Why Systematic Literature Reviews? 3 Systematic Literature Reviews 4 The entire process 5 Phase 1 - Planning 6 Phase 2 - Conducting 7 Phase 3 - Reporting 8 An example 9 Conclusions and comments
  • 5. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 5 / 78 A CIRCLE THAT CONTAINS HUMAN KNOWLEDGE. . . From “The illustrated guide to a Ph.D.” by Matt Might, CS Dept. of Univ. of Utah
  • 6. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 6 / 78 BY END OF ELEMENTARY SCHOOL . . . From “The illustrated guide to a Ph.D.” by Matt Might, CS Dept. of Univ. of Utah
  • 7. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 7 / 78 BY END OF HIGH SCHOOL From “The illustrated guide to a Ph.D.” by Matt Might, CS Dept. of Univ. of Utah
  • 8. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 8 / 78 A BACHELOR: A SPECIALTY From “The illustrated guide to a Ph.D.” by Matt Might, CS Dept. of Univ. of Utah
  • 9. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 9 / 78 A MASTER DEEPENS THE SPECIALTY From “The illustrated guide to a Ph.D.” by Matt Might, CS Dept. of Univ. of Utah
  • 10. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 10 / 78 READING PAPER TAKES TO THE EDGE From “The illustrated guide to a Ph.D.” by Matt Might, CS Dept. of Univ. of Utah
  • 11. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 11 / 78 THEN, YOU FOCUS . . . From “The illustrated guide to a Ph.D.” by Matt Might, CS Dept. of Univ. of Utah
  • 12. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 12 / 78 . . . PUSHING THE BOUNDARY From “The illustrated guide to a Ph.D.” by Matt Might, CS Dept. of Univ. of Utah
  • 13. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 13 / 78 . . . UNTIL THE BOUNDARY GIVES AWAY! From “The illustrated guide to a Ph.D.” by Matt Might, CS Dept. of Univ. of Utah
  • 14. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 14 / 78 HOW THE WORLD LOOKS NOW! From “The illustrated guide to a Ph.D.” by Matt Might, CS Dept. of Univ. of Utah
  • 15. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 15 / 78 BUT, PLEASE, DON’T FORGET THE BIGGER PICTURE! From “The illustrated guide to a Ph.D.” by Matt Might, CS Dept. of Univ. of Utah
  • 16. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 16 / 78 THE MOTIVATIONS TO THESE 2 SEMINARS • Doing literature review is not optional (obviously! ) • It must follow some guidelines, rules (a process). . . • . . . and use advanced tools and techniques • Objectives: completeness, soundness, usefulness, reusability
  • 17. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 17 / 78 THE ORGANIZATION OF THE SEMINARS • Two parts • First seminar on Systematic Literature Review: • principles • the process • some examples • Second seminar on Tools and Instruments to help on bibliographic search and SLRs
  • 19. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 19 / 78 MAP 1 Preamble to the two seminars 2 Why Systematic Literature Reviews? 3 Systematic Literature Reviews 4 The entire process 5 Phase 1 - Planning 6 Phase 2 - Conducting 7 Phase 3 - Reporting 8 An example 9 Conclusions and comments
  • 20. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 20 / 78 FIRST, WHAT IS A SYSTEMATIC LITERATURE REVIEW Definition of Systematic Literature Review (SLR) A Systematic Literature Review is a means of identifying, evaluating and interpreting all available research relevant to a particular research question, or topic area, or phenomenon of interest. • Original guidelines proposed by Barbara Kitchenham (2004) and, then from B.Kitchenham et. al (2004) for “Evidence-based Software Engineering”. • An evolution of experiences and process used in biomedical and healthcare context • Further evolved and used in several contexts (among them, whole CS, but also Social Science, Business and Economics)
  • 21. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 21 / 78 LITERATURE REVIEW/SURVEYS • What we really love to find: the most recent the better! • A study of all the available research work and results in a given subject • Possibly, with some critical approach to distinguish between well established results and still open issues • If it includes also ongoing work and future challenges, bingo! • Often called “secondary sources” as opposed to the primary, i.e. the original papers, where experiments/proofs are reported. • Producing a literature review is part of graduate/post-graduate work: a chapter of your PhD thesis will be a Literature Review of the topics you are dealing with • ⇒ a foundation to build novelty upon!
  • 22. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 22 / 78 REASONS FOR A SYSTEMATIC LITERATURE REVIEW • A standard path: from informal reviews and survey, moving on systematic searches of the literature • Important to summarize evidence concerning a methodology/technology • Helpful to identify gaps in current research (hint on further investigation needed!) • Crucial to provide a framework/background to position new research activities
  • 23. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 23 / 78 NOT ONLY FOR PHD STUDENTS DEVOTED TO ACADEMIA! • The ability to study and gain knowledge on current, state-of-the-art, technologies and methodologies is crucial for ICT • Not only in academia, a long path of continuous knowledge update is waiting you out there! • “New topics in CS are going to pop up like popcorn in the pan” (A.Omicini) • Crucial to be able to ensure that 1 you move quickly to the result (well documented process) 2 nothing significant was left out 3 you can easily share knowledge within the organisation
  • 25. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 25 / 78 MAP 1 Preamble to the two seminars 2 Why Systematic Literature Reviews? 3 Systematic Literature Reviews 4 The entire process 5 Phase 1 - Planning 6 Phase 2 - Conducting 7 Phase 3 - Reporting 8 An example 9 Conclusions and comments
  • 26. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 26 / 78 THE BIRTH OF SYSTEMATIC LITERATURE REVIEW • It is first developed in the healthcare domain • The need for a well-founded methodological approach to literature results emerges in medical research in the 90s • It gets popular in the 2000, and then formalised in terms of the so called “Cochrane Reviews” • Barbara Kitchenham, first, proposes it as a tool for the “Evidence-based Software Engineering” in 2004 (with a TR) and later adopted by SE and, by generalization, to many subfields of Computer Science
  • 27. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 27 / 78 IMPORTANCE OF A SYSTEMATIC LITERATURE REVIEW • Literature review must be thorough • Literature review must be fair • A predefined search strategy (protocol) help the assessment of the completeness • Every effort to identify research that does not support their preferred search hypothesis
  • 28. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 28 / 78 PLUS AND MINUS OF SYSTEMATIC LITERATURE REVIEW • Requires considerably more effort • Information about effects of a phenomenon across a wide range of methods • The phenomenon can be assessed as robust and transferable • Combines qualitative with quantitative methods (on metadata) • Agreed-upon, sound (⇒ publishable) result
  • 29. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 29 / 78 FEATURES OF A SLR • Definition of a review protocol that specifies the research question being addressed and the methods that will be used to perform the review. • Definition of a documented, search strategy that aims to detect as much of the relevant literature as possible. • Documentation is critical to assess its rigour and completeness. • It require explicit inclusion and exclusion criteria to assess each potential primary study. • Definition of the information to be analyzed from each primary study, including quality criteria • A systematic review is a prerequisite for quantitative meta-analysis
  • 31. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 31 / 78 MAP 1 Preamble to the two seminars 2 Why Systematic Literature Reviews? 3 Systematic Literature Reviews 4 The entire process 5 Phase 1 - Planning 6 Phase 2 - Conducting 7 Phase 3 - Reporting 8 An example 9 Conclusions and comments
  • 32. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 32 / 78 THE REVIEW PROCESS • A systematic review involves several discrete activities. • The process is meant as a standardisation of several existing guidelines for systematic reviews • The advantage of a standard, well-defined, process to be followed • Three main phases (subdivided in stages): 1 Planning the Review, 2 Conducting the Review, 3 Reporting the Review. • Although apparently sequential, many of the stages involve iteration. • Activities initiated in the Planning (i.e., the protocol development stage) can be later refined when the review takes place.
  • 33. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 33 / 78 THE STAGES • The stages associated with Planning the review are: 1 Identification of the need for a review 2 Development of a review protocol. • The stages associated with Conducting the review are: 1 Identification of research 2 Selection of primary studies 3 Study quality assessment 4 Data extraction & monitoring 5 Data synthesis. • Reporting the review is a single stage activity
  • 35. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 35 / 78 MAP 1 Preamble to the two seminars 2 Why Systematic Literature Reviews? 3 Systematic Literature Reviews 4 The entire process 5 Phase 1 - Planning 6 Phase 2 - Conducting 7 Phase 3 - Reporting 8 An example 9 Conclusions and comments
  • 36. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 36 / 78 THE STAGES OF THE PLANNING PHASE • The stages associated with Planning the review are: 1 Identification of the need for a review 2 Development of a review protocol.
  • 37. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 37 / 78 1.1 THE NEED FOR A SYSTEMATIC REVIEW • It arises from the requirement of researchers to summarise all existing information about some phenomenon in a thorough and unbiased manner. • The purpose can be to draw more general conclusion about some phenomenon than is possible from individual studies, • Act as a prelude to further research activities • Researchers should ensure that a systematic review is necessary. • identify and review any existing systematic reviews of the phenomenon of interest against appropriate evaluation criteria.
  • 38. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 38 / 78 HOW TO CHECK EXISTING REVIEWS • What are the review’s objectives? • What sources were searched to identify primary studies? Were there any restrictions? Missing sources? • What were the inclusion/exclusion criteria and how were they applied? • What criteria were used to assess the quality of primary studies and how were they applied? • How were the data extracted from the primary studies? • How were the data synthesised? How were differences between studies investigated? How were the data combined? Was it reasonable to combine the studies? Do the conclusions flow from the evidence?
  • 39. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 39 / 78 1.2 THE DEVELOPMENT OF A REVIEW PROTOCOL • A review protocol specifies the methods that will be used to undertake a specific systematic review. • A pre-defined protocol is necessary to reduce the possibility researcher bias: fairness. • Without a protocol, it is possible that the selection of individual studies or the analysis may be driven by researcher expectations. • Critical point: in some disciplines (i.e. medicine) review protocols are usually submitted to peer review.
  • 40. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 40 / 78 COMPONENTS OF A PROTOCOL • Background: rationale for the survey. • Research questions that the review is intended answer. • The search strategy for primary studies: search terms and resources to be searched • Study selection (include/exclude) criteria and procedures. • Study quality assessment checklists and procedures. • Data extraction strategy to define how the information required from each primary study would be obtained • in case of required manipulation, the protocol should make it explicit • Synthesis of the extracted data • Project timetable
  • 41. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 41 / 78 RESEARCH QUESTIONS • A specific problem (P) is tackled using some specific constraints, methods and/or approaches (C) to develop a system, application or algorithm (S). • Goal: what existing solutions are available, how they compare, what the strength of the evidence is and what implications these solutions have. • RQ1: What are the existing solutions to P? • RQ2: How does the different solutions found by addressing RQ1 compare to each other with respect to C? • RQ3: What is the strength of the evidence in support of the different solutions? • RQ4: What implications will these findings have when creating S?
  • 43. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 43 / 78 MAP 1 Preamble to the two seminars 2 Why Systematic Literature Reviews? 3 Systematic Literature Reviews 4 The entire process 5 Phase 1 - Planning 6 Phase 2 - Conducting 7 Phase 3 - Reporting 8 An example 9 Conclusions and comments
  • 44. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 44 / 78 THE STAGES OF THE CONDUCTING PHASE • The stages associated with Conducting the review are: 1 2.1 Identification of research 2 2.2 Selection of primary studies 3 2.3 Study quality assessment 4 2.4 Data extraction & monitoring 5 2.5 Data synthesis.
  • 45. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 45 / 78 2.1 IDENTIFICATION OF RESEARCH • The aim of a systematic review is to find as many primary studies relating to the research question as possible using an unbiased search strategy. • Determine and follow a search strategy (in collaboration with librarian) • 1. Preliminary searches aimed at both identifying existing systematic reviews and assessing the volume of potentially relevant studies. • 2. Trial searchers using various combinations of search terms derived from the research question (from many sources) • 3. Reviews of research results • 4. Consultations with experts in the field
  • 46. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 46 / 78 SEARCH TERMS • Break down the question into individual facets • Draw up a list of synonyms, abbreviations, and alternative spellings. • Pick other terms can be obtained by considering subject headings used in journals and data bases. • Sophisticated search strings can then be constructed using Boolean AND’s and OR’s.
  • 47. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 47 / 78 INFORMATION SOURCES • ACM digital library, • IEEE Xplore, • ISI web of knowledge, • ScienceDirect, • CiteSeer, • SpringerLink • DBLP • Wiley Inter Science • . . . • For domain specific sources, experts in the domain will normally know which are the best.
  • 48. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 48 / 78 SOME ADDITIONAL FEATURES OF THIS STAGE • Avoid publication bias: • positive results are more likely to be published than negative results • Recent results: • scan repository of unpublished-preprint results (arXiv, Google Scholar, . . . ) • Document the search (also with the search query): • it must be possible to assess the thoroughness of the search • Useful to be helped with bibliographic management tools
  • 49. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 49 / 78 2.2 STUDY SELECTION • Choose the study selection criteria to identify those primary studies that provide direct evidence about the research questions. • Som generally used criteria for removal: duplicates, same study in different sources (conference ⇒ journal), after/before a certain date • Avoid language bias • Avoid “clique” bias (well-known institutions, labs, researchers)
  • 50. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 50 / 78 2.3 STUDY QUALITY ASSESSMENT • Choose the study selection criteria to identify those primary studies that provide direct evidence about the research questions. • Som generally used criteria for removal: duplicates, same study in different sources (conference ⇒ journal), after/before a certain date • The protocol should define exactly which inclusion and quality criteria are employed • Avoid language bias • Avoid “clique” bias (well-known institutions, labs, researchers) • Example of inclusion criteria: “The study’s main concern is P”, “The study focuses on C”,“ The study describes an S”
  • 51. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 51 / 78 THREE-STEPS SCREENING • Title and Abstract: apply the inclusion criteria • Introduction and conclusions: apply the inclusion criteria • Full text: apply the inclusion criteria and the quality criteria
  • 52. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 52 / 78 QUALITY CRITERIA • They are designed to answer “What is the strength of the evidence in support of the different solutions?” • Examples are: • QC 1: Is there is a clear statement of the aim of the research? • QC 2: Is the study is put into context of other studies and research? • QC 3: Are system or algorithmic design decisions justified? • QC 4: Is the test data set reproducible? • QC 5: Is the study algorithm reproducible? • QC 6: Is the experimental procedure throughly explained and reproducible? • QC 7: Is it clearly stated in the study which other algorithms the study’s algorithm(s) have been compared with? • QC 8: Are the performance metrics used in the study explained and justified? • QC 9: Are the test results thoroughly analysed? • QC 10: Does the test evidence support the findings presented? • Example: score each QC in 0, 1/2, 1 and add them up for global rank
  • 53. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 53 / 78 2.4 DATA EXTRACTION • Objective: to design data extraction forms to accurately record the information researchers obtain from the primary studies • It will define a set of numerical values that should be extracted for each study (e.g. number of subjects, performance parameters, . . . ) • To ensure consistency among multiple researcher, some papers should be reviewed by all researchers so that inter-researcher consistency can be assessed.
  • 54. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 54 / 78 2.5 DATA SYNTHESIS • Data synthesis involves collating and summarising the results of the included primary studies. • Synthesis can be descriptive (non-quantitative) • It is sometimes possible to complement a descriptive synthesis with a quantitative summary • It is important to identify whether results from studies are consistent one with another (i.e. homogeneous) or inconsistent (e.g. heterogeneous) in a tabulated/graphical form
  • 56. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 56 / 78 MAP 1 Preamble to the two seminars 2 Why Systematic Literature Reviews? 3 Systematic Literature Reviews 4 The entire process 5 Phase 1 - Planning 6 Phase 2 - Conducting 7 Phase 3 - Reporting 8 An example 9 Conclusions and comments
  • 57. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 57 / 78 STRUCTURE OF A REPORT OF SLR - 1 • Title • The title should be short but informative. It should be based on the question being asked. In journal papers, it should indicate that the study is a systematic review. • Abstract • Context/Background: the importance of the research questions addressed • Objectives: The questions addressed • Methods: Data Sources, Study selection, Quality Assessment and Data extraction • Results: Main finding • Conclusions: Implications for practice and future research
  • 58. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 58 / 78 STRUCTURE OF A REPORT OF SLR - 2 • Background: Justification of the need for the review. Summary of previous reviews • Review questions: Each review question should be specified • Review Methods: • Data sources and search strategy • Study selection • Study quality assessment • Data extraction • Data synthesis • Included and excluded studies • Results: Findings (description of primary studies and results of any quantitative summaries)
  • 59. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 59 / 78 STRUCTURE OF A REPORT OF SLR - 3 • Discussion: • Principal findings • Strengths and Weaknesses • Meaning of findings (applicability/generalisability) • Conclusions: recommendations (practical implementations) and future research • Results: Findings (description of primary studies and results of any quantitative summaries) • Acknowledgments • Conflict of Interest • References • Appendices
  • 61. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 61 / 78 MAP 1 Preamble to the two seminars 2 Why Systematic Literature Reviews? 3 Systematic Literature Reviews 4 The entire process 5 Phase 1 - Planning 6 Phase 2 - Conducting 7 Phase 3 - Reporting 8 An example 9 Conclusions and comments
  • 62. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 62 / 78 A PAPER AS AN EXAMPLE • Artem Voronkov, Leonardo Horn Iwaya, Leonardo A. Martucci, and Stefan Lindskog. 2017. “Systematic Literature Review on Usability of Firewall Configuration”. ACM Comput. Surv. 50, 6, Article 87 (December 2017) 35 pages. • A nice survey on Firewall usability • Published on one of the major sources for Surveys
  • 63. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 63 / 78 THE ABSTRACT
  • 64. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 64 / 78 THE ABSTRACT WITH ANNOTATIONS
  • 65. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 65 / 78 SOURCES
  • 66. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 66 / 78 RESEARCH QUESTIONS
  • 67. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 67 / 78 INCLUSION CRITERIA
  • 68. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 68 / 78 INDEX TERMS: TWO QUERIES
  • 69. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 69 / 78 THE RESULTING QUERIES
  • 70. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 70 / 78 QUALITY CRITERIA
  • 72. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 72 / 78 MAP 1 Preamble to the two seminars 2 Why Systematic Literature Reviews? 3 Systematic Literature Reviews 4 The entire process 5 Phase 1 - Planning 6 Phase 2 - Conducting 7 Phase 3 - Reporting 8 An example 9 Conclusions and comments
  • 73. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 73 / 78 COMPARISON OF SLR VS LITERATURE SURVEY • A SLR defines a review protocol that specifies the (1) research question being addressed and (2) the methods that will be used to perform the review • A SLR defines the search strategy to help readers to determine its thoroughness and fairness (and also repeatability, up to a certain extent) • searches on digital libraries cannot be replicated • A SLR specifies quality criteria to evaluate each primary study.
  • 74. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 74 / 78 PERSONAL CONSIDERATIONS • Important to follow a process when documenting literature surveys • Secondary sources are important: citations and publication venues are available out there! • Useful tools and instruments can facilitate the effort • If not publishable, SLRs make, at least, a great chapter of a PhD Thesis: thoroughness, fairness and replicability
  • 76. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 76 / 78 READING MATERIAL AND CREDITS • Material and references available at www.isislab.it under “Research” • Slides are available on Slideshare under Open licence Common Creative - Attribution 4.0 International (CC BY 4.0)
  • 77. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 77 / 78 QUESTIONS? COMMENTS? ANYTHING?