Research Methods in Computer Science and Software Engineering
0 likes118 views
The document discusses research methods in computer science, emphasizing the differences between empirical inquiry and traditional scientific approaches. It highlights the importance of case studies, validity threats, and generalization challenges, advocating for careful selection of phenomena and contexts for research. Additionally, it outlines strategies for writing research papers and navigating the review process to increase the chances of acceptance.
![Research Methods
Spectrum of cases
12
created for explanation
• foo, bar examples
• simple model;
illustrates differences
accepted teaching vehicle
• “textbook example”
• simple but illustrates
relevant issues
real-life example
• industrial system,
open-source system
• context is difficult to grasp
benchmark
• approved by community
• known context
• “planted” issues
Toy-example
Exemplar
Case
Benchmark
Susan Elliott Sim, Steve Easterbrook, and Richard C. Holt. Using
Benchmarking to Advance Research: A Challenge to Software
Engineering, Proceedings of the Twenty-fifth International
Conference on Software Engineering, Portland, Oregon, pp.
74-83, 3-10 May, 2003.
Martin S. Feather , Stephen Fickas ,
Anthony Finkelstein , Axel Van
Lamsweerde, Requirements and
Specification Exemplars, Automated
Software Engineering, v.4 n.4,
p.419-438, October 1997
Runeson, P. and Höst, M. 2009.
Guidelines for conducting and reporting
case study research in software
engineering. Empirical Softw. Eng. 14,
2 (Apr. 2009), 131-164.
competition (tool oriented)
• approved by community
• comparing
Community
case
Mining Software Repositories Challenge.
[Yearly workshop where research tools compete
against one another on a common predefined
case.]](https://image.slidesharecdn.com/researchmethodsredux2024-240930131759-9e1b93ff/85/Research-Methods-in-Computer-Science-and-Software-Engineering-12-320.jpg)
![Research Methods
Case study — definition
A case study is an empirical inquiry that investigates a contemporary phenomenon within its
real-life context, especially when the boundaries between the phenomenon and context are
not clearly evident
[Robert K. Yin. Case Study Research: Design and Methods; p. 13]
• empirical inquiry: yes, it is empirical research
• contemporary: (close to) real-time observations
+ incl. interviews
• boundaries between the phenomenon and context not clear
+ as opposed to “experiment”
14
Treatment Outcome
Phenomenon
Context
Experiment Case Study](https://image.slidesharecdn.com/researchmethodsredux2024-240930131759-9e1b93ff/85/Research-Methods-in-Computer-Science-and-Software-Engineering-14-320.jpg)
![Research Methods
Misunderstanding 2: Generalization
One cannot generalize on the basis of an individual case; therefore the case study cannot
contribute to scientific development.
[Bent Flyvbjerg, "Five Misunderstandings About Case Study Research."]
• Understanding
+ The power of examples
+ Formal generalization is overvalued
- dominant research views of physics and medicine
• Counterexamples
+ one black swan falsifies “all swans are white”
- case studies generate deep understanding; what appears to be white often turns
out to be black
• sampling logic vs. replication logic
+ sampling logic: operational enumeration of entire universe
- use statistics: generalize from “randomly selected” observations
+ replication logic: careful selection of boundary values
- use logic reasoning: presence of absence of property has effect
16](https://image.slidesharecdn.com/researchmethodsredux2024-240930131759-9e1b93ff/85/Research-Methods-in-Computer-Science-and-Software-Engineering-16-320.jpg)
![Research Methods
Example: Threat to Instrument Validity
22
"HIS WEEK A dolphin's
demise
SCIENTIFIC PUBLISHING
A Scientist's Nightmare: Software
Problem Leads to Five Retractions
Until recently, Geoffrey Chang's career was on
a trajectory most young scientists only dream
about. In 1999, at the age of 28, the protein
crystallographer landed a faculty position at
the prestigious Scripps Research Institute in
San Diego, California. The next year, in a cer
emony at the White House, Chang received a
Presidential Early Career Award
for Scientists and Engineers, the
country's highest honor for young
researchers. His lab generated a
stream of high-profile papers
detailing the molecular structures
of important proteins embedded in
cell membranes.
Then the dream turned into a
nightmare. In September, Swiss
researchers published a paper in
Nature that cast serious doubt on a
protein structure Chang's group
had described in a 2001 Science
paper. When he investigated,
Chang was horrified to discover
that a homemade data-analysis pro
gram had flipped two columns of
data, inverting the electron-density
map from which his team had
derived the final protein structure.
Unfortunately, his group had used
the program to analyze data for
other proteins. As a result, on page 1875,
Chang and his colleagues retract three Science
papers and report that two papers in other jour
nals also contain erroneous structures.
"I've been devastated," Chang says. "I hope
people will understand that it was a mistake,
2001 Science paper, which described the struc
ture of a protein called MsbA, isolated from the
bacterium Escherichia coli. MsbA belongs to a
huge and ancient family of molecules that use
energy from adenosine triphosphate to trans
port molecules across cell membranes. These
so-called ABC transporters perform many
essential biological duties and are of great clin
ical interest because of their roles in drug resist
ance. Some pump antibiotics out of bacterial
cells, for example; others clear chemotherapy
drugs from cancer cells. Chang's MsbA struc
ture was the first molecular portrait of an entire
Sciences and a 2005 Science paper, described
EmrE, a different type of transporter protein.
Crystallizing and obtaining structures of
five membrane proteins in just over 5 years
was an incredible feat, says Chang's former
postdoc adviser Douglas Rees of the Califor
nia Institute of Technology in Pasadena. Such
proteins are a challenge for crystallographers
because they are large, unwieldy, and notori
ously difficult to coax into the crystals
needed for x-ray crystallography. Rees says
determination was at the root of Chang's suc
cess: "He has an incredible drive and work
ethic. He really pushed the field in the sense
of getting things to crystallize that
no one else had been able to do."
Chang's data are good, Rees says,
but the faulty software threw
everything off.
Ironically, another former post
doc in Rees's lab, Kaspar Locher,
exposed the mistake. In the 14 Sep
tember issue of Nature, Locher,
now at the Swiss Federal Institute
of Technology in Zurich, described
the structure of an ABC transporter
called Sav 1866 from Staphylococcus
aureus. The structure was dramati
cally?and unexpectedly?differ
ent from that of MsbA. After
pulling up Sav 1866 and Chang's
MsbA from S. typhimurium on a
computer screen, Locher says he
realized in minutes that the MsbA
structure was inverted. Interpreting
the "hand" of a molecule is always
a challenge for crystallographers,
Locher notes, and many mistakes can lead to
an incorrect mirror-image structure. Getting
the wrong hand is "in the category of monu
mental blunders," Locher says.
On reading the Nature paper, Chang
quickly traced the mix-up back to the analysis
Flipping fiasco. The structures of MsbA (purple) and Savl866 (green) overlap
little Heft) until MsbA is inverted {right).
http://www.jstor.org/stable/20035062
[…] in a ceremony at the White House, Chang received a Presidential Early Career
Award for Scientists and Engineers, the country’s highest honor for young
researchers. His lab generated a stream of high-pro
fi
le papers detailing the
molecular structures of important proteins embedded in cell membranes.
[…] Swiss researchers published a paper in Nature that cast serious doubt on a
protein structure Chang’s group had described in a 2001 Science paper.
[…] Chang was horri
fi
ed to discover that a homemade data-analysis program had
fl
ipped two columns of data, inverting the electron-density map from which his team
had derived the
fi
nal protein structure.
Dr. Chang had to withdraw
five high
profile widely cited papers](https://image.slidesharecdn.com/researchmethodsredux2024-240930131759-9e1b93ff/85/Research-Methods-in-Computer-Science-and-Software-Engineering-22-320.jpg)
![Research Methods
Rebuttal
41
Author Response Period
ICSE 2022 will offer a three day author response period. In this period the
authors will have the opportunity to inspect the reviews, and to answer
specific questions raised by the program committee. This period is scheduled
after all reviews have been completed, and serves to inform the subsequent
decision making process. Authors will be able to see the full reviews, including
the reviewer scores as part of the author response process.
ESEC/FSE 2022
[…] Authors will have an opportunity to respond to reviews
during a rebuttal period.](https://image.slidesharecdn.com/researchmethodsredux2024-240930131759-9e1b93ff/85/Research-Methods-in-Computer-Science-and-Software-Engineering-41-320.jpg)
Research Methods in Computer Science and Software Engineering
- 1. Research Methods in Computer Science Prof. Serge Demeyer GRASCOMP Seminar — September 2024
- 3. Research Methods Helicopter View 3 (Ph.D.) Research How to perform research? (and get “empirical” results) How to write research? (and get papers accepted) How many of you have done / will do a case-study?
- 4. Research Methods Computer Science 4 All science is either physics or stamp collecting (E. Rutherford) We study artefacts produced by humans Computer science is no more about computers than astronomy is about telescopes. (E. Dijkstra) Computer science Informatics Computer engineering Software Engineering
- 5. Research Methods Science vs. Engineering 5 Science Engineering Physics Chemistry Biology Mathematics Electro- Mechanical Engineering Civil Engineering Chemistry and Materials Electronics Geography ??? Computer Science ??? Software Engineering ???
- 6. Research Methods Interdisciplinary Nature 6 Science Engineering Economics Sociology Computer Science Psychology “Hard” Sciences “Soft” Sciences Action Research
- 7. The Oak Forest Robert Zünd - 1882
- 8. Research Methods The Allegory of the Cave (a.k.a. Plato’s Cave) 8 © calebmarcelo — https://www.deviantart.com/
- 9. Research Methods Dominant view on Research Methods Physics (“The” Scientific method) • form hypothesis about a phenomenon • design experiment • collect data • compare data to hypothesis • accept or reject hypothesis • … publish (in Nature) • get someone else to repeat experiment (replication) Medicine (Double-blind treatment) • form hypothesis about a treatment • select experimental and control groups that are comparable except for the treatment • collect data • commit statistics on the data • treatment difference (statistically significant) 9 Cannot answer the “big” questions … in timely fashion • smoking is unhealthy • climate change • darwin theory vs. intelligent design • … • agile methods
- 11. Research Methods Case studies 11 1. Feasibility study Proof-of-Concept; often by applying on a “CASE” 2. Pilot, Demonstrator Demonstrated on a simple yet representative “CASE” 3. Comparative study Score criteria check-list; often by applying on a “CASE” 6. Formal Model often explained using a “CASE” 7. Simulation: test prognoses with real observations obtained via a “CASE” 4. Observational Study Observing a series of “CASES” 5. Literature survey “CASES” = selected papers case studies are widely used in computer science “studying a case” vs. “doing a case study”
- 12. Research Methods Spectrum of cases 12 created for explanation • foo, bar examples • simple model; illustrates differences accepted teaching vehicle • “textbook example” • simple but illustrates relevant issues real-life example • industrial system, open-source system • context is difficult to grasp benchmark • approved by community • known context • “planted” issues Toy-example Exemplar Case Benchmark Susan Elliott Sim, Steve Easterbrook, and Richard C. Holt. Using Benchmarking to Advance Research: A Challenge to Software Engineering, Proceedings of the Twenty-fifth International Conference on Software Engineering, Portland, Oregon, pp. 74-83, 3-10 May, 2003. Martin S. Feather , Stephen Fickas , Anthony Finkelstein , Axel Van Lamsweerde, Requirements and Specification Exemplars, Automated Software Engineering, v.4 n.4, p.419-438, October 1997 Runeson, P. and Höst, M. 2009. Guidelines for conducting and reporting case study research in software engineering. Empirical Softw. Eng. 14, 2 (Apr. 2009), 131-164. competition (tool oriented) • approved by community • comparing Community case Mining Software Repositories Challenge. [Yearly workshop where research tools compete against one another on a common predefined case.]
- 13. Research Methods Case Study Research 13 Sources • Robert K. Yin. Case Study Research: Design and Methods. 3rd Edition. SAGE Publications. California, 2009. • Bent Flyvbjerg, "Five Misunderstandings About Case Study Research." Qualitative Inquiry, vol. 12, no. 2, April 2006, pp. 219-245. • Runeson, P. and Höst, M. 2009. Guidelines for conducting and reporting case study research in software engineering. Empirical Softw. Eng. 14, 2 (Apr. 2009), 131-164. Studying a Case vs. Performing a Case Study • Proposition • Unit of Analysis • Threats to Validity
- 14. Research Methods Case study — definition A case study is an empirical inquiry that investigates a contemporary phenomenon within its real-life context, especially when the boundaries between the phenomenon and context are not clearly evident [Robert K. Yin. Case Study Research: Design and Methods; p. 13] • empirical inquiry: yes, it is empirical research • contemporary: (close to) real-time observations + incl. interviews • boundaries between the phenomenon and context not clear + as opposed to “experiment” 14 Treatment Outcome Phenomenon Context Experiment Case Study
- 15. Research Methods Case Study — Counter evidence 15 Phenomenon Context • many more variables than data points • multiple sources of evidence; triangulation • theoretical propositions guide data collection (try to confirm or refute propositions with well-selected cases) Case studies also look for counter evidence
- 16. Research Methods Misunderstanding 2: Generalization One cannot generalize on the basis of an individual case; therefore the case study cannot contribute to scientific development. [Bent Flyvbjerg, "Five Misunderstandings About Case Study Research."] • Understanding + The power of examples + Formal generalization is overvalued - dominant research views of physics and medicine • Counterexamples + one black swan falsifies “all swans are white” - case studies generate deep understanding; what appears to be white often turns out to be black • sampling logic vs. replication logic + sampling logic: operational enumeration of entire universe - use statistics: generalize from “randomly selected” observations + replication logic: careful selection of boundary values - use logic reasoning: presence of absence of property has effect 16
- 17. Research Methods Sampling Logic vs. Replication Logic 17 Random selection generalize for entire population Selection of (boundary) value understand differences • propositions • units of analysis Boundary value
- 18. Research Methods Proposition (a.k.a. Purpose) 18 Boundary value Where to expect boundaries? Thorough preparation is necessary! You need an explicit theory. Exploratory Confirmatory Exploratory case studies are used as initial investigations of some phenomena to derive new hypotheses and build theories.(*) Confirmatory case studies are used to test existing theories. The latter are especially important for refuting theories: a detailed case study of a real situation in which a theory fails may be more convincing than failed experiments in the lab.(*) (*) Steve Easterbrook, Janice Singer, Margaret-Anne Storey, and Daniela Damian. Selecting empirical methods for software engineering research. In Forrest Shull, Janice Singer, and Dag I. K. Sjoberg, editors, Guide to Advanced Empirical Software Engineering, pages 285—311. Springer London, 2008.
- 19. Research Methods Units of Analysis What phenomena to analyze • depends on research questions • affects data collection & interpretation • affects generalizability Possibilities • individual developer • a team • a decision • a process • a programming language • a tool Design in advance • avoid “easy” units of analysis + cases restricted to Java because parser - Is the language really an issue for your research question? + report size of the system (KLOC, # Classes, # Bug reports) - Is team composition not more important? 19 Example: Clone Detection, Bug Prediction • the tool/algorithm + Does it work? • the individual developer + How/why does he produce bugs/clones? • about the culture/process in the team + How does the team prevent bugs/clones? + How successful is this prevention? • about the programming language + How vulnerable is the programming language towards clones / bugs? (COBOL vs. AspectJ)
- 20. Research Methods Threats to validity (Case Studies) • Source: Runeson, P. and Höst, M. 2009. Guidelines for conducting and reporting case study research in software engineering. 1. Construct validity • Do the operational measures reflect what the researcher had in mind? 2. Internal validity • Are there any other factors that may affect the results? > Critical when investigating causality! 3. External validity • To what extent can the findings be generalized? > Precise research question & units of analysis required 4. Reliability • To what extent is the data and the analysis dependent on the researcher (the instruments, …) Other categories have been proposed as well • credibility, transferability, dependability, confirmability 20
- 21. Research Methods Threats to validity = Risk Management No experimental design can be “perfect” … but you can limit the chance of deriving false conclusions • manage the risk of false conclusions as much as possible + likelihood + impact • state clearly what and how you alleviated/mitigated the risk + construct validity - precise metric definitions - GQM paradigm + internal & external validity - report the context consciously + Reliability - bugs in tools: testing, usage of well-known libraries, … - classification: develop guidelines & others repeat classification - search for evidence (mailing archives, bug reports, …): have an explicit search procedure 21
- 22. Research Methods Example: Threat to Instrument Validity 22 "HIS WEEK A dolphin's demise SCIENTIFIC PUBLISHING A Scientist's Nightmare: Software Problem Leads to Five Retractions Until recently, Geoffrey Chang's career was on a trajectory most young scientists only dream about. In 1999, at the age of 28, the protein crystallographer landed a faculty position at the prestigious Scripps Research Institute in San Diego, California. The next year, in a cer emony at the White House, Chang received a Presidential Early Career Award for Scientists and Engineers, the country's highest honor for young researchers. His lab generated a stream of high-profile papers detailing the molecular structures of important proteins embedded in cell membranes. Then the dream turned into a nightmare. In September, Swiss researchers published a paper in Nature that cast serious doubt on a protein structure Chang's group had described in a 2001 Science paper. When he investigated, Chang was horrified to discover that a homemade data-analysis pro gram had flipped two columns of data, inverting the electron-density map from which his team had derived the final protein structure. Unfortunately, his group had used the program to analyze data for other proteins. As a result, on page 1875, Chang and his colleagues retract three Science papers and report that two papers in other jour nals also contain erroneous structures. "I've been devastated," Chang says. "I hope people will understand that it was a mistake, 2001 Science paper, which described the struc ture of a protein called MsbA, isolated from the bacterium Escherichia coli. MsbA belongs to a huge and ancient family of molecules that use energy from adenosine triphosphate to trans port molecules across cell membranes. These so-called ABC transporters perform many essential biological duties and are of great clin ical interest because of their roles in drug resist ance. Some pump antibiotics out of bacterial cells, for example; others clear chemotherapy drugs from cancer cells. Chang's MsbA struc ture was the first molecular portrait of an entire Sciences and a 2005 Science paper, described EmrE, a different type of transporter protein. Crystallizing and obtaining structures of five membrane proteins in just over 5 years was an incredible feat, says Chang's former postdoc adviser Douglas Rees of the Califor nia Institute of Technology in Pasadena. Such proteins are a challenge for crystallographers because they are large, unwieldy, and notori ously difficult to coax into the crystals needed for x-ray crystallography. Rees says determination was at the root of Chang's suc cess: "He has an incredible drive and work ethic. He really pushed the field in the sense of getting things to crystallize that no one else had been able to do." Chang's data are good, Rees says, but the faulty software threw everything off. Ironically, another former post doc in Rees's lab, Kaspar Locher, exposed the mistake. In the 14 Sep tember issue of Nature, Locher, now at the Swiss Federal Institute of Technology in Zurich, described the structure of an ABC transporter called Sav 1866 from Staphylococcus aureus. The structure was dramati cally?and unexpectedly?differ ent from that of MsbA. After pulling up Sav 1866 and Chang's MsbA from S. typhimurium on a computer screen, Locher says he realized in minutes that the MsbA structure was inverted. Interpreting the "hand" of a molecule is always a challenge for crystallographers, Locher notes, and many mistakes can lead to an incorrect mirror-image structure. Getting the wrong hand is "in the category of monu mental blunders," Locher says. On reading the Nature paper, Chang quickly traced the mix-up back to the analysis Flipping fiasco. The structures of MsbA (purple) and Savl866 (green) overlap little Heft) until MsbA is inverted {right). http://www.jstor.org/stable/20035062 […] in a ceremony at the White House, Chang received a Presidential Early Career Award for Scientists and Engineers, the country’s highest honor for young researchers. His lab generated a stream of high-pro fi le papers detailing the molecular structures of important proteins embedded in cell membranes. […] Swiss researchers published a paper in Nature that cast serious doubt on a protein structure Chang’s group had described in a 2001 Science paper. […] Chang was horri fi ed to discover that a homemade data-analysis program had fl ipped two columns of data, inverting the electron-density map from which his team had derived the fi nal protein structure. Dr. Chang had to withdraw five high profile widely cited papers
- 23. Research Methods Replication 23 Results show that MSR authors use in general publicly available data sources, mainly from free software repositories, but that the amount of publicly available processed datasets is very low. © 7th IEEE Working Conference on Mining Software Repositories (MSR 2010), 2010, pp. 171-180, doi: 10.1109/MSR.2010.5463348.
- 24. Research Methods Data Management Plan 24
- 25. Research Methods Helicopter View 25 (Ph.D.) Research How to perform research? (and get “empirical” results) How to write research? (and get papers accepted) How many of you have submitted a paper recently? • Was the review helpful? • Was there a rebuttal phase? • How did you write the abstract?
- 26. Research Methods The Reviewer • volunteer + don’t waste his/her time • curious + catch his/her interest • constructive + supervises other Ph.D. • influential + wants to support “valuable” papers • anonymous + avoid tampering … unfortunately … • busy + read’s on train, bus, air-plane, … 26
- 27. Research Methods Review Process Steps 27 source: CyberChair (http://www.CyberChair.org) Bidding for Abstracts abstracts + key-words = “first date” with your reviewer Identify the Champion your reviewer needs arguments to support your paper
- 28. Research Methods Providing Keywords 28 ■Automated reasoning techniques ■Component-based systems ■Computer-supported cooperative work ■Con fi guration management ■Domain modelling and meta-modelling ■Empirical software engineering ■Human-computer interaction ■Knowledge acquisition and management ■Maintenance and evolution ■Model-based software development ■Model-driven engineering and model transformation ■Modeling language semantics ■Open systems development ■Product line architectures ■Program understanding ■Program synthesis ■Program transformation ■Re-engineering ■Requirements engineering ■Speci fi cation languages ■Software architecture and design ■Software visualization ■Testing, veri fi cation, and validation ■Tutoring, help, and documentation systems As many as possible? vs. As few as possible?
- 29. Research Methods Writing Abstracts Descriptive Abstract • outlines the topics covered in a piece of writing + reader can decide whether to read entire document • ≈ table of contents in paragraph form. Informative Abstract • provides detail about the substance of a piece of writing + readers remember key findings + reviewers find the claims • ≈ claim and supporting evidence in paragraph form 29 ≠ executive summary (abstracts use the same level of technical language)
- 30. Research Methods 4-line abstract guideline • source: Kent Beck “How to Get a Paper Accepted at OOPSLA” + https://ansymore.uantwerpen.be/system/files/uploads/courses/thesis_master/ BeckAbstract.html + https://plg.uwaterloo.ca/~migod/research/beckOOPSLA.html • 1) states the problem + WHO is suffering the problem? + Connect with your target audience • 2) why the problem is a problem + WHY is it a problem? + Cost / Art rather than a science / … • 3) startling sentence + WHAT is the claimed solution? + the one thing to say that will catch interest … and that you will actually demonstrate in the paper > must be falsifiable • 4) the implication of my startling sentence + WHERE can we use this solution? + implications for society, community, other researchers, … 30
- 31. Research Methods Identify The Champion (1/2) • source: Oscar Nierstrasz, “Identify the Champion,” in Pattern Languages of Program Design 4 • Make Champions Explicit + A: Good paper. I will champion it at the PC meeting. + B: OK paper, but I will not champion it. + C:Weak paper, though I will not fight strongly against it. + D:Serious problems. I will argue to reject this paper. - “The most important thing for a reviewer to decide is whether he or she thinks that the paper is worth defending at the PC meeting, not whether it is a great paper or not.” • Make Experts Explicit + X: I am an expert in the subject area of this paper. + Y: I am knowledgeable in the area, though not an expert. + Z: My evaluation is that of an informed outsider. > detect inexpert champion — expert fence-sitter These scores are *not* revealed to the authors 31
- 32. Research Methods Identify The Champion (2/2) • Identify the Conflicts (classify according to extreme reviews) + AA, AB: All reviews are positive, at least one champion. + AC: Likely accept; at least one champion, and no strong detractor. + AD: This is a serious conflict, and will certainly lead to debate. + BC: Borderline papers, no strong advocate nor a detractor. + BD: Likely to be rejected. + CC, CD, DD: Almost certain rejects. • inexpert champion + If all champions are Y (or Z) + If all reviews are Y or Z > solicit extra review • expert fence-sitters + Experts tend to be more critical > B or even C ratings by X may turn out to be champions (remember: PC members want to influence the research) 32
- 33. Research Methods Example: Easychair 33 • Clear accept at top • Clear reject at the bottom (not shown) • middle area: to discuss
- 34. Research Methods Make it Easy for your Champion • Select appropriate keywords + Why are you in the scope of the conference/journal/…? • Test the abstract + Start early with the abstract + Ask for early (external) feedback • Visible claims + Abstract + intro + conclusion have have visible claim(s) + Ask early feedback to summarize what reviewers think the claim is • Clear validation + Champion is then able to defend it against detractors • Write to the Program Committee + Target a PC member + Have a clear picture of your champion 34
- 35. Research Methods Shadow PC / Junior PC 35 Allows future PC members to learn first-hand about the peer-review process and gain experience as a reviewer and learn from the senior researchers on how to write a good review. The Shadow PC will provide reviews on a subset of submissions to the technical track of the conference (The authors will opt-in for their paper to be reviewed by the Shadow PC).
- 36. Research Methods Single Blind Reviewing 36 Author is Known Reviewers are Anonymous
- 37. Research Methods Double Blind Reviewing 37 Author is Anonymous Reviewers are Anonymous
- 38. Research Methods Triple Blind Reviewing 38 Reviewers are Anonymous (Also to one another) Author is Anonymous
- 41. Research Methods Rebuttal 41 Author Response Period ICSE 2022 will offer a three day author response period. In this period the authors will have the opportunity to inspect the reviews, and to answer specific questions raised by the program committee. This period is scheduled after all reviews have been completed, and serves to inform the subsequent decision making process. Authors will be able to see the full reviews, including the reviewer scores as part of the author response process. ESEC/FSE 2022 […] Authors will have an opportunity to respond to reviews during a rebuttal period.
- 42. Research Methods Good Advice 42 https://andreas-zeller.info/2012/10/01/patterns-for-writing-good-rebuttals.html • Understand the decision process • Identify the undecided • Identify the champion • Arm the champion • Identify the detractors • Answer the questions • Write for the PC chair • Write for the committee • Convince • Choose comments wisely • Organize your rebuttal • No tricks • Thank the reviewers • Don’t expect too much
- 43. Research Methods Target Audience 43 Target Audience Experts in sub-domain (in-crowd) Broader Audience (informed outsider) = arguing the problem and inviting others to contribute = preaching to the quire • Conferences: ICSE, ESEC/FSE • Journals: TSE, TOSEM • magazines: IEEE Software, IEEE Computer, Communications of the ACM
- 44. Research Methods Role of “Related Work” 44 Related Work Problem Statement (beginning of paper) Problem Context (end of paper) Other researchers do complimentary work crisp problem statement (difficult to write) Other researchers define the research agenda high entry barrier (for experts only)
- 45. Research Methods Advice on writing Style: Toward Clarity and Grace Joseph M. Williams, Gregory G. Colomb • guidelines + refactoring rules • Give a man a fish and you feed him for a day. Teach a man to fish and you feed him for a lifetime. 45
- 46. Research Methods Slide Deck - Full Tutorial 46 (Ph.D.) Research How to perform research? (and get “empirical” results) How to write research? (and get papers accepted) https://win.uantwerpen.be/~sdemey/Tutorial_ResearchMethods/