Empirical Software Engineering

Rahim Lotfi
AhmadReza Ghasemian

1

Table of Contents
Introduction









Need for empirical studies in the context of software engineering.
Goals and Benefits
Basic Concepts: Measurement, Collection, Analysis
Basic Concepts for Empirical Software Engineering
Empirical Strategies: survey, case study, controlled experiment
Process of an empirical study
Tools
State of the art

Empirical Software Engineering
•
•
•

•

•

Using other people’s experiences in management and development of softwares
Empirical, included in all areas of science
“I could have told you that from the start” or “That’s obvious.”
Software engineering needs to follow the model of other physical sciences and
develop an experimental paradigm for the field.
Judging empirical work by the standards of statistics texts would lead to
the suppression of many interesting results

ESE in the Literature
• “Empirical Software Engineering”
• “Experimentation in Software Engineering”
• “ESEM - Empirical Software Engineering and Measurement “

History of Empirical Software engineering

History of Empirical Software engineering

• Victor Basili(1996)
• Quotes
Like physics, medicine, manufacturing, and many other disciplines, software engineering
requires the same high level approach for evolving the knowledge of the discipline; the
cycle of model building, experimentation, and learning. We cannot rely solely on
observation followed by logical thought.

Key figure

http://academic.research.microsoft.com

Conferences (64) Empirical Software Engineering

Journals (29)


Table of Contents
Introduction
 Need for empirical studies in the context of software engineering.








Goals and Benefits
Goal Question Metric
Tools
State of the art

Need of Empirical Studies
Why should we conduct empirical studies?
Product evaluation, e.g., prototyping.
Process evaluation
1. Prototypes are not possible (simulation based on models).
2. A process is just a description until it is used by people
 Important for research: experimentation is mandatory in
other disciplines(e.g., medicine, physics, etc.)

Need of Empirical Studies
Empirical studies provides a systematic, disciplined,
quantifiable and controlled way of evaluating
human-based activities.
You will learn how to select, plan, conduct and analyse an empirical study.

Table of Contents
 Introduction

 Goals and Benefits







Tools
State of the art

Goals and Benefits
• The purpose of a study is

• The purpose of a study is
to explore ...
to find out what’s happening
to seek for new insights
to ask questions and to find answers
Measurement: usually qualitative

• to describe ...
portray accurate profile of situations, events, projects , technologies

•

•

Measurement: quantitative/qualitative
to explain ...
seek explanation of a situation/problem, usual in the form of causal
Relationships
Measurement: quantitative/qualitative
relationships, differences, changes

Empirical Studies in SE
• Empirical Studies is research to improve Software Engineering
Practice.
– Apply theoretical concepts in SE practice.
– Add experiences on their appliance to the SE ‘body of knowledge’
– Improve processes, methods and tools (SPPI approach).
– Verify theories and models.

Table of Contents
 Introduction
 Goals and Benefits

 Basic Concepts: Measurement, Collection, Analysis






Tools
State of the art

Some Basic Concepts
• Measurement
- is the process of capturing data which are connected to real-world attributes to
describe them.
- Why is measurement important?

• Data Collection
Collection of qualitative / quantitative data according to research questions

• Data Analysis
Analyzing the results according to the research questions ;
statistical tests

Measurement
• Quotes
- “You can’t manage what you can’t measure”, Tom DeMarco
– “What is not measurable make measurable”, Galileo Galilei

• Objectives
One objective of science is to find ways to measure attributes of entities
we are interested in.
– Measurement makes concepts more visible and thus more
understandable and controllable.

• Definition
Measurement is the process by which numbers or symbols are mapped
to attributes of entities in the real world in such a way as to describe
them according to clearly defined rules.

Table of Contents





Introduction
Goals and Benefits

 Basic Concepts for Empirical Software Engineering





Tools
State of the art

Quality Improvement Paradigm
Package &
store experience

Set
goals

Analyze
results

Execute
process

Project
learning

Characterize
& understand

Corporate
learning

Provide process
with feedback

Analyze
results

Choose
processes,
methods,
techniques,
and tools

The Experience Factory Organization
A Different Paradigm

Project Organization
Problem Solving

Experience Factory
Experience Packaging

Decomposition of a problem
into simpler ones

Unification of different solutions
and re-definition of the problem

Instantiation

Generalization, Formalization

Design/Implementation process

Analysis/Synthesis process

Validation and Verification

Experimentation

Product Delivery within
Schedule and Cost

Experience / Recommendations
Delivery to Project

Table of Contents










Introduction
Goals and Benefits
Tools
State of the art

Table of Contents








Introduction
Goals and Benefits
Goal Question Metric

 Tools
 State of the art

Table of Contents









Introduction
Goals and Benefits
Tools

 State of the art

State of the art
• Finding better ways to experiment and integrate the results of
the studies
• Propagate the empirical discipline
•

Build an empirical research engine for software engineering

•

Build test beds for experimentation and evolution of
processes

•

Build product models that allow us to make trade-off
decisions

•

Build decision support systems offering the best empirical
advice for selecting and tailoring the right processes for the
problem

• References
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•

V. Basili, G. Caldiera, D. Rombach: „The Goal Question Metric Approach“,2000.
B. Boehm, H.D. Rombach, M.V. Zelkowitz: „Foundations of Empirical
Software Engineering – The Legacy of Victor R. Basili“, Springer, 2005.
Freimut et al.: "Empirical Studies in Software Engineering", Tutorial, VISEK
Technical Report, 2002.
IESE Tutorials on Empirical Software Engineering.
A. Jedlitschka and D. Pfahl: "Reporting Guidelines for Controlled
Experiments in Software Engineering", ISESE, 2005.
B. Kitchenham: “Evidence-Based Software Engineering and Systematic
Literature Review”, Profes, 2006.
C. Wohlin, P. Runeson, M. Höst, M.C. Ohlsson, B. Regnell and A. Wesslen:
"Introduction to Experimentation in Software Engineering", Kluver, 2000.
M. V. Zelkowitz, D. R. Wallace: “Experimental Models for Validating
Technology”, IEEE Computer, 1997.
Site : http:// www.microstrategy.com
Site : http:// www.jamasoftware.com
Site : http://academic.research.microsoft.com

Acknowledgments
Bahman Zamani

Thank you for your attention

This research work has been supported by University of Isfahan

Empirical Software Engineering

More Related Content

What's hot (20)

Similar to Empirical Software Engineering (20)

Recently uploaded (20)

Empirical Software Engineering