Do Search-Based Approaches Improve the Design of Self-Adaptive Systems ? A Controlled Experiment

Do Search-Based Approaches Improve
the Design of Self-Adaptive Systems ?
A Controlled Experiment
Sandro S. Andrade and Raimundo J. de A. Macêdo
Distributed Systems Laboratory (LaSiD)
Department of Computer Science
Federal University of Bahia
{sandros, macedo}@ufba.br
CBSoft2014 – Congresso Brasileiro de Software: Teoria e Prática – Sandro S. Andrade – sandros@ufba.br - @andradesandro

Agenda
● Context & motivation
● Our search-based architectural design approach
● The quasi-experiment
● Contributions of this paper
● Final remarks

Context & Motivation
● The need for self-adaptive systems (SAS):
– Dynamic and uncertain operating environments
– Stringent requirements for scalability, dependability, energy-efficiency,
and performance
– Incomplete/partial requirements
– Problem/design space complexity approaching the limits of
human capability
– Rapidly varying workloads and data characteristics
– Some problems are inherently self-adaptive or self-organizing
(eg.: robots motion coordination, particles/swarm optimization,
etc)

● MAPE-K Reference Architecture [1]
[1] J. O. Kephart and D. M. Chess. Computer, vol. 36, no. 1, pp. 41–50, Jan. 2003.

● Current challenges:
– Intricate problem space
● How to effectively elicit the adaptation requirements ?
Goals:
● Type (regulation, tracking, disturbance rejection, optimization)
● Evolution (static, dynamic)
● Duration (persistent, temporary)
● Multiplicity (single, multiple)
● Dependency (independent, complementary, conflicting)
● Flexibility (rigid, constrained, unconstrained)
Change:
● Source (internal, external)
● Anticipation (foreseen, foreseeable, unforeseen)
● Frequency (rare, frequent)
● Type (functional, non-functional, technological)

– Large and complex solution space

● Type (parametric, structural)
● Mechanism (signaling, frameworks, profiling, etc)
● Uncertainty handling (redundancy, filtering, smoothing)

● Mechanism (symptoms databases, policies, queue
networks, game theory, etc)
● System/Environment representation (ad-hoc, models)

● Actuation law (intelligent agents, control theory, search-based, etc)
● Cardinality (single, utility functions, parent-front)
● Type (static, dynamic)

● Type (parametric, structural)
● Mechanism (function pointers, aspect weaving, metaobjects, middleware-based)
● Timeliness (best-effort, guaranteed)

● How to come up with an effective managing system architecture
for the adaptation requirements at hand ?
● How to judiciously evaluate all available design alternatives ?
● How to make well-informed decisions about quality attributes
trade-offs ?
– Consequences:
● Lack of organized design knowledge for routine use
● False intuition, design bias, and sub-optimal architectures

Our Search-Based Design
Approach – DuSE
● Research questions
To which extent may SAS design knowledge be systematically
represented for routine use ?
How to support well-informed decision making regarding
quality attribute trade-offs between alternative architectures for
SAS ?

Approach – DuSE
● We combine the use of ...
– Metamodeling and Domain-Specific Languages (DSL)
– Structured Architecture Design Spaces
– Multi-Objective Optimization
● … in order to …
– Enable a more disciplined and automated “handbook” of SAS
design
– Provide a solid basis for choosing between architectures which
exhibit different quality attributes

Approach – DuSE
Domain-Specific Supporting
Domain-Specific Design
Dimensions + Quality Metrics
Specified using the
DuSE language
Domain-Specific
Design space
UML Profile
Generic
Architecture
Optimization
Engine
Initial System Model
Annotated with
corresponding
UML Profile
Pareto-front of Candidate Architectures [2] Andrade and Macêdo. SEKE.
Boston, MA, USA: ACM, 2013.

Approach – DuSE
`
● Domain independent
● Metamodel dependent
● Design space navigation engine
● Optimization [3] Andrade and Macêdo. SASO. Philadelphia, PA, engine (NSGA-II)
USA: IEEE, 2013.

Approach – DuSE
[4] Andrade and Macêdo. SASO Demo Session. Philadelphia, PA, USA: IEEE, 2013.

Approach – SA:DuSE
● SA:DuSE
– A particular DuSE instance which captures the most prominent
design dimensions of SASs
● Evaluation dimensions:
– Is SAS design indeed a multi-objective problem ?
– To which extent the quality of Pareto-optimal architectures are
indeed observed in real prototypes ?
– Do search-based approaches improve the design of SAS ?
[5] Andrade and Macêdo. WoSiDA 2014. Florianópolis, SC, Brasil. 2014.

The Quasi-Experiment
● Analyze the design of SAS
● For the purpose of evaluating the search-based
design approach we propose and a design process
based on architecture styles catalogs
● With respect to the effectiveness and complexity of
resulting architectures, as well as the method's
potential for leveraging the acquisition of distilled
design knowledge by novice architects
● From the point of view of researchers
● In the context of graduate students endowing
systems with self-adaptation capabilities

● Quasi-experiment:
– Blocked subject-object study with a paired comparison design
● Subjects:
– 24 students of a graduate program in Distributed and
Ubiquitous Computing
● Experiment objects:
– Two UML models representing the managed system (web
server and MapReduce distributed architecture)
● Experiment tests:
– Design a managing system architecture for both objects
– Answer a questionnaire on quality attribute trade-offs in SAS

● Independent variable:
– Design method
● Search-based approach
● Style-based approach
● Dependent variables:
– Effectiveness of resulting SAS architectures
– Complexity of resulting SAS architectures
– The method's potential for leveraging the acquisition of distilled
SAS design knowledge by novice architects
● Experiment website:
– http://wiki.ifba.edu.br/tr-ce012014

● Experiment timetable

● Experiment tests

● Experiment
objects

● Measuring effectiveness
(Generational Distance)
[6] Deb and Kalyanmoy. John Wiley & Sons, Inc. NY, USA: 2001.

● Measuring complexity
(Component Point)
[7] Wijayasiriwardhane and Lai. Journal of Systems and Software. 2010.

● Measuring the acquisition of distilled design
knowledge:
– Questionnaire with 10 multiple choice questions
– Questions related to quality attributes trade-offs in the SAS
domain
– Questionnaire answered at the end of each experimentation day
– Normalized grades assigned to each student/day

● Hypotheses formulation
H0 - there is no difference in design effectiveness between
a feedback control loop design created using the style-based
approach (RA) and a feedback control loop design
created using the search-based approach (IA)
1
H0 - there is no difference in design complexity between a
feedback control loop design created using the RA and a
feedback control loop design created using the IA
2
H0 - there is no difference in the acquisition of distilled
design knowledge between a feedback control loop
design created using the RA and a feedback control loop
design created using the IA
3

● Descriptive statistics

● Assumptions of parametric tests:
– Data is taken from an interval or ratio scale
– Observations are independent
– Population variances are equal between groups
(homoscedasticity)
– Measured values are normally distributed
α=0.05

(homoscedasticity)
– Measured values are normally distributed Cannot reject
homoscedasticity
α=0.05

(homoscedasticity)
– Measured values are normally distributed Can reject normal
distribution
α=0.05

(homoscedasticity)
– Measured values are normally distributed
Wilcoxon Signed-Rank Test
α=0.05 Paired t-test

● Results of the statistical tests
α=0.05

● Threats to validity
– Construct:
● Inadequate preoperational explication of constructs
● Hypothesis guessing
● Objects representativeness
– Internal:
● Maturation
● Instrumentation
– External:
● Students acting as subjects
– Conclusion:
● Experiment holders creating the experiment objects

Contributions of this paper
● To the best of our knowledge, SA:DuSE is the first
effort in applying search-based approaches in the
SAS domain
● To the best of our knowledge, this is the second
controlled experiment in the SAS domain
(the first one is [8])
● We got some empirical evidence about the benefit
of search-based approaches when designing SASs
● The potential for leveraging knowledge acquisition
still deserves further investigation
[8] Weyns, Iftikhar, and Soderlund. SEAMS. 2013.

Final Remarks
● A lot of research towards principled and systematic
design of self-adaptive systems
● Effectiveness of systematic design spaces in
capturing intricate solution spaces
– However, qualitative approaches may be used to address
another facets of architectural design
● Future work:
– SA:DuSE expansion (NSGA-III)
– More investigation on real prototypes
– From design spaces to design theories

Thanks ! Questions ?
Sandro S. Andrade and Raimundo J. de A. Macêdo
{sandros, macedo}@ufba.br

Do Search-Based Approaches Improve the Design of Self-Adaptive Systems ? A Controlled Experiment

More Related Content

Viewers also liked (20)

Similar to Do Search-Based Approaches Improve the Design of Self-Adaptive Systems ? A Controlled Experiment (20)

Recently uploaded (20)

Do Search-Based Approaches Improve the Design of Self-Adaptive Systems ? A Controlled Experiment