Iso9126 based software quality evaluation using choquet integral

International Journal of Software Engineering & Applications (IJSEA), Vol.6, No.1, January 2015
DOI : 10.5121/ijsea.2015.6110 111
ISO9126 BASED SOFTWARE QUALITY EVALUATION
USING CHOQUET INTEGRAL
Abdelkareem M. Alashqar1
, Ahmad Abo Elfetouh2
and Hazem M. El-Bakry3
1,2,3
Information Systems Dept., Faculty of Computer and Information Sciences, Mansoura
University, EGYPT
ABSTRACT
Evaluating software quality is an important and essential issue in the development process because it helps
to deliver a competitive software product. A decision of selecting the best software based on quality
attributes is a type of multi-criteria decision-making (MCDM) processes where interactions among criteria
should be considered. This paper presents and develops quantitative evaluations by considering
interactions among criteria in the MCDM problems. The aggregator methods such as Arithmetic Mean
(AM) and Weighted Arithmetic Mean (WAM) are introduced, described and compared to Choquet Integral
(CI) approach which is a type of fuzzy measure used as a new method for MCDM. The comparisons are
shown by evaluating and ranking software alternatives based on six main quality attributes as identified by
the ISO 9126-1 standard. The evaluation experiments depend on real data collected from case studies.
KEYWORDS
Software Quality, Fuzzy Measure, Choquet Integral, Criteria Interaction, Multi Criteria Decision Making,
Aggregation Function, ISO9126.
1. INTRODUCTION
Software quality is an important and essential issue especially in current rapid change in the
software development industry. The process of evaluating software quality offers various benefits
during the development process as it let developers to deliver high qualified software product.
Quality is a relative term which differs with viewpoints transcendental view, product view,
manufacturing view, user view and value-based View [17]. Several models have been proposed
like ISO/IEC 9126 Model [16], McCall model [20], Boehm model [6], FURPS model [15],
Dromey [9] model , and FURPS model [14], to evaluate generic software application. A
comparative study of software quality models is found in [1] [26] [31]. Quality models define
software product qualities as a hierarchy of factors, criteria and metrics. A quality factor
represents behavioral characteristics of the system; a quality criterion is an attribute of a quality
factor that is related to software production and design; while a quality metrics is a measure that
captures some aspect of a quality criterion [32]. Most of the works that focus on software quality
evaluation do not reference the aggregation methods used to calculate the values of the different
element of the model. Sometimes a simple weighted average is used to summaries the various
quality measurements into a single score as proposed in [3] and [23]. Over the past decade, fuzzy
decision support has emerged as a means of providing effective tools and techniques for solving
MCDM problems. The authors in [11] used the aggregation function, a fuzzy decision-support
technique, to support the MCDM process in a game theory context. An MCDM approach based
on ordered weighted averaging (OWA) operators was proposed in [8] which permits a

112
sophisticated aggregation of measured atomic quality values using linguistic criteria to express
human expert’s evaluations. Recent studies as in [21] and [32], also adopts MCDM approach
based on fuzzy measures to evaluate software quality. In this study, we introduce a relatively
fuzzy decision support technique based on an aggregation function named the Choquet Integral
(CI) into the MCDM problem of evaluating quality of e-learning websites. The evaluation depend
on the main quality attributes defined in ISO/IEC 9126 standard, and take into account the
interactions between these quality attributes. In addition to that, the evaluations utilize real
empirical data collected from previous studies.
The rest of this paper in organized as follows: Section 2 mentions related work. Section 3
explains the proposed method that emphasizes the aggregations methods especially CI. Section 4
presents the experimental results, and Section 5 provides the conclusion and future work.
2. RELATED WORK
Some research studies attempt to evaluate software quality using fuzzy multi criteria approach
without considering interaction between criteria as [7] [29] and [30]. The work in [7] provides a
method to estimate the software quality criteria using fuzzy multi criteria approach. The method
used to quantify software quality for generic applications. The authors in [29] proposed a MCDM
approach to software quality assessment using fuzzy measures to model software experts’
decision making processes and help them to predict/evaluate software quality. Their approach
helps to predict/evaluate software quality with consistently over 60% accuracy. While the authors
in [30] present a software quality prediction model based on a fuzzy neural network. The
proposed model is a hybrid model of Artificial Neural Network (ANN) and Fuzzy Logic (FL),
which exploits the advantages of ANN and FL while eliminating their limitations.
A limited number of attempts apply fuzzy measure techniques to the modeling of MCDM process
in evaluating software quality attributes considering the interaction between these quality
attributes. In an early study carried out by [32], presents a fuzzy integral approach as an
aggregation operator to produce the overall assessment of quality attributes for web-based
applications (WBA). Later work [22] proposes a methodology based on fuzzy measure using CI
for comparing different software solutions based on the software requirements specification
(SRS) to a common problem. Our work is closest to work done in [22] [32] which use CI with
fuzzy measure to quantify the interacting software quality parameters and finds the best software
among different software and also made analysis of effect of interaction among criteria in finding
best alternative. However, it is different in that it uses CI to evaluate specific type of software that
is e-learning websites from user perspective depending on empirical collected data from previous
studies. Furthermore, our evaluations are based on ISO/IEC 9126 quality model. Furthermore,
while [22] assigned different values to ξ (an interaction degree among criteria as proposed by
[27]) to show how it affects the evaluation results, we use real interaction values as stated in the
literature in our evaluation purposes.
3. RESEARCH METHOD
This section presents a description of an aggregation function that is widely used by researchers
in quantitative evaluations, followed by a description of fuzzy measure method suing CI as a new
approach for evaluation.

113
3.1 Aggregation Function
MCDM is used if there are a set of alternatives and each is evaluated against several criteria. For
example, if a user wants to select the best one of several developed websites, it needs to consider
the best usability, the best efficiency, and so on. The degree to which an alternative satisfies a
criterion corresponds to a utility value [28]. The scores must then be combined in some ways to
produce an overall rating for that alternative. Such a process is very similar to an aggregation
function or aggregation operator, which combines several inputs into a single representative
output. The averaging aggregation is one of the most widely used type of aggregation functions.
The term “average” is commonly employed in everyday language when referring to the arithmetic
mean (AM). The AM of n values is the function: AMሺ‫ݔ‬ሻ =
ଵ
௡
ሺ‫ݔ‬ଵ + ‫ݔ‬ଶ + ⋯ + ‫ݔ‬௡ሻ =
ଵ
௡
∑ ‫ݓ‬௜
௡
௜ୀଵ ‫ݔ‬௜
(1)
For example for a given input ‫ݔ‬ = ሺ0.5,0.2,0.7ሻ, the AMሺ‫ݔ‬ሻ = ሺ0.5 + 0.2 + 0.7ሻ/3 = 0.47.
In case some criteria are considered as more important than others, it is common to consider the
aggregation function to be additive and to take the form of a weighted arithmetic mean (WAM).
WAM is a linear function with respect to a positive valued weighting vector ‫ݓ‬ with ∑ ‫ݓ‬௜
௡
௜ୀଵ = 1:
ܹ‫ܯܣ‬௪ሺ‫ݔ‬ሻ = ‫ݓ‬ଵ‫ݔ‬ଵ + ‫ݓ‬ଶ‫ݔ‬ଶ + ⋯ + ‫ݓ‬௡‫ݔ‬௡ = ∑ ‫ݓ‬௜
௡
௜ୀଵ ‫ݔ‬௜ (2)
For example if a given input ‫ݔ‬ = ሺ0.5,0.2,0.7ሻ has corresponding weights ‫ݓ‬ = ሺ0.5,0.3,0.2ሻ, then
the ܹ‫ܯܣ‬௪ሺ‫ݔ‬ሻ = 0.5ሺ0.5ሻ + 0.2ሺ0.3ሻ + 0.7ሺ0.2ሻ = 0.25 + 0.06 + 0.14 = 0.45.
Although the most often used aggregation operators are the WAM [19], it is not an appropriate
method when there is an interaction among the criteria. Therefore, we choose CI aggregation
function as a type of fuzzy measure that takes into account the interaction among criteria [28].
3.2 Fuzzy Measures and Choquet Integral
A fuzzy measure (also called a capacity) ߤ on a set of input criteria ܺ = {‫ݔ‬ଵ, ‫ݔ‬ଶ, … ‫ݔ‬௡} is a set
function ߤ: 2௫
→ [0,1] satisfying [12][28]:
(i) ߤሺ߶ሻ = 0 and ߤሺܺሻ = 1 (boundary conditions).
(ii) ‫,ܣ‬ ‫ܤ‬ ⊆ ܺ ‫ܣ‬ ⊆ ‫ܤ‬ implies ߤሺ‫ܣ‬ሻ ≤ ߤሺ‫ܤ‬ሻ (monotonicity).
Furthermore a capacity ߤ on ܺ is said to be [12] :
• Additive if ߤሺ‫ܣ‬ ∪ ‫ܤ‬ሻ = ߤሺ‫ܣ‬ሻ + ߤሺ‫ܤ‬ሻ for all disjoint sunsets ‫,ܣ‬ ‫ ܤ‬ ⊆ ܺ, and
• Cardinality-based if, for any ‫ܤ‬ ⊆ ܺ, ߤሺ‫ܤ‬ሻ depends only on the cardinality of ‫.ܤ‬
Note that there is only one capacity on ܺ that is both additive and cardinality-based. It is called
the uniform capacity and denote it by ߤ∗
. It is easy to verify that ߤ∗
is given by
ߤ∗ሺ‫ܤ‬ሻ = ܾ
‫ݔ‬ൗ , ∀‫ܤ‬ ⊆ ܺ
In the framework of aggregation, for each subset of criteria ‫ܣ‬ ⊆ ܺ, the number ߤሺ‫ܣ‬ሻ can be
interpreted as the weight or the importance of ‫.ܣ‬ The monotonicity of ߤ means that the weight of

114
a subset of criteria cannot decrease when new criteria are added to it. In the case ܺ = {‫ݔ‬ଵ, ‫ݔ‬ଶ, ‫ݔ‬ଷ}
the representation of fuzzy measures is shown as:
ߤሺ{∅}ሻ
ߤሺ{1}ሻ
ߤሺ{2}ሻ
ߤሺ{3}ሻ
ߤሺ{1,2}ሻ
ߤሺ{1,3}ሻ
ߤሺ{2,3}ሻ
ߤሺ{1,2,3}ሻ
The discrete CI with respect to a fuzzy measure ߤ is given by:
‫ܫܥ‬ఓሺ‫ݔ‬ሻ = ∑ ‫ݔ‬ሺ௜ሻൣߤ൫൛݆ห‫ݔ‬௝ ≥ ‫ݔ‬ሺ௜ሻ}൯ − ߤ൫൛݆ห‫ݔ‬௝ ≥ ‫ݔ‬ሺ௜ାଵሻ}൯൧௡
௜ୀଵ (3)
Where ‫ݔ‬ሺଵሻ, ‫ݔ‬ሺଶሻ, … , ‫ݔ‬ሺ௡ሻ is a non-decreasing permutation of the input ‫,ݔ‬ and ‫ݔ‬ሺ௡ାଵሻ = ∞ by
convention. For example, given an input ‫ݔ‬ = ሺ0.5,0.2,0.7ሻ and the fuzzy measure values:
ߤሺ{∅}ሻ = 0
ߤሺ{1}ሻ = 0.5
ߤሺ{2}ሻ = 0.3
ߤሺ{3}ሻ = 0.2
ߤሺ{1,2}ሻ = 0.7
ߤሺ{1,3}ሻ = 0.6
ߤሺ{2,3}ሻ = 0.9
ߤሺ{1,2,3}ሻ = 1
To calculate the CI, the input ‫ݔ‬ is arranged in a non-decreasing ሺ0.2,0.5,0.7ሻ). Then, the result
obtained from the input ‫ݔ‬ is:
‫ܫܥ‬ఓሺ‫ݔ‬ሻ = 0.2[ߤሺ{1,2,3}ሻ − ߤሺ{1,3}ሻ] + 0.5[ߤሺ{1,3}ሻ − ߤሺ{3}ሻ] + 0.7[ߤሺ{3}ሻ]
= 0.2ሺ1 − 0.6ሻ + 0.5ሺ0.6 − 0.2ሻ + 0.7ሺ0.2ሻ
= 0.2ሺ0.4ሻ + 0.5ሺ0.4ሻ + 0.7ሺ0.2ሻ = 0.42
The result value falls between the maximum and the minimum input expected in the bounding
condition of averaging functions such as AM.
It should be noted we adopted the additive fuzzy measure in this paper. A fuzzy measure is called
additive if ߤሺ‫ܣ‬ ∪ ‫ܤ‬ሻ = ߤሺ‫ܣ‬ሻ + ߤሺ‫ܤ‬ሻ whenever ‫ܣ‬ ∩ ‫ܤ‬ = ∅, super additive if ሺ‫ܣ‬ ∪ ‫ܤ‬ሻ ≥ ߤሺ‫ܣ‬ሻ +
ߤሺ‫ܤ‬ሻ whenever ‫ܣ‬ ∩ ‫ܤ‬ = ∅ and sub additive if ሺ‫ܣ‬ ∪ ‫ܤ‬ሻ ≤ ߤሺ‫ܣ‬ሻ + ߤሺ‫ܤ‬ሻ whenever ‫ܣ‬ ∩ ‫ܤ‬ = ∅.
Fuzzy measures are rich and important family. Furthermore, when they are used with some of the
integrals such as CI, they can generalize some well-known aggregation operators such as the AM.
The CI corresponds to the WAM functions when it is defined by the additive fuzzy measures. We
also adopted a k-additive as a recently concept developed by [10] to reduce the complexity in
fuzzy measure. The interactions between criteria are only considered for subsets of ݇ elements or
less, which reduces the number of variables to define the fuzzy measure. It allows for a trade-off
between modeling ability and complexity. A decision maker can decide how complex a fuzzy

115
measure that he wishes to consider by selecting a k-additive value ሺ1 ≤ ݇ ≤ ݊ሻ. When ݇ = 1 the
CI becomes equivalent to WAM, and when ݇ = ݊ the fuzzy measures are said to be unrestricted.
The Shapley value is an important concept related to CI used to measure the overall importance
of each criterion alone in all coalitions. Let ߤ be a fuzzy measure and ܺ = {‫ݔ‬ଵ, ‫ݔ‬ଶ, … , ‫ݔ‬௡} be the
set of criteria. The Shapley index for every input ‫ݔ‬௜ ∈ ܺ is defined as:
ߔఓሺ݅ሻ = ∑
ሺ௡ି|஺|ିଵሻ!|஺|!
௡!
[ߤሺ‫ܣ‬ ∪ {‫ݔ‬௜}ሻ − ߤሺ‫ܣ‬ሻ]஺⊆௑/{௫೔} (4)
The Shapley value is the vector ߔሺ௩ሻ = ߔଵ, ߔଶ, … , ߔ௡. The index ߔ௜ can be interpreted as a kind
of average value of the contribution of criteria ‫ݔ‬௜ in all groups of criteria. It also represents a true
sharing of the total amount ߤሺܺሻ as it must satisfy the condition ∑ ߔఓሺ݅ሻ = 1௡
௜ୀଵ .
A powerful capability of fuzzy measure approach using CI is the consideration of criteria
interaction. The interaction indices are interpreted as the behaviors of criteria in groups or as a
measurement of the interaction among criteria in the decision-making process. If ܺ =
{‫ݔ‬ଵ, ‫ݔ‬ଶ, … , ‫ݔ‬௡} is the set of criteria, then the interaction index for every set ⊆ ܺ :
‫ܫ‬ఓሺ‫ܣ‬ሻ = ∑
ሺ௡ି|஻|ି|஺|ሻ!|஻|!
ሺ௡ି|஺|ାଵሻ!஻⊆௑/஺ ∑ ሺ−1ሻห஺
஼ൗ ห
ߤሺ‫ܥ⋃ܤ‬ሻ஼⊆஺ (5)
This measure can include all combinations of groups of criteria where ‫ܫ‬ఓሺ‫ܣ‬ሻ ∈ [−1,1]. The
interaction index ‫ܫ‬ఓሺ݆݅ሻ for each pair ‫ܣ‬ = {‫ݔ‬௜, ‫ݔ‬௝} of criteria is usually used due to its convenience
of interpretation. For a pair of criteria ‫ݔ‬௜ and ‫ݔ‬௝ , if they have a positive interaction (complement),
then ‫ܫ‬ఓሺ݆݅ሻ > 0. Similarly, if ‫ݔ‬௜ and ‫ݔ‬௝ have a negative interaction (correlation), then ‫ܫ‬ఓሺ݆݅ሻ < 0.
When ‫ݔ‬௜ and ‫ݔ‬௝ have little or no interaction (independence), ‫ܫ‬ఓሺ݆݅ሻ = 0. The ‫ܫ‬ఓሺ݆݅ሻ is more than
just the interaction between a pair of criteria themselves because each pair is considered in the
presence of all groups.
3.3 A Comparison between WAM and CI
In the WAM aggregation method seen so far, only a single weight is considered for each data
element. Besides, we cannot consider how to measure the importance of a set of sources. For
example, in the case of the multi-criteria decision making problem of website evaluation, we can
state that the efficiency criterion has an importance of 0.3, and that usability is more important,
and thus, its weight is 0.5. However, we have not considered the importance of efficiency and
usability when considered together. Fuzzy measures permit us to incorporate considerations not
included in the weights for the WAM operator. In particular, they can be used to express
redundancy, complementariness, and interactions among information sources or criteria.
Therefore, tools that use fuzzy measures such as CI to represent background knowledge permit
the consideration of sources that are not independent. We can see that the advantage of CI lies in
the fact that the fuzzy measure can account for the importance and the interaction between every
subset of criteria [28].
Lest us consider a decision maker of an organization has to evaluate the software products
according to their level in functionality (F), usability (U), and efficiency (E). Suppose that the
organization put more importance in functionality and usability than efficiency and the assigned
weights could be for example 0.4, 0.4, and 0.2 respectively. Applying WAM, Product A is better
than Product B and Product C, since it has highest weighted score as shown in Table 1 although it
was weak in characteristic E. However if the organization wants the product that is well versed

116
with all the three characteristics then Product C is better than other products as it has scored
consistent scores in all the three characteristics. This is due to much importance is given to
characteristics F and U, which are in sense synergic. So this problem can be solved using CI with
fuzzy measure as input. As F and U are synergic characteristics, the weights used for the pair
characteristics F and U should be less than the sum of weights of characteristics F and U, and
hence the sum of weights of characteristics F and U, ߤሺ‫,ܨ‬ ܷሻ = 0.5. As product equally good at F
and E characteristics must be favored so that the weights for the pair of F and E must be greater
than sum of individual weights of F and E, so ߤሺ‫,ܨ‬ ‫ܧ‬ሻ = ߤሺܷ, ‫ܧ‬ሻ = 0.9, where ߤሺ‫,ܨ‬ ܷ, ‫ܧ‬ሻ = 1.0.
So according to CI approach Product C is better than Product A and B as it has highest CI value
shown in Table 1. In contrast to WAM score Product A was better than B and C. Therefore it can
be concluded that fuzzy measure using CI is a better approach than WAM because it takes into
account interaction and importance of each criterion.
Table 1: Results of software products evaluation by WAM and CI
Product F U E WAM CI
Product A 9 8 5 7.8 6.9
Product B 4 6 8 6 6.3
Product C 7 7 8 7.2 7.9
We used the application of the Kappalab R package [13]. Kappalab package includes
implementation of several methods for fuzzy measures (called capacity) identification in Choquet
integral based multi-attribute utility theory (MAUT). These methods are based on least-squares
based approaches, maximum split approach, minimum variance approach and minimum distance
approaches. Most of the methods for capacity identification proposed in the literature can be
stated as optimization problems. However, they differ according to their objective function and
the preferential information they require as input [12]. We developed a tool using RStudio [24]
that utilize these methods. The main parameters values used as input to the kappalab R package
methods are the number of criteria and their weights, the alternative instances, the interaction
indices and the k-additive. There are six criteria which represent the ISO/IEC9126 based six
quality attributes. The instances of alternatives as well as the criteria weights are collected
depending on empirical data from two previous case studies. These studies provide evaluation of
e-learning websites where each case includes five instances. The criteria weights and the
interaction indices are identified depending on industry experience. The k-additive value
corresponds to the number of selected criteria.
4. EXPERIMENTS AND RESULTS
The data used in this paper were collected from two separate previous case studies that provide
evaluations for e-learning websites from the student perspective (see [23] and [3]). The
evaluations in the first one (which we called Case 1) were carried out in Thailand universities,
and evaluations presented in the later one (which we called Case 2) were carried out in Jordan
universities. Each study provides evaluations of five different categories of e-learning websites
using traditional averaging methods. However the names and categories are different in both
studies. The categories in Case 1 are: Educational and Social Science, Humanities, Agriculture,
Science and Technology, and Business Administration, where we call them A1, A2, A3, A4 and
A5 respectively. The categories in Case 2 are: Arts, Business, Educational and Social Sciences,
Engineering and Science where we call them A1, A2, A3, A4 and A5 respectively.

117
In this paper we use the data described so far in evaluating e-learning websites based on ISO9126
quality attributes using CI approach. The data are shown in Table 2 and Table 3. Where ‘F’, ‘R’,
‘U’, ‘E’, ‘M’, and ‘P’ stand for “Functionality”, “Reliability”, “Usability”, “Efficiency”,
“Maintainability”, and “Portability”, respectively.
Table 2: Case 1- Total quality attributes evaluations for 5 groups of subjects found in [23]
QA A1 A2 A3 A4 A5
F 60.46 49.44 57.66 54.13 53.98
R 82.76 80.09 74.60 76.11 83.27
U 68.86 63.26 64.41 56.64 62.72
E 37.72 40.13 44.08 36.01 42.88
M 36.03 17.75 19.96 38.16 19.54
P 40.08 34.91 37.49 39.95 36.24
AM 54.47 47.60 49.70 50.17 49.78
Table 2: Case 2- Total quality attributes evaluations for 5 groups of subjects found in [3]
QA A1 A2 A3 A4 A5
F 71. 67 76 04 74. 69 67. 81 72.19
R 55. 83 58. 33 56. 67 54. 17 57.92
U 72.19 67. 6 71. 46 67. 4 68. 96
E 63. 33 63. 33 63. 33 54. 17 61. 25
M 70.21 67. 5 73. 44 67. 71 68. 75
P 66.67 67 08 64. 17 58. 75 6 0. 83
AM 66.65 66.65 67. 29 61. 67 64. 98
Software quality is a relative term which differs with viewpoints. So different perspectives of
quality can be considered; such as user view, developer view, manager view, etc. For example,
from the system user’s point of view, usability of the system is the most important one; while
from the viewpoints of developers and maintainers, maintainability is the most important one. In
addition to that, the overall quality of software product can be expressed by a combination of
different views. However, it must be taken into account that no software product can satisfy all of
the stakeholders’ needs at the same time. In our work, the users’ viewpoint will be considered.
Given the fact that there are some trade-offs between software product capabilities and it is not
possible to satisfy all of the software requirements at the same time, it is essential to determine
the weights of all quality aspects in any software product. Because of the absence of relative
importance of quality attributes represented by weights in the two mentioned previous case
studies, the quality attributes weights were adopted from [5] as a result of industrial experience
and experts refinement. We depend on these results on evaluating the e-learning websites. The
weights are: 0.3, 0.19, 0.24, 0.18 0.05, 0.04 as user’s view weights for ‘‘Functionality”,
‘‘Reliability”, ”Usability”, ‘‘Efficiency”, ‘‘Maintainability”, and ‘‘Portability”, respectively .
The relationships between quality attributes exist. Software products quality attributes are among
the important subjects that require trade-offs especially when facing conflicts in quality
requirements or customers' desires and in order to conduct a correct and a successful tradeoff a
rigorous analysis might be required [2]. The interaction between quality attributes is defined form
[2] and [33] as depicted in Table 4 which shows 15 relationships for each pair of the 6 quality
attributes.

118
Table 4: Quality attributes Relationships as identified in [2] and [28]
QA F R U E M
R +
U + +
E 0 0 -
M + + + -
P 0 0 0 - +
Where positive relationship means there is a synergy between the two quality attributes; in other
words good values of one attribute result in a good value of the other. And the relationship means
there is a conflict between the two quality attributes; in other words, a good value of one attribute
result in a bad value of the other. The synergic relationship represented as complementary
relationship in kappalab R package while the conflict relationship represented as substitutive
relationship.
For calculating CI values we used the minimum distance (MD) approach because it helps in
finding, if it exits, the closest capacity to a capacity defined by the decision maker (DM) and
compatible with his/her initial preferences. This initial capacity is typically an additive capacity
representing the DM’s prior idea of what the aggregation function should be. In the absence of
clear requirements a very natural choice for fuzzy measure ߤ is the uniform capacity ߤ∗
. Also we
apply CI with less restricted fuzzy measures where the overall performance of CI increased as ݇
increased, and it achieved the best performance when ݇ is equal to the number of criteria where
the number of criteria in our study is 6. In addition to that the use of the CI with less restricted
fuzzy measures can model the MCDM process in a way that is closer to reality [18] .
Table 5: Quality Attributes Evaluations for the 5 groups of subjects of case 1 obtained from WAM and CI
QA A1 A2 A3 A4 A5
F 60.46 49.44 57.66 54.13 53.98
R 82.76 80.09 74.60 76.11 83.27
U 68.86 63.26 64.41 56.64 62.72
E 37.72 40.13 44.08 36.01 42.88
M 36.03 17.75 19.96 38.16 19.54
P 40.08 34.91 37.49 39.95 36.24
AM 54.47 47.60 49.70 50.17 49.78
WAM 60.58 54.74 57.36 54.28 57.21
CI 57.25 52.57 53.98 52.16 54.45

119
Table 6: Quality Attributes Evaluations for the 5 groups of subjects of Case 2 obtained from WAM and CI
QA A1 A2 A3 A4 A5
F 71. 67 76 04 74. 69 67. 81 72.19
R 55. 83 58. 33 56. 67 54. 17 57.92
U 72.19 67. 6 71. 46 67. 4 68. 96
E 63. 33 63. 33 63. 33 54. 17 61. 25
M 70.21 67. 5 73. 44 67. 71 68. 75
P 66.67 67 08 64. 17 58. 75 6 0. 83
AM 66.65 66.65 67. 29 61. 67 64. 98
WAM 67.01 67.58 67.96 62.30 66.11
CI 65.90 67.35 66.92 61.79 65.58
Studying the results in Table 5 for Case 1, obtained by considering the Shapley values and
considering synergic and conflicts element with CI, can let us report that the rank of websites as
follows A1>A5>A3>A2>A4, which means that A1 followed by A5 outperform the other
alternatives, while the rank using WAM is A1>A3>A5>A2>A4.
And studying the results in Table 6 for Case 2, obtained by considering the Shapley values and
considering synergic and conflicts element with CI, can let us report that the rank of websites as
follows A2>A3>A1>A5>A4, while the rank using WAM is A3>A2>A1>A5>A4.
It is important to mention here that we compared the evaluation results of fuzzy measure using CI
to WAM only, because these methods take into account the weighting of criteria, while AM does
not consider this.
5. CONCLUSION AND FUTURE WORK
Evaluating software quality is always an interest of researchers working to support users and
developers in decision making. Traditional methods as AM and WAM are unable to explore
MCDM process completely and accurately. In this paper we introduce a fuzzy measure technique
that uses the CI approach in evaluating quality attributes of e-learning websites based on ISO/IEC
9126 and depending on real data by analyzing the Shapley values and the interaction between
criteria. The results show more convenient evaluation for users and developers in decision
making. The calculation results conducted on two example case studies in Section 4 illustrate that
using CI approach with the consideration of synergic and conflict elements can essentially change
the alternatives ranking.
Extension to this work include taking into account evaluating sub-criteria of quality attributes and
considering different software product views. Carrying out the experiments using large amount of
empirical data is also needed to refine the results. Furthermore, it is needed to develop a tool that
facilitates applying experiments emphasizing the approaches provided by Kappalab R package.
REFERENCES
[1] Al-Badareen, A. B., Selamat, M. H., Jabar, M. A., Din, J., & Turaev, S. (2011). Software Quality
Models: A Comparative Study. In Software Engineering and Computer Systems (pp. 46-55). Springer
Berlin Heidelberg.
[2] Aldaajeh, S., Asghar, T., Khan, A. A., & ZakaUllah, M. (2012). Communing Different Views on
Quality Attributes Relationships' Nature. European Journal of Scientific Research, 68(1), 101-109.

120
[3] Baklizi, M., & Alghyaline, S. (2011, May). Evaluation of E-Learning websites in Jordan universities
based on ISO/IEC 9126 standard. In Communication Software and Networks (ICCSN), 2011 IEEE
3rd International Conference on (pp. 71-73). IEEE.
[4] Barbacci, M., Klein, M. H., Longstaff, T. A., & Weinstock, C. B. (1995). Quality Attributes (No.
CMU/SEI-95-TR-021). CARNEGIE-MELLON UNIV PITTSBURGH PA SOFTWARE
ENGINEERING INST.
[5] Behkamal, B., Kahani, M., & Akbari, M. K. (2009). Customizing ISO 9126 quality model for
evaluation of B2B applications. Information and software technology, 51(3), 599-609.
[6] Boehm, B. W., Brown, J. R., & Kaspar, H. (1978). Characteristics of software quality.
[7] Challa, J. S., Paul, A., Dada, Y., Nerella, V., Srivastava, P. R., & Singh, A. P. (2011). Integrated
Software Quality Evaluation: A Fuzzy Multi-Criteria Approach. JIPS, 7(3), 473-518.
[8] Davoli, P., Mazzoni, F., & Corradini, E. (2004). Quality assessment of cultural web sites with fuzzy
operators. Journal of Computer Information Systems, 46(1), 44.
[9] Dromey, R. G. (1995). A model for software product quality. Software Engineering, IEEE
Transactions on, 21(2), 146-162.
[10] Grabisch, M. (1997). k-order additive discrete fuzzy measures and their representation. Fuzzy sets and
systems, 92(2), 167-189.
[11] Grabisch, M., & Roubens, M. (2000). Application of the Choquet integral in multicriteria decision
making. Fuzzy measures and integrals, (40), 348-375.
[12] Grabisch, M., Kojadinovic, I., & Meyer, P. (2008). A review of methods for capacity identification in
Choquet integral based multi-attribute utility theory: Applications of the Kappalab R package.
European journal of operational research, 186(2), 766-785.
[13] Grabisch, M., Kojadinovic, I., & Meyer, P. (2012). kappalab: Non-additive measure and integral
manipulation functions. R package version 0.4-6, URL http://cran.r-
project.org/web/packages/kappalab/index.html.
[14] Grady, R. B. (1992). Practical software metrics for project management and process improvement.
Prentice-Hall, Inc..
[15] Grady, R. B. (1994). Successfully applying software metrics. Computer, 27(9), 18-25.
[16] ISO/IEC 9126-1, (2001), “Software engineering —Product quality — Part 1: Quality model”
[17] Kitchenham, B., & Pfleeger, S. L. (1996). Software quality: The elusive target. IEEE software, 13(1),
12-21.
[18] Li, G., Law, R., Vu, H. Q., & Rong, J. (2013). Discovering the hotel selection preferences of Hong
Kong inbound travelers using the Choquet Integral. Tourism Management, 36, 321-330.
[19] Marichal, J. L. (2000). An axiomatic approach of the discrete Choquet integral as a tool to aggregate
interacting criteria. Fuzzy Systems, IEEE Transactions on, 8(6), 800-807.
[20] McCall, J. A., Richards, P. K., & Walters, G. F. (1977). Factors in software quality. General Electric,
National Technical Information Service.
[21] Pasrija, V., Kumar, S., & Srivastava, P. R. (2012). Assessment of Software Quality: Choquet Integral
Approach. Procedia Technology, 6, 153-162.
[22] Pasrija, V., Kumar, S., & Srivastava, P. R. (2012). Assessment of Software Quality: Choquet Integral
Approach. Procedia Technology, 6, 153-162.
[23] Pruengkarn, R., Praneetpolgrang, P., & Srivihok, A. (2005, July). An evaluation model for e-learning
Websites in Thailand University. In Advanced Learning Technologies, 2005. ICALT 2005. Fifth
IEEE International Conference on (pp. 161-162). IEEE.
[24] RStudio, http://www.rstudio.com/, Accessed on 6/12/2015.
[25] Srivastava, P. R., Singh, A. P., & Vageesh, K. V. (2010). Assssment of Software Quality: A Fuzzy
Multi–Criteria Approach. Evolution of Computationand Optimization Algorithms in Software
EngineeringApplications and Techniques, IGI Global USA, 200-219.
[26] Suman, and Wadhwa M. (2014). A Comparative Study of Software Quality Models. (IJCSIT)
International Journal of Computer Science and Information Technologies, Vol. 5 (4) , 2014, 5634-
5638
[27] Takahagi, E. (2005). λ fuzzy measure identification methods using λ and weights. Senshu University,
Japan.
[28] Torra, V., & Narukawa, Y. (2007). Modeling decisions. Information Fusion and Aggregation
Operators.

121
[29] Wang, X., Ceberio, M., Virani, S., Garcia, A., & Cummins, J. (2013). A Hybrid Algorithm to Extract
Fuzzy Measures for Software Quality Assessment. Journal of Uncertain Systems, 7(3), 219-237.
[30] Yang, B., Yao, L., & Huang, H. Z. (2007, August). Early software quality prediction based on a fuzzy
neural network model. In Natural Computation, 2007. ICNC 2007. Third International Conference on
(Vol. 1, pp. 760-764). IEEE.
[31] Youness, B., Abdelaziz, M, Habib, B. and Hicham M. (2013). Comparative Study of Software
Quality Models. IJCSI International Journal of Computer Science Issues, Vol. 10, Issue 6, No 1,
November 2013 ISSN (Print): 1694-0814 | ISSN (Online): 1694-0784.
[32] Zulzalil, H., Ghani, A. A. A., Selamat, M. H., & Mahmod, R. (2011). Using fuzzy integral to evaluate
the web-based applications. Malaysian Software Engineering Interest Group (MSEIG). Malaysia.
[33] Zulzalil, H., Ghani, A. A., Selamat, M. H., & Mahmod, R. (2008). A case study to identify quality
attributes relationships for web-based applications. IJCSNS, 8(11), 215.

Iso9126 based software quality evaluation using choquet integral

More Related Content

What's hot (17)

Viewers also liked (7)

Similar to Iso9126 based software quality evaluation using choquet integral (20)

Recently uploaded (20)

Iso9126 based software quality evaluation using choquet integral