chapter 6.docx

93
CHAPTER 6
TESTBED UTILIZATION ANALYSIS- FEEDBACK BASED
RANKING SYSTEM
6.1 TESTBED FEEDBACK BASED RANKING SYSTEM
The whole purpose of the testbed is achieved at its maximum utilization. In
order to achieve this and to offer the best performance based service to a user, it is
proposed to design a feedback based ranking algorithm. The parameters considered are
sustainability/availability, usability, reliability and performance.
6.2 BACKGROUND
The primary goals of a good testbed includes research productivity, ease of
management, minimal learning curve, user friendly graphical user interface,
customizable, time shared, utilization by maximum users, scalable, support variety of
experimentation scenarios, space optimized and provide almost realistic results [120]
[121] [122]. This research proposal has achieved all of the listed parameters which have
been satisfactorily proved in earlier chapters. This chapter attempts to prove the point
on utilization and the variety of users.
Earlier works have developed testbeds, but they have not been explored in terms
of the type of users, most used services or combination of services. The testbed
performance can be analyzed based on the utilization factor. In other words number of
users getting successful access per minute. This can be determined based on number of
users being served in a given period of time. Also the expected services if not available
can be added or scaled, measuring the need and demand in a given span of time.
Moreover, if one could predict or determine the pattern of type of users vs. services, the

94
most demanded services or group of services; it would be huge feedback closing the
loop, leading to better utilization and performance enhancement.
This chapter attempts to explore providing insights about how the above
mentioned factors are achieved, thereby enhancing the utilization, performance,
reliability and usability of the proposed testbed.
6.3 PROPOSED ALGORITHM FOR FEEDBACK BASED RANKING
As proposed, a feedback based ranking algorithm is developed to allocate the
best services to the users. Score ranges in between High(1), Medium(0.5), Low(0.25).
Initially all platform are assigned with High Score(1). After every usage of the platform,
user gives feedback. Previous score of the platform gets replaced with the new value
from recent user feedback.
6.3.1 Algorithm for Feedback BasedRanking
An algorithm is developed in the proposed research work as to arrive feedback
based ranking for the services under offering. The services available are looked up
based on dynamic service discovery. If availability is greater than the demand, the
services are ready to be offered. All the service platforms are given score of high (1).
The services are shuffled based on their feedback scores assigned based on their
performance. The highest performing service is offered as service to the users.

95
In case of tie, it is resolved using Decision tree based ranking. The user provides
the feedback score based on usage satisfaction of the testbed. The parameters adopted
here are Sustainability/Availability, Reliability/Performance, and Usability. These
values are inserted as input to Decision Tree as shown in Figure 6.1. The Decision Tree
provides the best available platform to the next user based on the score card.
Figure 6.1 Allocation decision tree
Figure 6.2a shows the state machine for the proposed feedback based testbed
service allocation. Let ti and tj refers to time between failures and restore respectively.
S denotes at least one successful resource available for service and F indicates failure
with degrading or nil available resource to offer. N denotes the number of platforms
comprising of sensors and actuators offered for service.

96
Figure 6.2a State machine for testbed service
Figure 6.2b State transition graph for testbed service
Figure 6.2b explains the transition logic between the states based on feedback
obtained based on resource availability and operational conditions. There are 5 states
formulated with 3 indicating states of success and 2 for states of failure. The failure
model can be catagorized as 1) occurred, detected, recoverable 2)occurred, latent
detect, recoverable 3) occurred, uncertain detect, uncertain recovery. The point 1 and
2 to probability of becoming operational is visibly high as against point 3. Point 3 may

97
lead to all platform failure in its worst case scenario. The proposed system handles point
1 and 2 to offer demanded services to users using the feedback based ranking
mechanism by classifying between scores of 1, 0.5 and 0.25. 1, 0.5 and 0.25 denotes
high, medium and low based on its performance/reliability. The initial configuration to
services in this proposed research work is 1 as against 0.5, which is observed in most
existing literature. The ideal condition is to assure that all the services will be fully
functional to start with, later on degrade based on various factors like wear and tear,
malfunction, natural disasters and many more.
The testbed services are offered to various users and the usage pattern of the
services, probable cluster services etc., are studied based on 500 users comprising of
academicians, industrialists, data analysts, researchers and basic user.
6.3.2 Services vs. User Feedback Matrix
The feedback from 500 users comprising of academicians(Aci),
industrialists(Ii), data analysts(Di), researcher(Ri) and novice user(Ni) is considered for
evaluation. Based on the feedback collected, Table 6.1 depicts the services vs. user’s
feedback table for representational purpose.
Table 6.1 Services vs Users Feedback matrix
Based on log report, the job completion rate, signal strength and
request/response rate it was observed that the testbed was reliable (score ~ 1) even when
the user feedback scores were moderate. The utilization score is arrived based on the

98
following rule: When scores are low, there may be 2 possibilities. The issue may be on
the provider side or issue on the receiver/user side. Hence, job completion, network and
request/response time log, is considered to decide accordingly. If there was any failure
or malfunctioning or any network issue on the system side and job was not completed
or incomplete or faulty, the system was graded 0.25. If there was network issue and job
was completed with the delay, the system was graded justifiably reliable (0.5). A
confusion matrix is constructed as shown in Table 6.2. The number of users was 500.
Table 6.2 Confusion Matrix
N = 500
Prediction Class
User score Low User score High
Actually
Low
40 05 45
Actually
High
65 390 455
105 395
True positive: 390, True negative: 40, False positive: 5, False negative: 65. This
was performed to predict how much time the testbed utilization (reliability) was
detected low when the testbed was performing fairly high. Based on the observations,
the results of needed parameters are calculated as follows:
Accuracy= TP+TN/Total=86%
Error rate=1-accuracy=14%
True positive rate=TP/Total positive=86%
False positive rate/Sensitivity/Recall=FP/Total negative=11%
Specificity: 1-FPR=TN/Total negative=89%
Precision=TP/total user score positive=98%
Prevalence=actual high/total=91%
Also to prove that the proposed testbed is reliable or utilized well, joint
probability function of the discrete random variables R (Reliability) and P′
(Performance) is utilized. P(R = ri, P′
= Pi
′
) denotes probability that R takes ri and P

99
P i i
takes pi
′
, it is the probability of intersection of events, namely reliability low and
performance low will be P(R = rl, P′
= pl
′
). P(ri, pi
′
) is the probability mass function,
where i= l, m, h. Table 6.3 shows the joint probability function of the discrete random
variables reliability(R) and performance(P′).
Table 6.3 Joint probability function of the discrete random variables R
(Reliability) and P′ (Performance)
P′
R
Low Medium High ∑aij=1 to 3
Low a11 a12 a13 Sl
Medium a21 a22 a23 Sm
High a31 a32 a33 Sh
∑aij=1 to 3 Tl Tm Th ∑Si=∑Ti
By rule, ∑Si = ∑Ti, where i = l, m, h
Marginal probability function of Reliability R
PR(ri) = P(R = ri)
= P(R = ri, P′
= Pl
′
)+ P(R = ri, P′
= Pm
′
)+ P(R = ri ,P′
= Ph
′
)=> rio (1)
R = ri, where i = l, m, h
The set (ri, pio) is the marginal distribution of reliability.
Similarly, to find the marginal distribution of performance P′, considering the marginal
probability function of Performance P′
P ′(p′) = P(P′ = p ′)
= P(R = rl, P′
= Pi
′
)+ P(R = rm, P′
= Pi
′
)+ P(R = rh, P′
= Pi
′
)=> pio (2)
P′
= pi, where i = l, m, h

100
The set (rio, p′i) is the marginal distribution of performance.
Having found the marginal distribution of R and P′, the conditional probability
distribution
F(P′/R) = f(R, P′)/f(R) and F(R/P′) = f(R, P′)/f(P′) (3)
The conditional distribution of reliability when performance is low is given as
F(R/P′
) = f(R, P′
)/f(P′
= low) where f(P′
= low) = Tl (4)
The conditional distribution of reliability when performance is moderate is given as
F(R/P′
) = f(R, P′
)/f(P′
= medium) where f(P′
= medium) = Tm (5)
Thus, the conditional distribution of reliability when performance is low is given as
follows:
P(R = l/P′
= l) = a11/Tl (6)
P(R = m/P′
= l) = a21/Tl (7)
P(R = h/P′
= l) = a31/Tl (8)
Similarly, the conditional distribution of reliability when performance is medium is
given as follows:
P(R = l/P′
= m) = a12/Tm (9)
P(R = m/P′
= m) = a22/Tm (10)
P(R = h/P′
= m) = a32/Tm (11)
As the data set acquired was voluminous, manual calculation would be
infeasible. Thus the data set was run in statistical tool and has proved that the proposed
testbed in most times have recorded performance moderate/high even at times when the
scores were low/moderate. The results are discussed in detail in subsequent sections.

101
6.4 PRINCIPLE COMPONENT ANALYSIS/ FACTOR ANALYSIS ON
TESTBED DATA
The average performance of each service is shown in Table 6.4, which services
were most demanded is shown in Table 6.5, which combination of sensors is used the
most and also which set of users have mostly used the testbed services is analyzed. 78
services listed earlier are considered for this proposed testbed framework. This analysis
can greatly help to decide on the scalability needs and new deployments. This also gives
an insight, to some extent the probable research trend and domain. Principal Component
Analysis (Factor analysis) is the approach deployed in this research.
6.4.1. Average Performance of Each Service
The developed services are offered as service to the users. 78 services were
made available for varied users for a span of 30 days duration. This experiment was
conducted to understand two factors namely 1. How well the services are performing
(utilization)? 2. Verify the claim of condition distribution. Table 6.4 reveals acceptable
scores of 0.5 and above, clearly indicating that the proposed testbed with offered
services are operating satisfactorily with performance and reliability well assured.
Similar results were recorded for other services as well.
Table 6.4 Sample set of performance measure of services
S1 S2 S3 S4 S5 S6 S7 S8 S9 S10
0.541244 0.573455 0.564429 0.663641 0.542526 0.63916 0.577314 0.659782 0.626282 0.658494
S1, S2, S3, S4, S5, S6, S7, S8, S9 and S10 refers to temperature, humidity, gas
service raw, gas service leakage, color, getRed, getGreen, getBlue, rainfall intensity,
rainfall raw respectively.
6.4.2 Most Demanded Services
The services mostly demanded were calculated using frequency distribution
analysis. The Table 6.5 is representational set of the results. The services mostly
demanded were used to improve scalability and future deployments. This is also used

102
to determine the score apart from user scores in the dynamic resource allocation process
based on user needs.
Table 6.5 Sample set of most demanded services
S1, S2, S3, S4, S5, S6, S7, S8, S9, S10 refers to services as temperature,
humidity, gas service raw, gas service leakage, color, getRed, getGreen, getBlue,
rainfall intensity, rainfall raw respectively. One could observe S1, S2, S3, S4, S5, S9,
S16, S20, S34, S36, S60, S63 are the set of most demanded services and the services
map to temperature, humidity, gas service raw, gas service leakage, color, rainfall
intensity, lcd_display, fan_rotate service, arduino1 service, Pi1 service, arduino2
service and arduino3 service respectively. The result shows very clearly that usage span
across variety of services ranging from sensors till platforms. The research study is also
interested to know the type of users who have utilized these services, the pattern of
services used which in turn would provide insight about the utilization pattern and the
need for scalability. The result is based on the experimental set up offered for access to
users for 30 days period. This is also indicative of the fact that the services were well
utilized.
6.5 RESULTS AND DISCUSSION
6.5.1 Average Performance Measurement of Each Service
The proposed testbed offers 78 services and it is observed from Figure 6.3 the
usage pattern of the offered services. The service usage pattern is arrived based an

103
average performance rating of all the users who used the testbed for a month. The same
experiment was repeated for a month with varied users. It can be clearly inferred from
Figure 6.3 that the utilization (successful access) rate of each service varies between
0.25, 0.5 and 1. 1 indicates service provided successful result as expected. 0.5 indicates
service provided result but with a delay. 0.25 indicates incomplete response for reasons
not limited to failure or malfunction or network issues etc., It shows the testbed score
of 0.25 is at the maximum of 21%, 0.5 ranges between 54% to 76% and 1 range between
45% and 93%.
Figure 6.3 Service usage pattern and the utilization (successful access) rate in %
6.5.2 Empirical Results of Most Demanded Services
With reference to the service usage pattern, the classification is attempted based
on most demanded services by varied users. The users are classified as novice,
academician, industrialist, data analyst and researcher. Figure 6.4 illustrates the
individual service usage of the testbed. The usage percentage of the testbed as a whole
is arrived based on mean of the usage values. This shows 36% of novice users, 23% of
industrial users, 38% of academicians, 47% of data analyst, 58% researchers have used
95
0.5 0.25 1
50
45
40
75
35
30
55 25
20
15
35
10
5
15 0
Services
Utilization
rate
%
s1
s4
s7
s10
s13
s16
s19
s22
s25
s28
s31
s34
s37
s40
s43
s46
s49
s52
s55
s58
s61
s64
s67
s70
s73
s76

104
the proposed testbed services. This not only proves the suitability of the proposed
testbed among varied users, but also the improvement of usage of services by data
analyst and novice users, which is not the case with existing testbeds [38].
Figure 6.4 User usage rate of the proposed testbed
Si: Services
Figure 6.5a Novice user usage pattern
It is very vivid from the graph in Figure 6.5a the most frequent services used by
normal users or in other words any person who has zero knowledge on the technology
and just interested to use the service in a crud sense. These are kind of users who are
only interested to explore the data as service mostly except for few exceptions.
According to the expectation of this research, the results have shown the services
mostly used are s1 to s10, s37 to s46. The services s1 to s10 and s37 to s46 are data
based services.

105
90%
60%
30%
0%
Services
Figure 6.5b Academic user usage pattern
It is clear from the graph in Figure 6.5b that almost all the services are being
exploited. As job profile of academicians might demand use of the testbed service to
teach or research or test or as a raw data source. The results are convincingly mapping
the expectation.
Usage (% )
Figure 6.5c Industrial user usage pattern
Figure 6.5c shows the services used by industrialists. The typical understanding
of an industrial user was he or she may be interested in more of testing of algorithms or
business logic or protocols basically. So in this regard the test results clearly reveal the
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Usage (%)
Linear (Usage (%))
y = -0.0009x + 0.7465
R² = 0.0636
s1 s5 s9 s13s17s21s25s29s33s37s41s45s49s53s57s61s65s69s73s77
Services
Testbed
usage
%
Testbed
usage
%
s1
s3
s10
s12
s14
s16
s18
s20
s22
s24
s26
s28
s30
s32
s34
s36
s48
s50
s52
s54
s56
s58
s60
s62
s64
s66
s68
s70
s72
s74
s76
s78

106
service sets s1, s3, s10, s12, s16, s18, s20, s22, s24, s26, s28, s30, s32, s34, s36, s48,
s50, s52, s54, s56, s58, s60, s64, s66, s70, s72, s74, s76, s77 corresponding to
temperature, gas service raw, rainfall service raw, buzzer off, lcd display, lcd lock, fan
rotate, fan lock, fan release, gsm call, gsm release, rgb led red, rgb led lock, arduino 1,
Pi 1, buzzer off, fan off, fan rotate, gsm sms, lcd clear, rgb led green, arduino 4, buzzer
on, fan off, gsm call, lcd display, rgb led red, rgb led green are the most used services
among the offered set. This well fits as expected that most used services are platforms,
actuators compared to sensor based services.
s1
s8
s41
s2 s3 s4 s5 s6 s7
s9 s10 s37 s38 s39 s40
s42 s43 s44 s45 s46
94%
92%
90%
88%
86%
84%
82%
80%
78%
0 5 10 15 20 25
Number of services
Figure 6.5d Data analyst usage pattern
Data analyst are one who is merely interested on the data and mostly ignorant
on the infrastructural set up involved behind the data generation. Most of the existing
testbeds demand atleast a minimal skill set to operate on the testbed. One example is
FIT-IoT were the access procedure is complex, makes it really tough for non-IoT or
other field individuals to reap the needed benefit. The proposed system by providing
downloadable platform independent API makes it convinient just extract the needed
detail. Also a bulk data download option is provided too, to make it even more simpler
for them access the needed data set. The data analyst can download the data in data
s3
s1 s4 s10
s2 s38
s41
s40
s46,92%
s9 s45
s37 s39
s43
s42
s5
s44
s6
s7
s8
Testbed
usage
%

107
format of his choice XML,CSV or JSON. As anticipated the results in Figure 6.5d have
shown the services ~ s1 to s10, ~ s37 to s46 are the ones mostly preferred.
91%
78%
65%
52%
39%
26%
13%
0%
Services
Figure 6.6e Researcherusage pattern
It is candid from the graph in Figure 6.5e that almost all the services are being
exploited. As any researcher working on IoT would be interested to analyze the data
set, observe or infer or decide based on the results, to test actuation as his or her output
model or to test the algorithm or protocol developed for its successful implementation.
The results have shown a broad coverage across almost all the services offered.
6.6 CONCLUSION
Thus, the above results conclude that the proposed testbed framework is reliable
even when the performance was moderate or low based on the score metrics. The sensor
performance and other measurements were analyzed based on the usage pattern of over
500 users. 78 services have been considered and have successfully identified the most
demanded services and also the top users of the testbed.
Usage (%) Linear (Usage (%))
y = -0.0013x + 0.8264
R² = 0.158
Testbed
Usage
%
s1
s3
s5
s7
s9
s11
s13
s15
s17
s19
s21
s23
s25
s27
s29
s31
s33
s35
s37
s39
s41
s43
s45
s47
s49
s51
s53
s55
s57
s59
s61
s63
s65
s67
s69
s71
s73
s75
s77

chapter 6.docx

More Related Content

Similar to chapter 6.docx (20)

More from Sami Siddiqui (20)

Recently uploaded (20)

chapter 6.docx