KM System Evaluation using Four dimensional Metric Model, Database and RESTful Resources

International Journal on Web Service Computing (IJWSC), Vol.3, No.3, September 2012
DOI : 10.5121/ijwsc.2012.3302 17
KM System Evaluation using Four dimensional
Metric Model, Database and RESTful Resources
D.Venkata Subramanian1
, Angelina Geetha1
, K.M. Mehata1
,
K.Mohammed Hussain2
1
Department of Computer Science and Engineering
2
Department of Computer Applications
B.S. Abdur Rahman University, Chennai, India
ABSTRACT
A Knowledge Management (KM) System plays a crucial role in every industry as well as in Higher Learning
Institutions. A RESTful resource is anything that is addressable over the Web. The resources can be
accessed and transferred between clients and server. The resources can be accessed and transferred
between clients and servers. Based on our earlier research works, we have developed a comprehensive KM
System framework, evaluation method, mult-dimensional metric model and useful metrics which are helpful
to assess any given knowledge management system. In this proposed work, we first describe the actual
implementation steps for building the KM System metric database using the multi-dimensional metric model.
Secondly we describe the approaches for designing a multi-dimensional Restful Resources and Web Services
using the mutli-dimensional metric model and demonstrate how the KM system can be ranked and rated for
its effectiveness using WAM and RESTful Resources.
KEYWORDS
Knowledge Management Systems (KMS), Metrics Model, KM Metrics, Multi-dimensional model,
Evaluation, Metrics Database, Ranking, Rating, Weighted Average Mean(WAM), RESTful Resources
1. INTRODUCTION
Knowledge Management (KM) provides an innovative methodology for creating and modifying to
promote knowledge creation and sharing. Many companies and institutions are working towards
building an effective KMS as well as using collaborative tools for increasing the knowledge
sharing with their knowledge workers. One of the key challenges with the KMS’s is evaluation of
the knowledge available within the organization and assessing the capabilities and effectiveness of
the KMS infrastructure [1]. Assessing the worth of the information and the infrastructure within
the organization is a crucial step if an organization wants to change the knowledge capture
methods and rewarding system for employees who have contributed the best knowledge asset.
Considering the intangible nature of the knowledge asset, complexity and dynamics of building the
KM System infrastructure, one of an approach is, to determine the strengths and weakness through
metrics. Metrics can be collected through an evaluation methodology such as Goal Question
Metrics (GQM), Weighted Balanced Score Card and/or using Hybrid methodology [2]. Based on
many research works, it has been identified that there are no proven reliable metric model and
metric database to estimate and report the worth of the knowledge being shared and also the worth
of KM Systems. This paper first describes a comprehensive KM System framework, Metrics and

18
Measurement Process which are critical for conducting an evaluation exercise on a KM System.
Secondly this paper describes the process of building metric database using the multi-dimensional
metric model for capturing the measures and metrics which are collected through evaluation
methods and illustrate how to apply WAM Method to rank and rate the effectiveness of a given
KM system. This paper finally describes the process of building RESTful Resources and Web
Services for evaluating KM Systems.
2. KMS FRAMEWORK
Knowledge management (KM) system is a collective term that is used to describe the creation of
knowledge repositories, respective interface components, improvement of knowledge access and
sharing as well as communication through collaboration, enhancing the knowledge environment
and managing knowledge as an asset for an organization. Considering the fundamental
capabilities of KMS and typical KMS infrastructure topology, we have identified a suitable KMS
framework [2] which is mentioned in the below figure (Figure 1). This framework represents all
the components which make up the KMS for industries as well as higher learning institutions and
in particular focused on the needed quality factors to develop a measurement model based which
are helpful for measuring the effectiveness of KMS.
Figure 1: KMS Framework
3. KM SYSTEM EVALUATION PROCESS
3.1 Metrics
A useful metric should be measurable, independent, accountable & precise. As much as possible,
the KM measures should be related to, or the same as, existing measures in the organization that
are used to monitor the success of performing mission objectives. For evaluating the capability of
the KM system, the metric must indicate the capabilities of the KMS.

19
3.2 Evaluation
An experiment process deliberately imposes a treatment on a group of objects or subjects in the
interest of observing the response. As illustrated in the Figure 2, the basic KM System evaluation
process consist of selection of the quality dimensions and classify them in to subjective or
objective and then applying the hybrid method of using GQM and Score card for data collection.
The selected measures which are generated manually or through system way should be stored in
any available database and later those will be retrieved for ranking and rating using Weighted
Average Mean Method.
Figure 2: Evaluation Process of KM Systems
3.3 Quality Factors
The factors whose effects need to be quantified are called primary quality factors. The features
often discussed concerning the overall quality of the knowledge management system are
capability, availability, reliability, usability, maintainability and completeness. In the proposed
metric model, these quality factors are considered as dimensions. We categorized the factors in
two groups such as primary and secondary and evaluators can choose the factors based on the
evaluation consideration and outcomes. Continuous Inputs for the KMS Measurement process
can be done through standard system programs or tools to monitor the usefulness and
responsiveness of supporting technology or the framework or components used for the KMS. The
system generated factors may also indicate usability problems and supporting policies for the KMS
by introducing the agents which collect these measures.
3.4 Output Metrics
Output metrics measure characteristics at the project or task level, such as the effectiveness of
lessons-learned for future operations. Direct process output for users provides a picture of the
extent to which personnel are drawn to actually using the knowledge system. For a given subject
present in the KMS the metric database can be queried and provide the outcomes such as
OverallFunctionalityRating, OverallUsabilityRating, OverallAvailabilityRating and
OverallEfficiencyRating with respect to the subject or Knowledge Asset available in the
KMS/knowledge repository. The output metrics can be calculated or derived based on the

20
objective and subjective feedback analysis from the knowledge sharing portal or the whole
knowledge infrastructure. Some of these measures can be calculated using system level statistics
and also by developing some background agents or web services or through a manual process.
4. FOUR-DIMENSIONAL METRIC MODEL
As described in the KM System Evaluation process in the earlier section, there are factors and
measures which play a critical role for producing the desired outcomes in the KMS evaluation. So,
KM dimensions also called as categories or view-points should be extensively correlated to many
factors influencing the results as much as possible. The Knowledge management quality factors
enable organizations to strongly indicate what they consider to be important. There are many
forces within an organization and Knowledge Management Systems affecting people’s learning,
sharing, and efficiency, so, it is important to consider the secondary factors as well (Quality Sub
Factors). In this section we describe some of the quality dimensions and the respective measures
and explain the chosen multi-dimensional metric model [3]. The key is selecting appropriate
Quality dimension through factors as represented in the figure (Figure 3) which contains KM
quality dimensions such as Functionality, Usability, Availability and Efficiency. The following
section discusses about measures and metrics corresponding to some of the prime quality factors
which will be given weights in the 80% category. The diagram below represents the four key
dimensions considered for the evaluation of the KM systems. The needed dimensions and
attributed can be added as per the evaluation or prediction.
Figure 3: Four-Dimensional KM Metric Model
4.1 Functionality Measures
The functionality of KMS can be considered as an entity object in the metric model and its
expected behavior will be captured as requirements or attributes of an entity object.

21
4.2 Usability Measures
The knowledge component should be easily understandable, learnable, and applicable. Usability
attribute are the features and characteristics of the software/product/sites that influence the
effectiveness, efficiency and satisfaction with which users can achieve specified goals.
4.3 Availability Measures
In the context of KMS, Knowledge Availability is whether (or how often) a given knowledge asset
is available for use by its intended users.
4.4 Efficiency Measures
The knowledge component should state the quickest solution with the least resource requirements.
4.5 KM Metric Database
The implementation of metric database is shown in the figure (Figure 4) below which was created
to hold the user and expert feedback of the considered dimensions and measures from the database.
The database used in the existing infrastructure can be also considered for storing the metrics and
measurements. As the volume of data and amount of transactions used for the KM System
measurement is very less, there is no need for a dedicated or high performance database and
existing database used for infrastructure maintenance or application database can be used to store
the schema and data. If the models and databases are systematically developed to be flexible,
effective, scalable and surely the data can be explored using mining technique or using WAM
Method. The metric database can be created using any industry specific database systems. There
are four Key steps in implementing the multi-dimensional metric database:
1. Gather the evaluation factors for assessment.
2. Decide the Quality Factor and Sub Factors
3. Create Data Objects (Tables/XML to hold the Quality Factors and Measures)
4. Upon collecting the measures, store them in the data objects
The metric database for our evaluation experiment, has been implemented using Microsoft
Sqlserver 2008 and the data were collected the metrics through Hybrid Approaches [2], manual
feeds and also through customized programs. The data got generated based on the inputs from
user, expert and system. As you see from the below database ER diagram, our metric database is
designed to be flexible to hold any and metric as per the rating from normal user or expert.

22
Figure 3: KM Metric Database ER Diagram
The KM Metric database consists of five key tables namely, KA_BASE_TBL which is the base
table which contains the knowledge asset created/modified in the knowledge portal or repository.
The KA_USER_TBL hold the information about the user name, type and their details.
KA_USER_RATING table holds the user feedback on the given measure. The
KA_METRIC_BASE_TBL which is our key table which holds the attributes like metric id (MID),
Description (MDESC), weight (MWEIGHT) and the corresponding quality factor (DIMENSION).
5. THE EXPERIMENT – KM SYSTEM ANALYSIS
The participants were selected from multiple departments of top two engineering colleges with
moderate and frequent usage of knowledge portals and knowledge repositories. Initially 360
candidates were selected, but only 162 candidates, with similar profiles, were actually used in the
experiment. Prior to conducting the actual experiment, the pilot tests were conducted to validate
the approaches and tasks involved. The main aspects of the normal user profiles of the
participants used were similar in the following ways:
• Computing knowledge, knowledge of using collaborative tools and basic knowledge in
the subject areas present in the KM systems
• Above 20 years of age and below 23 years of age, with English as their learning
language for all the subjects
The main aspects of the expert user profiles of the participants used were similar in the following
ways:
• Teaching or Training skills; Expert knowledge in the subject area
• Willingness to review and provide feedback of the knowledge assets
• Less than 69 years of age and above 27 years of age with apt qualification
For this experiment, we have considered four important dimensions and allocated evaluation
percentage as below:
• 30% for Functionality; 20% for Usability
• 20% for Availability; 20% for Efficiency
The Table-1 for functionality dimension shows how the evaluation percentage (30%) is
distributed among multiple functionality measures. The third column contains the captured rating
which contains aggregated value which is stored in the metric database based on user, expert and
22
ways:
22
ways:

23
system feed. The tables 2, 3 and 4 are populated for Usability, Availability and Efficiency
Dimension using the aggregated value from the metric database. As mentioned in the earlier
sections, for evaluation purpose, we have also considered non-weighted measures and added 10%
to our overall calculation. For an effective and successful KM system, the organization must have
multiple support factors such as Environment, Infrastructure, Domain Knowledge and
participant’s thrust for knowledge sharing. So, we have taken the feedback on these parameters
and added to the metric database and populated in the Table 5.
[
TABLE 1: WEIGHTAGE TABLE FOR FUNCTIONALITY
KM Functionality
Measure/Metric
Weight
(30%)
Captured Rating(from
Metric DB)
Weighted
Calculation
F_M1 OverallFunctionalityRating 20% 3.5 0.7
F_M2 RequirementVerified 10% 3.2 0.32
F_M3 RequirementValidated 10% 3.6 0.36
F_M4 RequirementSeverity 10% 2.1 0.21
F_M5 TacitCategory 10% 4.1 0.41
F_M6 ExplicitCategory 10% 2.4 0.24
F_M7 InnovativeCategory 30% 3.1 0.93
100% Total Score 3.17
KM Dimension Score 0.951
TABLE 2: WEIGHTAGE TABLE FOR USABLITY
KM Usability Measure/Metric Weight
(20%)
Metric DB)
Weighted
Calculation
U_M1 OverallUsability_Rating 20% 3.8 0.76
U_M2 RatingOnOperability 20% 4 0.8
U_M3 RatingOnCommunicativeness 10% 2.9 0.29
U_M4 RatingOnAccessibility 20% 3.5 0.7
U_M5 UserRatingOnUI 20% 4 0.8
U_M6 ExpertRatingOnUI 10% 3.6 0.36
TABLE 3. WEIGHTAGE TABLE FOR AVAILABILITY
KM Availability Measure/Metric Weight
(20%)
Metric DB)
Weighted
Calculation
A_M1 OverallAvailablityRating 20% 3.2 0.64
A_M2 AccessCount 10% 3.9 0.39
A_M3 UpdateCount 10% 3.2 0.32
A_M4 PositiveCommentsCount 10% 4.2 0.42
A_M5 NegativeCommentsCount 10% 1 0.1
A_M6 NeutralCommentsCount 10% 3.7 0.37
A_M7 ActiveThreadsCount 30% 4.1 1.23

24
TABLE 4. WEIGHTAGE TABLE FOR EFFICIENCY
KM Efficiency Measure/Metric Weight
(20%)
Metric DB)
Weighted
Calculation
E_M1 OverallEfficencyRating 20% 3.9 0.78
E_M2 StorageEfficiencyRating 20% 3.5 0.7
E_M3 UserRatingOnEfficiency 30% 4.1 1.23
E_M4 ExpertRankingOnEfficiency 30% 3.6 1.08
TABLE 5. NON-WEIGHTED MEASURES TABLE
KM Non Weighted Measure/Metric (10%) Captured Rating(from Metric DB)
NW_M1Supporting KM Infrastructure 3
NW_M2 Management Support 2
NW_M3 Condusive Environment 1
NW_M4 Particpant’s Subject Knowledge 2
NW_M5 Participants Thrust for Knowledge
Collabration 3
Total Non Weight Value 11
Average 2.2
10 % of Non Weighted Dimension Score 0.22
TABLE 6. OVERALL EFFECTIVENESS TABLE
Allocation Type Derived Score
90% Weighted Dimension
Group 2.79
10% Non Weighted
Dimension Group 0.22
Overall KM System Score 3.01
TABLE 7. RANKING AND RATING TABLE
Rank Category Rating
1 Outstanding 5
2 Extremely Effective 4
3 Effective 3
4 Below Effective 2
5 Not Effective 1
The Table 6 is the summary table and contains both weighted and non-weighted dimension group.
The weighted dimension group score is 2.79, which has been derived by adding all the four
quality dimension scores. The non-weighted dimension group was given 0.22 and both derived
scores were added to make up overall KM System Effectiveness Score. The KM system was
evaluated by the users and experts with the score 3.01. The data obtained for this experiment
concerned with the effectiveness of the KM system of the engineering subjects/knowledge assets.

25
6.RESTFUL RESOURCES
The term REST comes from Roy Fielding's PhD dissertation, published in 2000, and it stands for
REpresentational State Transfer. REST by itself is not an architecture; REST is a set of
constraints that, when applied to the design of a system, creates a software architectural style. A
RESTful resource is anything that is addressable over the Web. The resource can be accessed
and transferred between clients and servers and it is a logical, temporal mapping to a concept in
the problem domain for which we are implementing a solution. Because we are using HTTP to
communicate, we can transfer any kind of information that can be passed between clients and
servers. A Uniform Resource Identifier, or URI, in a RESTful web service is a hyperlink to a
resource, and it's the only means for clients and servers to exchange representations.
In modern web application development we limit design and implementation ambiguity, because
we have four specific actions that we can take upon resources—Create, Retrieve, Update, and
Delete (CRUD). Therefore, with the delineated roles for resources and representations, we can
now map our CRUD actions to the HTTP methods POST, GET, PUT, and DELETE. The
method GET is used to RETRIEVE resources. We need to determine what a resource is in the
context of our web service and what type of representation we're exchanging.
Figure 5: KM System Evaluation Framework using RESTful Architecture
The above framework has been redefined based on our KMS technology Framework [2] found in
Figure-1, to adapt RESTful Architecture to evaluate any KM System using RESTful resources
and Web services. The metrics based on the feedback from Knowledge Workers, Experts, IT
25
6.RESTFUL RESOURCES
25
6.RESTFUL RESOURCES

26
Infrastructure Staff, management staff and also from the System Agents can be stored in the KMS
Metric/Evaluation database or in the form of XML. The RESTFul Webserver can be designed in
such a way, to retrieve the metrics from the KM Metric database and produce the results in the
form of RDF Message.
The Reporting Agent can be used to transform the metric database through RESTful server to
represent in a Text or Graphical or XML form. The agents can be developed to feed the data
through messages to RESTful Web server first and then it can be stored in a traditional metric
database. In order to handle the KM System evaluation, we need to construct a web service first.
The XML representation of the key dimensions from the four dimensional model is given below:
<KM-Functionality>
<Metric-Id> F_M1 </Metric-Id>
<Metric-Name> OverallFunctionalityRating </Metric-Name>
<Metric-Weight> 20 </Metric-Weight>
<Actual-Score> 3.5 </Actual-Score>
</KM-Functionality>
<KM-Usability>
<Metric-Id> U_M1 </Metric-Id>
<Metric-Name> OverallUsabilityRating </Metric-Name>
</KM-Usability>
<KM-Availability>
<Metric-Id> A_M1 </Metric-Id>
<Metric-Name> OverallAvailabilityRating </Metric-Name>
</KM-Availability>
<KM-Efficiency>
<Metric-Id> E_M1 </Metric-Id>
<Metric-Name> OverallEfficiencyRating </Metric-Name>
</KM-Efficiency>
With the representations defined, we now assume URIs of the form http://KM-System-
Eval.com/KM-Functionality to access a list of functionality metrics, and http://KM-System-
Eval.com/KM-Functionality/{Metric-Id} to access a Metric that has the unique identifier of value
Metric-Id. For instance, if we wanted the retrieve or record for a metric with the metric id U_M1,
we make a request to the URI http://KM-System-Eval.com/KM-Functionality/U_M1.
The method GET is used to RETRIEVE resources, the method POST is used to CREATE
resources, the method PUT is used to UPDATE resources and the method DELETE is used to
DELETE representations. The table 8 contains the URI and HTTP verb mapping for business
operations of the KM System Evaluation.

27
TABLE 8: URI-HTTP MAPPING TABLE
URI HTTP
Method
COLLECTION OPERATION EVALUATION OPERATION
/ F_M1 GET /KM-Functionality RETRIEVE GET SCORES
/F_M1 POST /KM-Functionality CREATE STORE SCORES
/ U_M1 GET /KM-Usability RETRIEVE GET SCORES
/U_M1 POST /KM-Usability CREATE STORE SCORES
/ A_M1 GET /KM-Availability RETRIEVE GET SCORES
/A_M1 POST /KM-Availability CREATE STORE SCORES
/ E_M1 GET /KM-Relevance RETRIEVE GET SCORES
/E_M1 POST /KM-Relevance CREATE STORE SCORES
Designing RESTful web services is not different from designing traditional web applications. We
can think of RESTful web service design as being similar to Object Oriented Design (OOD).
The underlying RESTful web service design principles can be summarized in the following four
steps:
1. Requirements gathering for KM System Evaluation
2. Resource identification based on proposed multi-dimensional model
3. Resource representation definition—between clients (User Interfaces or Agents) and
servers (KM metric database or Web Server). For KM metrics, we can use XHTML.
4. URI definition—with resources in place, we need to define the API, which consists of
URIs for clients and servers to exchange resources' representations.
CONCLUSION
In this research work, we have attempted to build a four dimensional model, the metric database
and a proven statistical technique Weighted Average Mean to validate the effectiveness of the
knowledge sharing in the KM systems. By referring to Table 6 and the guideline Table 7 for
ranking and rating, it is clear, that the evaluated KM system is effective as the KM system got the
overall KM System Score of 3.01. The results obtained through the experiment, proves that the
combination of Metric Database with a Statistical technique such as WAM could be useful to
predict the usefulness and effectiveness of the KM Systems. In complex KM Systems, the KM
System Infrastructure is heterogeneous and contains multiple social and economic factors [2] and
hence more dimensions have to be considered. Automation would be flexible and cost effective
using RESTful Resources and Web Services when used in conjunction with the metric database
for better metrics collection and reporting.
REFERENCES
[1] Dayanandan Venkata Subramanian, Angelina Geetha (2011), “Guidelines and Effectiveness of Using
Social Media Tools For Knowledge Sharing Communities”, National Conference on Knowledge
Intelligence and Telematics, Gujarat, India
[2] Venkata S Dayanandan, Angelina Geeteha(2011), “Adaptation of Goal Question Metric Technique
For Evaluation of Knowledge Management Systems”, Review of Knowledge Management Journal,
Volume 1, Jan-June 2011, ISSN:2231-5845

28
[3] Venkata S Dayanandan, Angelina Geeteha, Mohammed Hussain(2011), “Measurement Process and
Multi-dimensional Model For Evaluating Knowledge Management Systems”, International
Conference on Research and Innovation in Information Systems, International Islamic University,
Malaysia (IIUM) and Universiti Teknologi Malaysia(UTM), Indexed by IEEE & SCOPUS,
November 2011, IEEE ISBN: 978-1-61284-295-0
[4] Diwakar Krishnamurthy, Jerry Rolia, and Min Xu1(2011), “WAM – The Weighted Average Method
for Predicting the Performance of Systems with Bursts of Customer Sessions, University of Calgary,
Calgary, AB, Canada, IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, 0098-5589/11
[5] Jiawei Han, Yizhou Sun, Xifeng Yan, Philip S. Yu(2010), “Mining Knowledge from Databases: An
Information Network Analysis Approach”, SIGMOD’10, June 6–11, 2010, Indianapolis, Indiana,
USA. ACM 978-1-4503-0032-2/10/06.
[6] Behrang ZadJabbari, Pornpit Wongthongtham, Farookh Khadeer Hussain(2010),“Ontology based
Approach in Knowledge Sharing Measurement”, Communications of the IBIMA Volume 11”, 2009
ISSN: 1943-7765
[7] Ashleigh-Jane Thompson, Elizabeth A Kemp, “Web 2.0: Extending the framework for heuristic
evaluation”, Proceedings of the 10th international conference NZ Chaperte of the ACM's special
interest group on Human Computer Interface, 2009
[8] Stephanie D Swindler, Christian E Randall, Gary S Beisner II, “An application of heuristic techniques
for evaluation of collaborative systems interfaces”, nternational Symposium on Collabarative
Technologies and systems, 2006.
[9] A.Kankanhalli and B.C.Y. Tan, National University Of Singapore, “K.M Metrics : A Review and
Directions for future research”, Proceedings of the 37th Annual Hawaii International Conference in
Knowledge Management, 2004
[10] B. Martens and W. Jabi, “Knowledge Portal as a new paradigm for scientific publishing and
collaboration”, IT conference, 2004

KM System Evaluation using Four dimensional Metric Model, Database and RESTful Resources

More Related Content

What's hot (14)

Similar to KM System Evaluation using Four dimensional Metric Model, Database and RESTful Resources (20)

Recently uploaded (20)

KM System Evaluation using Four dimensional Metric Model, Database and RESTful Resources