Combining Two Models of Successful Information System Measurement

TELKOMNIKA, Vol.16, No.4, August 2018, pp. 1793~1800
ISSN: 1693-6930, accredited First Grade by Kemenristekdikti, Decree No: 21/E/KPT/2018
DOI: 10.12928/TELKOMNIKA.v16i4.7737  1793
Received November 4, 2017; Revised March 26, 2018; Accepted June 29, 2018
Combining Two Models of Successful Information
System Measurement
Pualam Dipa Nusantara*, Nyoman Ayu Gita Gayatri, Martin Suhartana
Computer Science Department, School of Computer Science, Bina Nusantara University
Jln. K. H. Syahdan No. 9, Jakarta 11480, Indonesia
*Corresponding author, e-mail: pualamd@gmail.com
1
, ngayatri@binus.edu
2
,
martin.suhartana@gmail.com
3
Abstract
This paper purposes is to measure successful of Academic Advisory information system by
combining two models of information system measurement. DeLone & McLean IS Success Model use to
measure the successful of system while COBIT framework is to measure system maturity level. Result of
this research showed that the successful of Academic Advisory IS affected by User Satifaction, Quality of
Service, Quality of System while Maturity level at 3.7. The result also showed there’s a relation between
level of maturity system with the success of system.
Keywords: DeLone McLean, COBIT, maturity level, information system
Copyright © 2018 Universitas Ahmad Dahlan. All rights reserved.
1. Introduction
Implementation of information system can support organization to achieve its goals.
According to James O’Brian [1] Information systems have become as integrated into our daily
business activities as accounting, finance, operations management, marketing, human resource
management, or any other major business function. Information systems and technologies are
vital components of successful businesses and organizations. In educational organizations such
as university information systems have also been implemented, one of the information systems
academic advisory. At BINUS University Academic Advisory information system is a means
provided by the campus so that students can consult about their academic activities with
lecturers that appointed as mentors. In addition to assisting lecturers and students in conducting
communication and scheduling to conduct meetings, academic advisory information system
also helps in the data collection of academic achievement of students ranging from grades,
course schedules, and courses taken in the current semester. But the problem arises when the
supervisor says that the student often does not come on a set schedule while the student is
reasonably late in knowing the information or not even knowing the information. These
circumstances may prevent students from obtaining good academic advisory services. Because
it is necessary to measure whether the system has been running as expected.
Measurements of the information system have been performed in the following studies.
In the previous study Fuad Budiman [2] in his research measure the success of the
implementation of regional management information system using Technology Acceptance
Model (TAM) approach. While in her research Junita Juwita [3] perform analysis of TAM factors
that influence in the use of knowledge management applications for small and medium
enterprises in the creative industry. But TAM focuses more on providing general explanations of
what determines technology acceptance. Another paper by Setiawan Assegaf use DeLone and
McLean information system success model (D & M model) to measure social media success for
knowledge sharing [7]. While study conducted by Johan and Angelia [8] use the 6 dimensions of
D & M model to measure BINUS University Information System. Information system
measurement can be considered as audit of the system. Audit system can be applied to
evaluate whether information system implemented effectively. Enterprises need to measure
where they are and where improvement is required. Maturity models to enable benchmarking
and identification of necessary capability improvements. In a study conducted by Diema and
Fia [16] maturity level of COBIT framework was applied to evaluate academic information
system in order to improve service for user satisfaction. An empirical study also done by

 ISSN: 1693-6930
TELKOMNIKA Vol. 16, No. 4, August 2018: 1793-1800
1794
Irmawati [13], Diana and Michel [14], Azhari and Melia [15] to evaluate information system
maturity level. On the other studies Haryanto and Sarno [17] conduct a research to propose the
use of COBIT Maturity Model (CMM) and Structural Equation Model (SEM) to measure the
alignment between the University Academic Regulations and Information Technology Goals
where the results of this study proved that the alignment measurement using CMM and SEM
gave relatively the same results, which described the same priority list of maturity levels of the
IT processes.
Based on the exposure of the previous studies above, through this research we
combine two measurements of information systems to measure the academic advisory
information system. D & M model to measure the success of academic advisory information
systems and CMM to measure the maturity level of academic advisory information system also
to analyze the relation between the success of academic advisory information system with the
system maturity level to find the affected factors of successful information system.
2. Research Method
According to DeLone and McLean [5], where have been revised [6], the implementation
of information systems is said to be successful if organizations get the net benefits of
information systems, while the net benefits gained due to user satisfaction in using the system.
In this case, user satisfaction in using the system is influenced by the information quality,
service quality and system quality. The D & M model proposed by DeLone and McLean as
depicted as shown in Figure 1.
Figure 1. D & M information system success model
On the other hand according to The IT Governance Institute [12] the advantage of a
maturity model approach is that it is relatively easy for management to place itself on the scale
and appreciate what is involved if improved performance is needed.
2.1. Measurement and Indicators Development
In this study data collection is done through questionnaire sheet. The distribution of
questionnaires was conducted to users of the Academic Advisory Information System. Of the
200 questionnaires that spread as many as 150 were returned with details to measure the
success of information systems with a total of 140 users of system users. Meanwhile, to
measure the maturity level of the system as much as 10 respondents.
Measurement variables for D & M model used Likert scale from strongly disagree to
strongly agree. The scale is indicated by the following criteria: number 1 means strongly
disagree (STS), 2 means disagree (TS), 3 means sufficient (C), 4 means agree (S), 5 means
strongly agree [18]. Indicators for D & M model shown in Table 1.
On the other measurement, COBIT framework has defined information technology
activities in four domain that is Plan and Organize, Acquire and Implement, Deliver and Support,
Monitor and Evaluate. Maturity levels in COBIT framework are designed as profiles of IT
Information Quality
System Quality
Service Quality
System Use
User Satisfaction
Net
Benefits

TELKOMNIKA ISSN: 1693-6930 
Combining Two Models of Successful Information System... (Pualam Dipa Nusantara)
1795
processes that an enterprise would recognise as descriptions of possible current and future
states. The maturity levels scale are 0-non existent, 1-Initial/Ad-hoc, 2-Repeatable but Intuitive,
3-Defined Process, 4-Manage and Measureable, 5-Optimised [12]. The questionnaires to asses
maturity level of information system was taken from the statement in each COBIT Maturity
level [12].
Table 1. Variables and Indicators
VARIABLE INDICATOR Source
Quality of System (Qsys)
X1 = System flexibility
X2 = System availability
X3 = integration completeness
X4 = Integration successfulness
X5 = Response speed
X6 = Response consistency
[19 ]
X7 = Error recovery
X8 = Recovery completeness
X9 = Access convenience
X10 = ease to use
X11 = Command used
X12 = Command ready
Quality of Information (QI)
X13 = Information consistency
X14 = Information availability
X15 = Iinformation accuracy
X16 = Consistency and accuracy [19 ]
X17 = Actual information
X18 = on time information
X19 = output simplicity
X20 = ease to understand
Quality of Service (QServ)
X21 = Tangibles
X22 = Reliability
X23 = Responsiveness
X24 = Assurance
X25 = Emphaty
[20 ]
User Satisfaction (USatisfy)
Y1 = Easy to use system
Y2 = Happy to use system
Y3 = informatin availability
Y4 = Grows motivation
Y5 = System flexibility
[19 ]
Net Benefits (NetB)
Y6 = Performance improvement
Y7 = Accelerate the task
Y8 = Productivity improvement
Y9 = Effectiveness improvement
Y10 = Easier the task
Y11 = Usefull
[19 ]
2.2. Proposed Model
In research conducted by Livari [9] provide empirical evidence that the Quality of the
System and the Quality of Information does not have a significant effect on the ntensity of Use,
but has significant effect on User Satisfaction. This is because the object of research using a
mandatory system. Other research conducted by McGill [10] find that Quality of the System and
Quality of the Information was a significant predictor to User Satisfaction, but not a significant
predictor for System of Use. Academic information system is a mandatory system. Based on
exposure above the developed model for this research dropped System of Use variable, as
shown in Figure 2.
3. Results and Analysis
Since this study purpose is to analyze relationship between variables the researchers
use Structural Equation Modelling (SEM) to analyze the proposed research model. SEM is a
multivariate statistical technique that is a combination of factor analysis and regression analysis,
which aims to examine the relationships among variables that exist in a model [21].

 ISSN: 1693-6930
1796
3.1. DM Model
In SEM, Confirmatory Factor Analysis (CFA) measurement intended to confirm that
indicator are valid constructor to its latent variable. The result of CFA measurement showed that
all indicators estimation above 0.5 which is fulfilled validity criteria (> 0.5) as shown in Table 2.
Figure 2. Proposed research model
Table 2. Indicator Validity
Variable Indicator Estimate Validity ( > 0.5)
X1 .782 valid
X2 .767 valid
X3 .795 valid
X4 .736 valid
X5 .795 valid
Quality of System (Qsys) X6 .865 valid
X7 .787 valid
X8 .733 valid
X9 .813 valid
X10 .787 valid
X11 .667 valid
X12 .735 valid
X13 .851 valid
X14 .866 valid
X15 .878 valid
Quality of Information (QI) X16 .900 valid
X17 .896 valid
X18 .799 valid
X19 .830 valid
X20 .793 valid
X21 .887 valid
Quality of Service (Qserv) X22 .843 valid
X23 .875 valid
X24 .825 valid
X25 .792 valid
Y1 .819 valid
Y2 .897 valid
User Satisfaction (USatisfy) Y3 .893 valid
Succesful
Information
System
D & M
Model
System Quality
Information Quality
Service Quality
User Satifaction
Net Benefits
COBIT Domain
Monitor and
Evaluate
Monitor and evaluate IT
performance
Monitor and evaluate internal
control
Ensure compliance with internal
requirements
Provide IT
governance

1797
Table 2. Indicator Validity
Variable Indicator Estimate Validity ( > 0.5)
Y4 .771 valid
Y5 .801 valid
Y6 .860 valid
Y7 .902 valid
Net Benefits (NetB) Y8 .910 valid
Y9 .884 valid
Y10 .822 valid
Y11 .854 valid
After finding in confirmatory factor analysis that all indicators are valid to its variable, the
next step is to analyze the structural model. At this stage we analyzed the overall model
conformity test and the significance of the causality relationship buit into the model. Based on
AMOS software calculation we found that Quality of Information (QI) have P=0.51 (see Table 3)
which is above the cut off of 0.05 and negative value in relation with User Satisfaction
(USatisfy) (see Table 4). Quality of Service (Qserv) and Quality of System (Qsys) have a
relation to User Satisfaction (Usatisfy) 0.64 and 0.35 respectively. Furthermore User
Satisfaction (Usatisfy) have a relation with Net Benefits (NetB) as big as 0.83 (see Table 4).
Table 3. Regression Weights of Research
Model
S.E. C.R. P Label
Usatisfy <--- QI .045 -.657 .511
Usatisfy <--- Qserv .077 8.262 ***
Usatisfy <--- Qsys .033 5.206 ***
NetB <--- Usatisfy .061 17.348 ***
Table 4. Standardized Regression Weights of
Research Model
Estimate
Usatisfy <--- QI -.048
Usatisfy <--- Qserv .643
Usatisfy <--- Qsys .354
NetB <--- Usatisfy .835
Because P=0.51 as shown in Table 3 does not meet the requirement and negative
impact from Quality of Information (QI) to User Satisfaction (USatisfy) as shown in Table 4 we
modify the model as the last model by dropping Quality of Informatin (QI) variable. After
dropping Quality of Information (QI) variable the next step is to re-calculate the estimation. The
result of the modification model calculation shows that Quality of Service (Qserv) and Quality of
System (Qsys) have a relation to User Satisfaction(Usatisfy) 0.61 and 0.33 respectively.
Furthermore User Satisfaction (Usatisfy) have a relation to Net Benefits (NetB) 0.83 as shown in
Table 5.
Table 5. Standardized Regression Weights of Last Model
Estimate
Usatisfy <--- Qserv .614
Usatisfy <--- Qsys .339
NetB <--- Usatisfy .835
Overall, the result can be described as follow: Quality of Service and Quality of System
have a relation to User Satisfaction although Quality of Service have more strongest relation to
User Satisfaction. User Satisfaction have a strong relation to Net Benefits.
Comparing this study to other papers, the results is support previous researches
conducted by Livari [9] and McGill [10] that use DeLone and McLean [6] Information System
Success Model for measuring successful of information system with results that were only
partially proven.
3.2. Maturity Level
The data collection in this study was carried out by spreading questionnaires to
respondent had meet the criterria of RACI chart. COBIT defines RACI chart as the duties, which

 ISSN: 1693-6930
1798
are Responsible, Accountable, Consulted, and Informed. The questions of questionnaires is
taken from control objectives of Monitor and Evaluate (ME) domain [12]. The result of maturity
level based on questionaires on Monitor and Evaluate domain shown by Table 6.
Table 6. Maturity Level of ME Domain
Domain
Level
Total
Maturity
level0 1 2 3 4 5
ME1.1.1 4 2 1 25 3.6
ME1.1.2 2 4 1 27 3.9
ME1.1.3 4 2 1 25 3.6
ME1.2.1 2 4 1 27 3.9
ME1.2.2 1 1 5 25 3.6
ME1.2.3 6 1 22 3.1
ME1.3.1 4 2 1 25 3.6
ME1.3.2 4 2 1 25 3.6
ME1.4.1 2 5 26 3.7
ME1.5.1 1 2 4 24 3.4
ME1.5.2 3 4 25 3.6
ME1.5.3 3 3 1 26 3.7
ME1.6.1 2 4 1 27 3.9
ME1.6.2 2 5 26 3.7
ME2.1.1 1 5 1 27 3.9
ME2.2.1 1 2 4 24 3.4
ME2.3.1 2 5 26 3.7
ME2.3.2 2 4 1 27 3.9
ME2.4.1 3 4 25 3.6
ME3.1.1 3 2 2 27 3.9
ME3.2.1 1 5 1 28 4
ME3.3.1 2 5 26 3.7
ME3.4.1 3 4 25 3.6
ME3.5.1 1 2 4 24 3.4
ME4.1.1 1 1 4 1 26 3.7
ME4.1.2 1 6 27 3.9
ME4.2.1 1 2 4 24 3.4
ME4.2.2 2 4 1 25 3.6
ME4.2.3 2 4 1 27 3.9
ME4.2.4 3 2 2 27 3.9
ME4.3.1 1 5 1 28 4
ME4.3.2 2 5 26 3.7
ME4.3.3 2 5 26 3.7
ME4.4.1 3 3 1 26 3.7
Average 3.7
There is a gap when we compare between the result of existing maturity level and the expected
maturity level. We can see the gap as the Figure 3 shown.
Figure 3. Maturity level gap

1799
3.3. Maturity Level Relation with Information System Success
After we calculate DM model measurement and get the result of existing maturity level
of BINUS University Academic Advisory we combine the model to find out if there's a relation
between ME domain of Cobit maturity level and Academic Advisory success model. As shown
of the Table 7 above we can see there's an impact from Cobit maturity level ME domain to Net
Benefits variable which is the impact of successful information system. The result shown that
there’s a relation between Cobit maturity level to information system success of 0.59 . The result
about this study supports the research conducted by Johan and Angelia [8] that there’s a
relation between the maturity level of system and successful of information system.
Table 7. Standardized Regression Weights COBIT ME and DM IS Success Model
Estimate
NetB <--- ME .590
4. Conclusion
The result of this paper provides affected factors to the successful of Academic
Advisory information system. The finding prove that Quality of Information (QI) is not the
affected factor to the successful of BINUS University Academic Advisory information system.
The success of the Academic Advisory Information System is affected by Quality of System
(Qsys), Quality of Service (Qserv), User Satifaction (Usatisfy) and Net Benefits (NetB). Where
Quality of System has an impact of 0.33 to User Satisfaction and Quality of Service has an
impact of 0.61 on User satisfaction and User Satisfaction has an impact of 0.83 against Net
Benefits. In this research variable Quality of Information (QI) has a negative impact of -0.04 on
User Satisfaction.
Academic Advisory system maturity level is at level 3.7 where the gap with level 4 is
quite small (0.3). However, recommendations are given for improvements to all sub-processes
in the ME domain accordance with the COBIT framework documentation [8], especially in sub-
processes that have a low enough value (ME1.2.3, ME1.5.1, ME2.2.1, ME3.5.1, ME4.2.1). The
result of this research also shows that there is a relationship between system maturity level and
the success of information system. In other word maturity level is the affected factor to the
successful of information system. However, the relationship between the maturity level of the
system and the success of the information system is not very strong relation. The next research
will be done by adding more data collection and modification of relevant indicator.
References
[1] James A. O’Brian. Introduction to Information Systems. Fifteenth edition. New York: McGraw-Hill.
2010.
[2] Fuad Budiman, Fefri Indra. Pendekatan Technology Acceptance Model Dalam Kesuksesan
Implementasi Sistem Informasi Manajemen Daerah. Jurnal WRA. 2013; 1(1): 87-110.
[3] Junita Juwita Siregar, RA Aryanti Wardaya P, Anita Rahayu. Analysis of Affecting Factors
Technology Acceptance Model in the Application of Knowledge Management for Small Medium
Enterprises in Creative Industry. Procedia Computer Science. 2017; 116: 500-508.
[4] Ives B., Olson MH, Baroudi IJ. The Measurement of User Information Satisfaction. Communication of
the ACM. 1983; 26(10): 785-793.
[5] WH Delone, ER McLean. Information systems success: the quest for the dependent variable.
Information Systems Research. 1992; 3(1): 60–95.
[6] WH Delone, ER McLean. The DeLone and McLean model of information systems success: a ten-
year update. Journal of Management Information Systems 2003; 19(4): 9–30.
[7] Setiawan Assegaff, Hendri, Akwan Sunoto, Herti Yani, Desy Kisbiyanti. Social Media Success Model
for Knowledge Sharing (Scale Development and Validation). TELKOMNIKA (Telecommunication
Computing Electronics and Control). 2017; 15(3): 1335-1343.
[8] Johan Muliadi Kerta, Angellia Debora Suryawan. Analysis of Information System Implementation In
Binus University using DeLone and McLean Information System Success Model and COBIT
Framework. CommIT. 2013; (1): 13-17.
[9] Iivari, Juhani. An Empirical Test of the DeLone-McLean Model of Information System Success. The
Database for Advances in Information Systems. Spring. 2005; 36(2): 8-27.

 ISSN: 1693-6930
1800
[10] T Hobbs V, Klobas J McGill. User-Developed Aplications and Information Systems Success: A Test
of DeLone and McLean’s Model. Information Resources Management Journal. 2003.
[11] Radityo, Dody. Pengujian Model DeLone dan McLean Dalam Pengembangan Sistem Informasi
Manajemen (Kajian Sebuah Kasus). Simposium Nasional Akuntansi X. Makasar. 2007: 1 - 25.
[12] IT Governance Institute. COBIT 4.1 Framework Control Objectives Management Guidelines Maturity
Models. Il: ITGI. 2007: 17.
[13] Irmawati Carolina. Analisa Penilaian Maturity Level Tata Kelola TI Berdasarkan Domain DS dan ME
Menggunakan COBIT 4.1. Seminar Nasional Inovasi dan Tren (SNIT). 2015: 191-196.
[14] Diana Trivena Yulianti, Michel Canggih Patria. Audit Sistem Informasi Sumber Daya Manusia pada
PT X Menggunakan Cobit Framework 4.1. Jurnal Sistem Informasi: 2011; 6(1): 15-33.
[15] [Azhari Shouni Barkah, Melia Dianingrum. Evaluasi Penerapan Sistem Informasi dan Teknologi
Informasi Menggunakan COBIT Framework Di STMIK AMIKOM Purwokerto. ProBisnis. 2015; 8(1):
22-30.
[16] Diema Hernyka, Fia Mahanani. Audit Sistem Informasi Akademik Perguruan Tinggi XYZ
Menggunakan Kerangka Kerja COBIT 4.1. Seminar Nasional Aplikasi Teknologi Informasi (SNATI).
Yogyakarta. 2014: 1–6.
[17] Haryanto Tanuwijaya. Riyanarto Sarno. Comparation of Cobit Maturity Model and Structural Equation
Model for Measuring the Alignment between University Academic Regulations and Information
Technology Goals. IJCSNS International Journal of Computer Science and Network Security. 2010;
10(6): 80-92.
[18] U Sekaran, R Bougie. Research Methods for Business: A Skill Building Approach. 5th Ed. New
Jersey: John Wiley and Sons. 2010.
[19] Jogiyanto, HM. Model Kesuksesan Sistem Teknologi Informasi. Yogyakarta: ANDI. 2007.
[20] Aritonang, RL. Kepuasan Pelanggan. Jakarta: PT Gramedia Pusaka Utama. 2005.
[21] Syarah Widyaningtyas, Triastuti Wuryandari, Moch. Abdul Mukid. Pengaruh Marketing Mix Terhadap
Kepuasan dan Loyalitas Konsumen Menggunakan Metode Structural Equation Modelling (SEM).
Jurnal GAUSSIAN: 2018; 5(3): 553–562.

Combining Two Models of Successful Information System Measurement

More Related Content

What's hot (14)

Similar to Combining Two Models of Successful Information System Measurement (20)

More from TELKOMNIKA JOURNAL (20)

Recently uploaded (20)

Combining Two Models of Successful Information System Measurement