A comparison between evaluation of

International Journal of Software Engineering & Applications (IJSEA), Vol.5, No.1, January 2014

A COMPARISON BETWEEN EVALUATION OF
COMPUTER BASED TESTING AND PAPER BASED
TESTING FOR SUBJECTS IN COMPUTER
PROGRAMMING
Samantha Mathara Arachchi 1, Kapila Dias 2, R.S. Madanayake 3
1,2,3 Department of Communication and Media Technology
University of Colombo School of Computing,
35, Reid Avenue, Colombo 7, Sri Lanka.
4

Eddy Siong Choy Chong4,
Management and Science University, Malaysia.

Kennedy D. Gunawardana 5
5

Department of Accounting, University of Sri Jayawardhanapura, Sri Lanka

ABSTRACT
Evaluation criteria for different disciplines vary from subject to subject such as Computer science,
Medicine, Management, Commerce, Art, Humanities and so on. Various evaluation criteria are being used
to measure the retention of the students’ knowledge. Some of them are in-class assignments, take home
assignments, group projects, individual projects and final examinations. When conducting lectures in
higher education institutes there can be more than one method to evaluate students to measure their
retention of knowledge. During the short period of the semester system we should be able to evaluate
students in several ways. There are practical issues when applying multiple evaluation criteria. Time
management, number of students per subject, how many lectures delivered per semester and the length of
the syllabus are the main issues.
The focus of this paper however, is on identification, finding advantages and proposing an evaluation
criterion for Computer based testing for programming subjects in higher education institutes in Sri Lanka.
Especially when evaluating hand written answers students were found to have done so many mistakes such
as programming mistakes. Therefore they lose considerable marks. Our method increases the efficiency
and the productivity of the student’s and also reduces the error rate considerably. It also has many benefits
for the lecturer. With the low number of academic members it is hard to spend more time to evaluate them
closely. A better evaluation criterion is very important for the students in this discipline because they
should stand on their own feet to survive in the industry.

KEYWORDS
Computer based testing, Paper based testing, Hypotheses testing

1. Introduction
This research was conducted by using the 2nd year students who follow computer subjects. The
main purpose of this research is to enhance the students’ performance to score more marks with
less programming mistakes such as syntax, punctuations errors and also to reduce the staff stress
DOI : 10.5121/ijsea.2014.5105

57


due to paper marking [12] and save time, since there are more than five hundred answer scripts to
be marked for some subjects. When marking papers in bulk (such as more than five hundred) the
man-power is not enough to mark all these within the given period. Often deadlines have to be
extended. Therefore marking is a terrible burden [3].
Computer based testing is a very convenient method. In addition to this, it increases the
transparency of marking and students get the freedom to develop the application using their own
concepts and methods. It develops their creativity as well as thinking power. Another reason is
that reading coding is very hard because most of their hand writing is not so neat. Since there is
no way to debug their coding there are so many errors and mistakes. This research has
encountered advantages for the students as well as for the lecturers, in conducting computer based
assignments in subject in computer programming [7],[9].
Students have another advantage in using machines as they do not need to remember unnecessary
theories or memorise some points [4]. As we know when we are in front of the computer we will
remember what to do next automatically compared with when answering a paper using a
computer.
Computer training is necessary for the software developers as an on the job training. In-house
staff development centers also conduct paper based examinations to measure their performance
and understand about the latest concepts and technology or new programming languages [16].
Even though they are getting on the job training as well as they are in touch with the
programming. According to survey questionnaire they also preferred practical based testing then
the paper passed testing [17].

2. Methodology
2.1. Research Methodology
To test the proposed research model we used a survey method and collected data for analysis. The
approaches followed for conducting the research include Chi Square, Analysis and Frequency
Analysis.
Initially two groups have been formed as group “A” and Group “B” including 50 students for
each group and the same computer programs were given to them to develop. Computers were not
allocated for group “A” and were assigned to write on paper. Group A is the paper based testing
group. The Group “B” has been provided computers to develop the same programs within the
same time. Group B is the computer based testing group. Group “A” papers were evaluated by 20
lecturers and the same paper was given for Group B evaluated by the same 20 lecturers. Before
filling the questionnaire which was given to the lecturers. The results were evaluated to identify
the advantages of the evaluation criteria. The same tasks have been given to students who are in
the group “B” to develop. Lecturers have checked only its output. But there were students who
could not develop and get the final output. Only at that time the computer was used to examine
the source code to assign marks [20].
After the test a questionnaire was given to both groups to identify and measure the advantages of
the method. The results have been evaluated to get the feedback from them. Another
questionnaire was given to the lecturers who have marked paper based testing and computer
based testing to collect their views and feedback.
58


A special questionnaire has been given to the software developers who are in the industry whose
the main job is computer programming to develop software and Enterprise Resource Planning
(ERP) applications. The objective of this is to measure the willingness or the views of computer
based testing from the people who need to follow in-house on the job training for knowledge
improvement and increments [17].
2.1.1. The Sample
Data was collected from 50 students of the 2nd year who followed computer subjects and 20
lecturers who teach computer subjects for higher education institutes as configuration teams.
Twenty five students were females with an average age of 21 years and the other Twenty five
students were males with an average age of 21 years.
The second sample was based on the 50 system developers who are really engaging with writing
programming to develop software applications.
2.1.2. Evaluate students’ paper based testing
The paper based testing answer scripts were evaluated according to the properly designed
marking scheme. After marking the answers it should work perfectly without any mistakes. A
Model answer script is shown below;

Figure 1. Model Answer Script 1

The answer scripts of students’ who were in the group “A” (paper based testing) were evaluated
and generated the following statistics. When evaluating answer scripts the marks were allocated
for the key main factors of syntax, punctuations, methodology and also for the final output
[20],[22]. The statistics of number of students who received marks and did not score full marks
for the paper based testing as shown in the Table 1.

59

Table 1. Number of students who received and did not score full marks from paper based testing

No

Types of Mistakes

1
2
3
4
5
6
7
8
9
10

Number of students Number of students who
who received marks
did not score full marks

Semi Colon
Colon
Period
Single Quotation
Double Quotation
Comma
Simple Brackets
Square Brackets
Methodology
Final out put

18
10
12
15
16
14
22
17
35
9

32
40
38
35
34
36
28
33
15
41

Only nine students have done the coding excellently and obtained the final output with full marks.
The percentage of getting full mark is 18% while 82% were there who could not score full marks
due to the above reasons. The statistics of the number of students who repeatedly did mistakes for
paper based testing is shown in the Table 2.
Table 2. Number of students who repeatedly did mistakes for paper based testing

Number of Types of Mistakes
1
2
3
4
5

One Mistakes
Two Mistake s
Three Mistakes
Four Mistakes
More than Four Mistakes

Number of
Students
8
12
7
9
5

According to the above table it shows that more than 60% of students have done more than two
mistakes and it is more than the average.
Screenshots of the students answer sheets are shown below. The mistakes that they have done
were encircled and categorized according to one, two, three, four and more than four mistakes.

Figure 2. Student’s Answer Script with one mistake

60


The above figure (Figure 2) shows one student’s paper based answer script and it has one mistake.
He has missed the single quotation where the examiner has encircled. When writing a program on
a piece of paper it never catches the eye or there is no way to debug the program. If you used the
same coding in the machine even if the mistake is just missing a single quotation the program
does not work and it takes time to find the error. Through programming experience people can
catch this error. But when developing a system these tiny errors make big mistakes and take a
long period recover or solve the problems.

Figure 3. Student’s Answer Script with one mistake

Figure three (Figure 3) also shows a single mistake. When marking answer scripts it does not
catch the eye as discussed earlier. Because of that error the program also does not work. Then
how can the examiner give full marks to the student because this piece of coding also does not
give the output we expect.

Figure 4. Student’s Answer Script with three mistakes

The above figure four (Figure 4) has three mistakes. An examiner has to read all the lines with
tiny symbols or needs to follow the programme syntax to check whether the piece of coding is
working or not. It is a big issue to read a lot of coding as well as many lines for the programming
based subjects. All these coding needs to be read line by line to catch mistakes considering
61


possible error types such as syntax, semantic, logical and reserved key words in the mind of the
examiner. Without hundred percent knowing what the student wanted to write, how can the
examiner allocate full marks for the students?

Figure 5. Student’s Answer Script with four mistakes

Figure 6. Student’s Answer Script with more than four mistakes

The above figure (Figure. 6) has many mistakes where the examiner has encircled and underlined.
When preparing and allocating marks for the above type paper it is hard because the examiner
needs to have a predefine marking scheme showing how to deduct or allocate marks for the
correct and incorrect points. All the marks should be able to justify.

Figure 7. Student’s Answer Script with more than four mistakes and incomplete lines

62


Figure 8. Student’s Answer Script with more than four mistakes and incomplete lines

The above figures (Figure 7 and Figure 8) have many mistakes. Also the student has missed some
important lines. Therefore the examiner should be able to completely understand and check the
entire coding. Sometimes we cannot reject the student’s answers saying it is not in the marking
scheme, because that answer is also correct and the examiner never thought about that point of
view to solve the problem.

Figure 9.1. Student’s Answer Script with simple brackets and square brackets mistakes

Figure 9.2. Student’s Answer Script with simple brackets and square brackets mistakes

The students have been done mistakes in selecting the correct types of brackets. For instance
some of the students have used simple brackets instated of square brackets. Either it could be
carelessness or they really do not know about the correct type of brackets as shown in the figure
9.1 and 9.2 (Figure 9.1 and 9.2).

63


Figure 10. Student’s Answer Script with spelling mistakes

Some of the students used wrong spelling for reserved keywords in that particular language such
as “mysql_connect”. In here students have spelled it as “mysql_conect” as shown in the above
figure 10 (Figure 10). This coding is also not working correctly to get the expected output. Most
of the programming tools do not have spelling checkers. The Examiner as well as the
programmer needs to be careful about spelling and self confident in order to reduce the mistakes
and save time.

Figure 11. Student’s Answer Script with mistakes in the methodology

Some answer scripts were found with wrong programming methodology or wrong structure.
Figure 11 (Figure 11) has such a mistake instead of the “for loop”, the while loop has been used.
2.1.3. Evaluation of the students’ questionnaire
After doing the above paper based testing students who were in the group “A” filled a
questionnaire to express their own attitudes towards this type of paper based testing. The
statistics are shown in the Table 3.

64

Table 3. Number of students who answered questionnaires for paper based testing

No

Reason

Number of Students

1

Need to memorise concepts and unnecessary theory

Yes
40

No
10

2
3
4
5
6
7
8
9

Reduce stress
More confident about answers
Answer is visible
Working in front of machine is comfortable
Interface provides the extra support
Can debug time to time to make it correct
More freedom to come up with alternative methods
Easy to score maximum marks

15
17
0
50
45
0
0
8

35
23
50
0
5
50
50
42

2.1.4. Evaluation of students - computer based testing
Students who were in the group “B” were allocated for the computer based testing [5], [23] and
the results given in table 4 have been obtained. When evaluating these outputs initially the
examiner has checked whether the students have received the correct outputs or not and the
coding was evaluated only for the students who couldn’t complete it within the given time [24].
The statistics of the number of students who did not score full marks from computer based testing
is shown in the Table 4.
Table 4. Summary of Number of students who received and who did not score full marks from computer
based testing

No

Types of Mistakes

1
2
3
4
5
6
7
8
9
10

Semi Colon
Colon
Period
Single Quotation
Double Quotation
Comma
Simple Brackets
Square Brackets
Methodology
Final out put

Number of students who
received marks
42
45
43
42
47
46
48
39
41
38

Number of students who
did not score full marks
8
5
7
8
3
4
2
11
9
12

According to the above statistics only thirty eight students have done the coding excellently with
the final output and obtained full marks. The percentage of those getting full marks is 76%.
The number of students who have repeatedly done mistakes was also less due to facilities for
debugging while they were developing programmes. The system itself has shown to them the
syntax errors. There were students who could not complete the programme during the allocated
time but their performance is better than for paper based testing. The statistics of the number of
students who repeatedly did mistakes for computer based testing is shown in the Table 5.
65

Table 5. Number of students who repeatedly did mistakes for computer based testing

1
2
3
4
5

Number of Types of Mistakes
One Mistake Type
Two Mistake Types
Three Mistake Types
Four Mistake Types
More than Four Mistake Types

Number of Students
6
3
1
2
0

After doing the above computer based testing [6] the students who were in the group “B” filled a
questionnaire to express their own attitudes towards this type of computer based testing. The
statistics are shown in Table 6.
Table 6. An Analysis of feedback given by students who answered the questionnaire for computer based
testing

No

Reason

Number of Students

1

Need to memorise the concepts and unnecessary theory

Yes
10

No
40

2
3
4
5
6
7
8
9

Reduce stress
More confident about answers
Answer is visible
Working in front of machine is comfortable
Interface provides the extra support
Can debug time to time to make it correct
More freedom to come up with alternative methods
Easy to score maximum marks

42
35
50
50
45
50
50
46

8
15
0
0
0
0
0
4

2.1.5. Feedback from the lecturer’s who evaluated paper based testing and computer based
testing
The twenty lecturers who evaluated paper based examinations have given their feedback
according to the questionnaire. The statistics are shown in the Table 7.

66

Table 7. An Analysis of feedback given by lecturers who answered the questionnaire for computer based
testing

No

1
2
3
4
5
6
7
8
9
10

Reason

Number of
Lecturer

Reduce the stress, over head and ambiguity
Reduce Paper marking time
Increase the accuracy of marking
More effective and efficient
No need to read line by line to check syntax for mistakes
Reduce the unpleasant situation of reading untidy handwriting
Increase the transparency of marking
100% certainty about the correctness of the coding and It gives working
outputs
Ability to use different techniques or methods by their own need not
stick to one method
Debugging method can be used to evaluate the incomplete programmes.

Yes
20
15
20
20
20
20
20
20

No
0
5
0
0
0
0
0
0

20

0

18

2

2.1.6. Feedback from Software Developers

The selected fifty software developers went through the training and really faced the pre-test and
post test as a requirement of on the job training to move to the new project which is totally based
on different languages or to obtain the next increment level [19],[21]. The statistics are given in
the table 8.
Table 8. Feedback from the software developers in the Industry

Type of test

No of
Employees

No of
Employees (%)

No of employees who preferred computer based testing to 43
evaluate themself after the training session

76

No of employees who does not prefer computer based
testing to evaluate themself after the training session

07

14

No of employees who does not response for the 05
Questionnaire

10

Software developers and the IT industry used this practical based method especially for the junior
employees or those undergoing training programmes to enhance their knowledge. With the
practical knowledge and the experience they can face this type of evaluation criteria more
confidently [24].

67


3. Hypotheses testing
3.1. Comparison between Paper based Testing and Computer based Testing (Using two
sample proportion tests)
Table 8. Result of two sample proportion tests

Paper based Testing
Computer based Testing
Total

Received Marks
168
431
599

Lost Marks
332
69
401

Total
500
500
1000

Therefore we can conclude that; the true proportion of students who received marks from
computer based testing is higher than the true proportion of students who received marks from
paper based testing.
Ho:
The true proportion of students who received marks from paper based testing and
Computer base testing is the same
H1:
The true proportion of students who received marks from computer based testing is
higher than the true proportion of students who received marks from paper based testing.
Test statistic:

Since the p value (
level of significance.

) is extremely small we have enough evidence to reject Ho at any

3.2. Comparison between the Computer based and Paper based questions among students
(Using two sample proportion test)
Table 9. Comparison between the Computer based and Paper based questionnaires

Type of test
Paper based testing
Computer based testing

Yes
175
378

No
265
67

Total
440
445

68


Ho:
The true proportions of students who prefer computer based questions are the same as the
true proportions of students who prefer paper based questions
H1:
The true proportions of students who prefer computer based questions are higher than the
true proportions of students who prefer paper based questions
Test statistic:

Since the p value (
at any level of significance.

is extremely small we have enough evidence to reject

Therefore we can conclude that; the true proportions of students who prefer computer based
questions are higher than the true proportions of students who prefer paper based questions.
3.3. Evaluation of the questionnaire feedback from lecturers for Computer based Testing
Let

be the true proportion of lecturers who prefer computer based questions

Test statistic:

Since the p value (0.0000) is extremely small we have enough evidence to reject Ho at any level
of significance. Therefore we can conclude that the true proportions of lecturers who prefer
computer based testing are higher than 0.5 and hence most of them prefer computer based testing.
3.4. Evaluation of the questionnaire feedback from software developers in the industry for
computer based Testing
According to the selected software developers 76% were preferred for the computer based testing
and 14% did not prefer and 10% did not respond. Therefore the people like software developers
whose job is really writing programmes also preferred to have computer based testing than a
paper for pre or post test.

69


Figure 3. Feedback Pie chart from software developers

4. Conclusion and Discussion
This section of the study highlights the key findings and is discussed in relation to the objectives
defined at the prior stages of this research. The main aim of the discussion is to bring to light new
findings and value additions to the area of testing for the higher education sector especially in
computer related subjects to conduct the computer based testing than the paper based testing. It
would be beneficial to the students and the lecturers.
According to the students’ questionnaire 85% said that computer based testing is much better and
they have proved it is more successful than the paper based testing due to the following
advantages;
No need to by-heart concepts and unnecessary theory, Reduce stress, Build more confidence
about the answers, The answer is visible, Working in front of computer is more comfortable,
Interface provides the extra support, Ability to debug time to time to make it correct, More
freedom to come up with alternative methods and Easy to score maximum marks.
The data was also analysed according to the given questionnaire to summarize the results from
the lecturers’ point of view. Ten key important factors have been identified for conducting
computer based testing;
Reduce the stress, Over-head and ambiguity, Reduce marking time, increase the accuracy of
marking, It is a more efficient and effective method, Does not need to read line by line to check
syntax for mistakes, Reduces the unpleasant situation of reading untidy handwriting, Increases the
transparency of marking, Hundred percent of certainty about the correctness of the coding,
Production of a working output, Ability to use different techniques or methods on their own, No
need to stick to one method and debugging method can be used to test the logical errors in the
programs.
According to the survey conducted the software developers from the industry also preferred to
have a testing method for their in-house on the job training programmes.

70


REFERENCES
[1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14]
[15]
[16]
[17]
[18]
[19]
[20]
[21]
[22]
[23]

[24]

Anderson D., Black D.,Brown S., Day A. , McDowel S.l, McDowell L.,Peters L.& Smith B. 1999, p.
125.
Computer-based testing: a comparison of computer-based and paper-and-pencil assessment. [Online].
Available: 12.03.2013
http://www.thefreelibrary.com/Computer-based+testing%3a+a+comparison+of+computerbased+and...-a0241861838
170
139).
Computer
Based
Testing.
[Online].
Available:
12.03.2013
http://www.nremt.org/nremt/about/CBT_home.asp
What are important aspects of converting an exam to computer-based testing? [Online]. Available:
15.03.2013.
http://www.proftesting.com/test_topics/cbt.php
Roy Clariana and Partricia Wallace: Paper based versus computer based assessment: Vol 33 No. 5
2002 593-602” British Journal of Educational Technology
Heather F., Steve K. & Stephanie M. 1999, A Hand book for Teaching and Learning in Higher
Education, Kogan Page Limited .p. 134
113
Badders, W 2000, Methods of Assessment, viewed 3 August 2012,
<http://www.eduplace.com/science/profdev/articles/badders.html>.
Habeshaw S.,Gibbs G.,Habeshaw T. 1986, p. 57, 53 Interesting ways to assess your students,
Newble, Cannon R. & David 2000, p. 202, A Hand book for Teachers in Universities and Colleges
Honey P & Mumford A 1992, The Manual of Learning Styles, 3rd edn, Maidenhead: pete Honey.,
Viewed 15 May 2012 www.studyskills.soton.ac.uk/studyguides/Learning%20styles.doc>
Anderson D., Black D.,Brown S.,Day A. McDowel S.I.McDowell L.,Peters L & Smith B.1999, P.4
A Hand book for Teachers in Universities and colleges (Newble, Cannon R & David 2000, pp. 173205)
Littlejohn, A 1993, ‘Teaching school aged learners : a framework for understanding and some
indicators for change’.
R.D. Dowsing and S. Long,(1997),"The Do's and Don'ts of Computerising IT Skills
Assessment",University of East Anglia, NORWICH NR4 7TJ, UK
Helen Drury, (1997),"Integrating the Teaching of Writing Skills into Subject Area IT Programs",
Learning Assistance Centre, The University of Sydney
Michael J. Hannafin, (1997), “Educational Technology Research & Development (1997), 45(3), 101117., University of Georgia,Athens, GA USA 30602
Mary T. Rice , (1997),“Finding out What Works: Learning from Evaluations of Information
Technologies in Teaching and Learning”, Deakin Centre for Academic Development, Deakin
University
Lynette Schaverien and Mark Cosgrove, (1997), “Computer Based Learning Environments in Teacher
Education: Helping Students to think Accurately, Boldly and Critically”.

Authors
Samantha Mathara Arachchi, is a Lecturer at the Department of Communication and
Media Technology (CMT), University of Colombo School of Computing (UCSC), Sri
Lanka, who obtained his degree in Bachelor of Science in 1999 from the Open University
of Sri Lanka. He holds a Postgraduate Diploma in Computer Technology in 2002 from
the University of Colombo and a Postgraduate Diploma in Information Management in
2007 from Sri Lanka Institute of Information Technology, and also holds a Masters in Information
Management (2008) with merit pass. He is reading for PhD in Malaysia in Computer Science.
71

Dr. Kennedy D Gunawardana is a Professor in the Accounting Depatment at University
of Sri Jayawardenapura, He teaches Cost and Management Accounting, Accounting
Information Systems, and Artificial Neural Networks for Accountig for the
undergraduates and Post graduates students.With over 50 articles published in refereed
Conferences procedings, numerous other papers and monographs, 2 chapters of the
international books and 3 English Medium text books and 3 Sinhala medium edited books.
R.S. Madanayake is a Lecturer attached to the University of Colombo School of
Computing, Sri Lanka. He obtained his B.Sc. from the University of Colombo, Sri
Lanka in 2001. He obtained his M.Sc. in IT from the University of Colombo, Sri Lanka
in 2010. He is currently reading for his M.Phil. at the University of Colombo, Sri Lanka.
G.K.A. Dias is a Senior Lecturer at the University of Colombo School of Computing
and is currently the head of its Department of Communication and Media Technologies.
He obtained his B.Sc. from the University of Colombo in 1982. He obtained a Post
Graduate Diploma in Computer Studies from the University of Essex, U.K. in 1986. He
obtained an MPhil by research from the University of Wales, U.K. in 1995.

72

A comparison between evaluation of

More Related Content

What's hot (20)

Viewers also liked (10)

Similar to A comparison between evaluation of (20)

Recently uploaded (20)

A comparison between evaluation of