Yoshida thesis

1University of Aizu, Graduation Thesis. March, 2012 s1170173
Abstract
There is scant literature in language studies
suggesting the efficiency with which EFL learners
with low English language proficiency are able to
process English website content, and their opinions
on the same. Structured website analysis has the
potential to be a stimulating exercise because it
addresses various types of questions that are very
unique to the medium and might not be immediately
applicable to other contexts. This article reported on
an experiment where 41 junior-level Japanese EFL
readers were asked to study an English tourism
website closely, and then answer a set of questions
which focused entirely on readers’ ability to excavate
information by navigating through the website
content. Further, several survey questionnaires (QUIS,
CSUQ, and MPRC) were used wherein readers were
asked to self report their levels of comfort with the
website, and the words they would choose to describe
their feeling about the website. Results suggested
relatively higher levels of proficiency ordering the
steps required to navigate and search for specific
information from the given website. Moreover,
readers were mostly comfortable searching for
information from all over the website. Self-reports
suggested relatively moderate levels of comfort with
different tasks and access features (overall reaction to
the website; webpage design, terminology and
website information; learning; website capabilities)
related to the website. Finally, data shows that the
number of positive words chosen to describe the
tourism website is way more than the negative words
chosen.
1. Introduction
Readers of a traditional printed text read in order
of ascendency. However, website readers must
choose between hyperlinks which could possibly take
them to different related or unrelated sections of the
broader webpage, and often to other web pages as
well. Carrell (1987) suggests that organization of a
text affects reading comprehension as well as recall.
She said that signaling devices in the text help
readers recall the information in the hierarchical
structure of the text and improves text comprehension.
The signaling devices help connect one part of the
text to another and improve content comprehension
(Kintsch & van Dij, 1978). However, such
conclusions on reading ability and comprehension
might be completely lost when reading hypertext-rich
websites are accessed. In this context the hierarchical
structure of the text is completely lost because of the
skipping around to different parts of the text that is
involved (Charney, 1994).
The efficiency in reading depends on readers’
schema which helps detect the structure of a text. The
readers’ schema functions as an abstract script and
the reader expects the script to match the text he/she
reads (de Beaugrande, 1980). The readers’ schema is
often influenced by the website’s graphic
manifestations that lead to a shift from the centrality
of text (Burbules, 1998). However, the author
suggested that image does not replace the need for
text. Rather, the idea behind use of graphics is to
summarize the importance of the message that text
communicated, but in a quick glance. Graphics tend
to serve a pointing function. Boardman (2004)
suggested that webpage creators often choose short,
dense phrases, rather than full sentences to
summarize the core of the message.
However, we are not entirely sure of how this
experience translates to an EFL context where
readers’ English proficiency is at a very low level.
On a very specific level the question is the efficiency
with which EFL learners with low English language
proficiency are able to process English website
content, and what is their opinion about the website?
Structured website analysis with open-ended
feedback as a tool for foreign language learning has
the potential to be a stimulating exercise because it
addresses various types of questions that are very
unique to the medium and might not be immediately
applicable to other contexts. Very little is known
about user’s ability to provide open-ended feedback
on website content (Elling et al., 2012). Moreover, in
an EFL context, website analysis might deal with a
range of issues related to learning difficulties
resulting from lack of language proficiency, variable
EFL Student’s Ability for Website Information Comprehension and
Perceptions on Website Usability
Makoto Yoshida s1170173 Supervised by Prof. Debopriyo Roy

use of cognitive and meta-cognitive strategies for
information comprehension on the web (Lomicka,
1998) or in an intensive English learning context
(Hong-Nam and Leavell, 2006), individual
differences in learning with hypermedia (Knight,
1994), besides complexity in the web content itself.
These above-mentioned areas of research directly or
indirectly contribute to our general understanding
about how L2 learners perceive English website
content and design, and how specific reading
strategies (self-reported) might impact design
decisions.
As part of the experiment reported in this article,
Japanese EFL readers were asked to study a website
closely, answer design questions related to the
website and how it is structured, and then answer a
set of questions which focused entirely on the content
of the website. The purpose behind this questionnaire
was to judge whether readers actually have read
through the website, and could identify where the
information is placed in the website. Answering the
questions in this questionnaire successfully
demanded both reading ability and identifying links
that takes the reader to the required information.
Further, readers were asked to self report what they
thought specifically about the website with different
questionnaires, and the words they would choose to
describe their feeling of the website. This paper is
solely based on exploring readers’ ability to identify
and read website content successfully, and self-
reports of their feelings of the website. The purpose
behind this activity was to identify what bothered
them about the website information comprehension,
what they liked about the navigation, their levels of
motivation and feelings about the website etc. This
will allow L2 researchers to get some sense of how
EFL readers with low language proficiency approach
the task of reading an English website.
2. Method
Participants (N=17) for the first stage of the
analysis (Week # 1) are junior level students (age
group: 18-20 years) in their third year undergraduate
program specializing in computer science in a
Japanese technical university.
First Week: During the 1st week of the actual
experiment, a randomly selected section of the
students analyzed the Belize tourism website based on
the 8 open-ended design questions about the website.
However, the analysis of how efficiently students
could answer the design questions is beyond the
scope of this study.
Second Week: During the 2nd
week, readers (all
students in the class - N = 59) were given a set of
questionnaires to answer on the same Belize tourism
website. The websites focused on three different
aspects related to information comprehension and
online reading strategies for an EFL context. The
questionnaires focused on three different aspects.
• Their ability to navigate through and look for
information from the website.
• Their ability to meaningfully self-report on the
usability of the website in terms of navigation,
content, and organization.
• Their ability to meaningfully self-report on their
reading strategies when analyzing the Belize tourism
website.
All the questionnaires with the above-mentioned
focus were completed in class.
Instruments:
The Belize tourism website was chosen with the
following reasonable conditions in mind.
• The content in the website is not text heavy and
clear navigation is possible.
• Information could be searched directly from the
home page.
• Attractive pictures are available to keep the reader
engaged in the task of finding information.
• Japanese version of the webpage is NOT available
so that readers are forced to look for information
from the English version alone.
The instructions for the first week of the assignment
(where readers had to respond to 8 open-ended
questions) were all in English, largely because readers
already had practice from the weeks before regarding
what is expected of them. For the second part of the
experiment (held during the 2nd
week), all the
questionnaires were administered in Japanese, to
ensure proper understanding of the content and
instructions. Readers were handed out an instruction
sheet in both Japanese and English explaining what is
exactly expected of them for the task. The second part
of the experiment comprised of several tasks.
- 1st
Task: To check for readers’ ability to navigate
through and look for information from the Belize
tourism website, a website information comprehension
questionnaire was designed specific to the Belize
tourism website. The questionnaire had to answer 8
questions. For the 1st
question, readers were given a
specific scenario where they were asked to find out a

hotel in a specific location in a given price range.
Readers were given a list of steps they had to follow in
order to find the information from the Belize tourism
website. For all other questions except one, they had to
choose the correct option regarding information that is
available in the website. They can only choose the
correct option when they have actually tried to look for
information from the website. This should not be
called a questionnaire in the true sense, because the
questions were geared towards searching for
information from the website and were completely
unrelated from each other, with no pattern or similarity
or were not part of any specific sub-group of analysis.
- 2nd
Task: A set of three usability questionnaires was
handed out as part of this task. The questionnaires are
named as QUIS, CSUQ, and Microsoft Product
Reaction Card respectively. The first two
questionnaires used a Likert scale for each question.
The review of the literature has more details about
these questionnaires. These questionnaires are
designed not only to elicit readers’ impression of the
Belize tourism website in terms of content, navigation
and usability, but also judge the extent to which their
impression on items matches or deviates from the
judgement made by the more experienced coders.
Assessing Website Usability:
According to the IBM technical report, most usability
evaluations gather both subjective and objective
quantitative data in the context of realistic scenarios-
of-use, as well as descriptions of the problems
representative participants have trying to complete the
scenarios. Subjective data are measures of
participants' opinions or attitudes concerning their
perception of usability. Objective data are measures of
participants' performance (such as scenario completion
time and successful scenario completion rate) (Lewis,
1993). Usability is recognized as an important quality
factor of any modern website (Avouris et al., 2003).
Literature on web usability has reported the use of
variety of questionnaires for reporting perceived
usability of interactive systems (Tullis & Stetson,
2004). As part of this study, several questionnaires
were used, based on the study performed by Tullis and
Stetson (2004) on comparison of questionnaires for
assessing website usability.
QUIS (Questionnaire for User Interface Satisfaction)
was developed at the University of Maryland, and
composed of 27 questions. However, for website
context, we adopted 24 questions that are appropriate
(http://www.isr.umd.edu/news/news_story.php?id=409
9). This questionnaire was used for obtaining self-
reported impression of the website on five categories
of information. They are (1) Overall Comprehensive
Evaluation of the website (2) Website design (3)
Terminology and website information (4) Learning
from the website and (5) Website capabilities.
CSUQ (Computer System Usability Questionnaire)
also used for this study was developed at IBM and was
originally developed for computer systems. However,
it was adopted for websites in this context
(http://drjim.0catch.com/usabqtr.pdf). The purpose of
this website is to get self-reports on overall ease of
using the website, and related efficiency of use.
Finally, the Microsoft Product Reaction Card was
used for this study, to obtain qualitative reviews on the
impression about the website. A modified list, based
on originally developed 121 words, was presented to
the user and was asked to choose the words that best
represented their interaction with a website. Readers
were free to choose any many or as few words as they
deemed appropriate (Benedek and Miner, 2002). The
above questionnaires were used for the study as
reported in this article.
For the Website Information Comprehension
Questionnaire, QUIS questionnaire, CSUQ and
Microsoft Product Reaction Card: The three coders
were independently asked to grade the questionnaires
that readers completed during the second week
analysis and reporting.
For the website information comprehension
questionnaire, grading was straightforward in terms of
whether the reader could correctly identify the correct
option after searching for information from the website.
However, for each question, I asked the coders to
consult amongst themselves as to whether there is
some information that is truly difficult to locate, or if
there are alternate ways of finding the same
information. However, for each reader, and for each
question there was only one grade assigned (2=correct
or 0= incorrect). The project coordinator offered
coders with expert advice as to how they can search
for answers to the questions asked in the questionnaire,
and then grades the responses for accuracy. For this
questionnaire, each response was graded only once by
three coders who worked together to decide on a grade.
For the QUIS, CSUQ and Microsoft Product Reaction
Card questionnaires, coders recorded the self-reports
by the readers coming together as a group. This group
activity was important to make sure that no wrong data
was entered into the system during data collection.

3. Findings
3.1. Website Information Comprehension
Questionnaire
The website information comprehension
questionnaire showed relatively moderate levels of
accuracy, indicating the fact that text comprehension
from the web pages, understanding the navigation
styles of the Belize website, understanding the linked
pages, etc. were moderately successful. However,
since the questions in this questionnaire were
designed to test whether readers are actually capable
of searching through the information in the website,
the different questions in the questionnaire can’t be
considered to be testing anything specific. Rather,
different questions tested searching ability for
different types of information from the website, and
in no order of complexity or similarity of content.
This was evident from the reliability test done for the
questionnaire, with a Chronbach’s alpha value
of .086 (from Q2 ~ Q9). So, it probably won’t make
sense to term this as a questionnaire. Instead the
purpose would be to see the accuracy with which
individual questions were answered and relate it to
the self-reports in other questionnaires.
For most questions, students scored in the range of
1.19 ~ 1.97 showing some variability in the scores.
This indicates that some questions were relatively
easier to answer, while some were more difficult
depending on how much the information had to be
searched from the linked pages in the website. Table
1 shows the details of the website information
comprehension data. On an average, for questions 2 ~
8, the data shows accuracy of searching for
information from the website at around 83%. For the
first question which asked readers to sequence steps
when searching for specific information from the
English website, the accuracy could be pegged at
86.6%. Table 1 shows the mean accuracy for the
different questions asked in the website information
comprehension questionnaire. The mean accuracy
scores from Q2 ~ Q8, shows high values for Q2 and
Q3, but a drop in the mean score for Q3, and Q4, and
more overall fluctuations.
Data shows significant correlation vales between
accuracy scores for the different questions asked.
Results indicate statistically significant correlation
values involving Q2, Q7, and Q8. This suggests
comparable amounts of accuracy between the
questions, indicating searching for some kinds of
information and/or levels of difficulty when
searching for information from the website were
comparable for the questions concerned.
We also wanted to test if there is any statistically
significant difference between the accuracy scores in
the website information comprehension questionnaire.
Results of the Friedman test suggested that responses
are indeed significantly different (χ2
(2) = 78.120, P =
0.000). Since p-value = 0.00 ≤ 0.01 = α, we rejected
the null hypothesis that there is no difference between
the mean ranks for the accuracy scores.
Following the website information comprehension
questionnaire, participants self-reported their
impression of the software on different categories
using the CSUQ questionnaire.
3.2. CSUQ and QUIS Questionnaire Results
The purpose of the CSUQ Questionnaire was to
explore self-reported ease of using the Belize tourism
website. Results show a significant correlation in
most cases between all questions asked in the
questionnaire, with a handful of exceptions. QUIS
questionnaire had 5 different categories namely (1)
overall reaction to the website (2) screen (3)
terminology and website information (4) learning and
(5) website capabilities. Table 5 shows the mean and
standard deviation values for all 5 categories in the
QUIS questionnaire. For this questionnaire, in this
case of reliability testing, a Chronbach’s alpha
of .687 was obtained for category 1, .809 for category
2, .637 for category 3, .785 for category 4 and .731
for category 5. Data also reported the percentage
levels of agreement with the different categories in
the QUIS questionnaire. 100% agreement shows a
mean value of 5 on a category in a 1 ~ 5 Likert scale
whereas a mean value of 1 shows 0% agreement with
a statement. Results here suggest 59.4% agreement
on the” overall reaction to the website” category,
which goes up to 64.8 for “web page design”
category, but then slides back to around the 44 ~ 54%
level for the other three categories. Data also show
the number of cases in various categories where
responses to questions show significant correlation
with responses to other questions either in the same
category or in a different category. Except for
question 2, 23, and 24, all questions demonstrated
significant levels of correlation with all other
questions across the questionnaire. This result
suggests significant similarity in what website readers
think about the different features of the website, ease
of learning from the website, ease of reading and
using the website interface, navigating through the
website, website accessibility and so on.

Next, we wanted to explore if there is significant
similarity or difference between the self-reported
scores on the Likert scale in the CSUQ questionnaire.
Null Hypothesis: Mean Ranks for all the self-reported
scores on the CSUQ questionnaire are equal;
Alternative Hypothesis: Not all the mean ranks are
equal. Table 7 shows us a Friedman analysis for
responses to all the questions combined.
The Friedman non-parametric statistical analysis
was performed to find out if there is an overall
statistically significant difference between the mean
ranks of the self-reported scores in the different
questions asked. This test tells us whether there are
overall differences between self-reported scores but
does not pinpoint which questions in particular differ
from each other. To do this we need to run post-
hoc tests, but post-hoc analysis was not considered as
part of this study. Results suggest that there was a
statistically significant difference in self-reported
sores for a combination of all the questions asked in
the CSUQ questionnaire, depending on the overall
weight of how readers felt about accessing the
English website, χ2
(2) = 126.626, P = 0.000. Since p-
value = 0.00 ≤ 0.01 = α, we reject the null hypothesis.
Next, data explored if there is significant similarity
or difference between the self-reported scores on the
Likert scale in the QUIS questionnaire for the
combined responses on the 5 different categories
studied. Friedman analysis was done for each of the 5
categories in the QUIS questionnaire and was
considered separately. This test tells us whether there
are overall differences between self-reported scores
within a category, but does not pinpoint which
questions in particular differ from each other. To do
this we need to run post-hoc tests, but post-hoc
analysis was not considered as part of this study.
Results suggest that there was a statistically
significant difference in self-reported sores for a
combination of all the questions asked in the “overall
impression of the website” (χ2
(2) = 21.532, P =
0.000), “web page design” (χ2
(2) = 24.818, P =
0.000), “terminology and website information” (χ2
(2)
= 60.670, P = 0.000) and “website capabilities” (χ2
(2) = 75.683, P = 0.000) categories. However, data
shows there was no significant difference between
responses when it comes the “learning” (χ2
(2) =
5.769, P = .217) category. Since p-value = 0.00 ≤
0.01 = α, we reject the null hypothesis.
3.3. Microsoft Product Reaction Card Results
For the MPRC questionnaire that was handed out
to participants in the final stage, following interaction
with QUIS and CSUQ questionnaire, we obtained the
following results. In total, 116 words were selected
by the readers (out of 121). Readers chose 77 positive
words and 41 negative words. Also, the total
frequency of positive words chosen was way more
than the total frequency of negative words chosen.
For the 10 most frequently chosen words, 8 of these
words are positive ones and 2 of these words has a
negative sense.
4. Discussion
Question 1 in the website information
comprehension questionnaire intended to explore the
efficiency with which readers are able to follow and
confirm the sequence of activities that is necessary
for finding certain information. A score of 8.66 in a
scale of 10 for all correct answers, suggest high levels
of efficiency in following the steps correctly from an
English webpage. Also, scores on other questions
(related to searching for information from Belize
website) in this questionnaire suggest moderate mean
scores. However, there is probably hardly any
literature to suggest that ability for search in an
English context, from an English website might also
indicate higher levels of English content
comprehension ability.
Further, not a significant number of correlations
were observed between accuracy scores on the
website information comprehension questionnaire
and self-reported scores on the CSUQ and QUIS
questionnaire. This goes on to suggest that self-
reports on the questionnaires had a general outlook
about the website and might not have been specific to
the accuracy scores. But, the Friedman test of
accuracy scores did suggest a significant variation
between the results of accuracy scores. This goes on
to show that in significant number of cases, some
readers could and some could not search through the
required information and locate the answer in this
multiple-choice website information comprehension
questionnaire. This possibly indicates towards the
fact that individuals with high reading and content
comprehension ability might have a higher statistical
probability to search for and locate answers from
English website more successfully.
The CSUQ questionnaire made positive statements
in different categories and asked readers to respond to
what extent they agree with the statement. Self-
reported scores in the range of 3 ~ 4.3 suggest
moderate agreement with the statements. This also

indicates towards the fact that readers were not highly
comfortable with the website as for its features,
design, etc. Further, we see a very high number of
significant correlations between the self-reported
scores in the CSUQ questionnaire, suggesting similar
range of responses for most questions. However,
Friedman test suggests a different result because this
non-parametric analysis considered the mean rank for
each question, and all the results were considered
together, and not in a bivariate context.
Finally, we could clearly see that the number of
positive words chosen to describe the Belize tourism
website is way more than the negative words chosen,
and also the positive words are chosen with more
total frequency. This goes on to show a positive
overall impression of the website. However, this
should just be taken as an indicator and whether it
translates to overall better comprehension of the
website information, and general learning is a
different matter altogether.
5. Conclusion
From this study we see conclusive evidence that
there is wide variability in the efficiency with which
readers in this EFL context are able to analyze an
English website. Future studies could systematically
focus on testing accuracy related to different aspects
of searching and reading information from English
websites. These aspects could be directly related to
demonstrating efficiency in understanding website
design, website capabilities / features, learning aspect
of the website, terminology and website information
etc. Further, these accuracy scores would then be
matched to self-reported scores on the CSUQ and
QUIS questionnaire. This exploratory analysis as
discussed in this article could help us obtain initial
data on how EFL readers in a typical context as this
would perform with English website information and
the type of impression they have about the website.
This will help us judge their levels of proficiency and
the types of English websites they could be exposed
to for various kinds of assignments, task-based
language learning etc.
6. References
Carrell, P.L. (1987). Content and formal
schemata in ESL reading. TESOL Quarterly, 21,
461-481. 1987 TESOL International
Association
Van Dijk, T. A., & Kintsch, W. (1983).
Strategies of discourse comprehension. New
York: Academic Press.
Charney, D. (1994). The effect of hypertext on
processes of reading and writing. New York:
Modern Language Association.
de Beaugrande, 1980 Text, Discourse, and
Process: Toward a Multidisciplinary Science of
Texts
Nicholas C. Burbules. "Rhetorics of the Web:
Hyperreading and critical literacy" in Page to
Screen: Taking Literacy into the Electronic Era,
I. Snyder (editor), Routledge, London, 1998.
Boardman, M. (2004). The Language of
Websites (Intertext). New York: Routledge
Elling et al., (2012). Users’ Abilities to Review
Web Site Pages. Journal of Business and
Technical Communication Volume 26 Number
2 (April 2012)
Lomicka, L. (1998). "To gloss or not to gloss":
An investigation of reading comprehension
online. Language Learning & Technology, 1(2),
41-50. Retrieved May 1, 2005, from
Lomicka, L. (1998). "To gloss or not to gloss":
An investigation of reading comprehension
online. Language Learning & Technology, 1(2),
41-50. Retrieved May 1, 2005, from
http://llt.msu.edu/vol1num2/article2/default.ht
ml.
Hong-Nam, K., & Leavell, A. G. (2006).
Language learning strategies of ESL students in
an intensive English learning context.System,
34(3), 399–415.
University of North Texas, Denton: Department
of Teacher Education and Administration
Knight, J. (1994). Internationalization:
Elements and checkpoints (Research
Monograph, No. 7). Ottawa, Canada: Canadian
Bureau for International Education.

Yoshida thesis

More Related Content

Viewers also liked (20)

Similar to Yoshida thesis (20)

More from Debopriyo Roy (20)

Recently uploaded (20)

Yoshida thesis