Analysing The Data

129
© The Author(s) 2017
K. Dikilitaş, C. Griffiths, Developing Language Teacher Autonomy
through Action Research, DOI 10.1007/978-3-319-50739-2_6
6
Analysing the Data
Let us suppose that some interesting-looking data has been collected, and
now we want to know what to do with it. Let us be quite clear that we
do not actually have to do anything with it unless we want to. Maybe we
are quite happy just to have performed the collection exercise, perhaps we
have gained some insight into whatever it was that was puzzling us, and
that is as far as we want to go.
If, however, we want to go beyond this point, analysing the data can
be quite an intimidating prospect. So, let us have a look at it, and see if
we can break it down into manageable pieces which will produce robust
results.
Types of Data: Quantitative
Since there is often confusion with these basic concepts, and they are
not always easy for non-mathematicians to understand, the most impor-
tant from an action research point of view will be explained here. This
is important, since inappropriate analysis renders what might otherwise
have been interesting findings quite invalid.

130
1. Numerical data. This term refers to data which are actually real num-
bers. Examples are:
(a) Age—we can assume that someone who is 20 is twice as old as
someone who is 10.
(b) Income—someone earning $15,000 is earning half as much as
someone earning $30,000.
(c) Test scores—a student who scores 90 has scored 50% more than
a student who scores 60.
This kind of data is sometimes also called interval (because the val-
ues have regular intervals between them) or continuous (operating over
a range). Numerical data can be analysed using parametric tests (which
operate within set parameters, e.g. means, Pearson’s correlation, t-tests,
ANalysis Of VAriance (ANOVAs), Multivariate ANalysis Of VAriance
(MANOVAs), multiple regression) as long as they are normally distrib-
uted (see later section).
2. Ordinal data. This term is used to refer to data where particular atti-
tudes (such as level of agreement or disagreement) or subjective assess-
ment of a particular phenomenon (e.g. degree of frequency, level of
importance) are given numbers for the sake of convenience in that this
allows the data to be analysed by computer. Clearly, however, these fig-
ures are not really numbers (they are non-numerical). For example:
(a) We cannot say that someone who gives a questionnaire item a rat-
ing of 4 for agreement is twice as much in agreement as someone
who gives it a 2. In other words, we cannot assume that the inter-
val between the ratings is equal.
(b) It is clearly nonsense to say that the average of agree (4) and disagree
(2) is agree-and-a-half! In other words, this is not a continuous scale.
Likert-type instruments produce non-numerical data, and they should be
analysed using nonparametric tests such as medians, Spearman’s correlation,
and Mann–Whitney U or Kruskal–Wallis H tests of difference. Actually,
these tests are just as easy to use, and often produce results that are similar
to their parametric equivalents, though often slightly weaker. It is therefore
difficult to see why researchers should not use the appropriate tests.
Developing Language Teacher Autonomy through Action Research

131
3. Nominal data. These kinds of data are even further removed from
“real” numbers than ordinal data. They are generated when variables
are divided into categories (hence they are also often called categorical
data) and given numbers, often quite arbitrarily, for the sake of being
able to enter them into a computer programme and analyse them sta-
tistically. For example:
(a) Males = 1, females = 2
(b) Chinese = 1, Europeans = 2, Africans = 3, Americans = 4
Clearly this kind of data is incapable of being analysed numerically.
How could we make any sense of it? If we add males (=1) and females
(=2) together, do we get 3? Would the average of a European and an
American in example (b) above be an African? When considered logically,
this is clearly nonsense. Nominal data can, however, be useful for group-
ing purposes, for instance if we want to find out if our male students
are doing better than our female students, or if students from a particu-
lar national background are doing better or worse than students from
elsewhere. Nonparametric tests of difference (e.g. Mann–Whitney U or
Kruskal–Wallis H) or chi-square (x2
) should be used for this.
How Can I Do These Analyses?
Again, it is up to us. We can get some quite interesting results just
using a simple calculator. If we don’t have too many participants (and
this is typical of action research), it is easy enough to add the scores
together and divide by the number of scores, and there we have the
mean (average).
For those who know how to use Excel, there is a lot more available.
And these days there are a number of extremely user-friendly online plat-
forms such as Google forms, which can easily produce some very interest-
ing results.
But by far the best is SPSS (Statistical Package for Social Sciences).
For those who know how to use Excel, SPSS is not difficult—it operates
along fairly similar lines, but does so much more from a researcher’s point
of view. For those who want to get serious about research, SPSS is worth
the investment of a bit of time to learn how to use it.
6 Analysing the Data

132
Types of Statistical Procedures
In fact, there are many possible procedures, far too many for us to deal
with in depth here. Furthermore, the more complicated procedures often
depend on a knowledge of maths that the ordinary language teacher
rarely has. But let us look at the most common ones an action researcher
is likely to want to deal with:
1. Reliability
2. Factor analysis
3. Normality of distribution
4. Mean
5. Frequencies
6. Percentages
7. Median
8. Mode
9. Correlation
10. Difference
11. Effect size
We will use the small-scale action research study on classroom culture
we have been using in previous chapters to provide examples which we
will work through here together.
• In the case of this study, there were 35 survey sheets (actually, not so
many!), each with nine items, each of which has a rating from 1 to 5.
• In addition, since we want to be able to see if female students perceive
classroom culture differently from their male classmates, we need to go
through and code each one according to gender (male = 1, female = 2).
• Furthermore, we are also interested to know if the local students view
classroom culture differently from international students, so this is also
coded (local = 1, international = 2).
• All of this has to be entered onto the SPSS spreadsheet, under the vari-
able headings GEN (gender), NAT (nationality) and Q1–9 for the
questionnaire items.

133
When this is done, we will have a grid, the first ten rows of which will
look a bit like Fig. 6.1:
Having got this far, we can now begin analysing in earnest! For the sake
of exemplification, we will work through procedures 1–11 listed above.
Reliability
Reliability refers to the extent to which the items in our instrument are
consistent. In order for our instrument to be considered a valid measure
of the concept we are trying to measure, it must be reliable. In order to
analyse for reliability:
• Go up to the task bar at the top, click ANALYZE, go down to SCALE,
go across to RELIABILITY ANALYSIS
• Highlight the items you want to test (in this case Q1–9) and click
them across into the treatment box
• Click OK
GEN NAT Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9
2.00 1.00 4.00 4.00 5.00 4.00 4.00 5.00 4.00 5.00 4.00
1.00 2.00 4.00 5.00 3.00 2.00 5.00 4.00 1.00 2.00 3.00
1.00 2.00 4.00 2.00 3.00 4.00 3.00 1.00 2.00 4.00 1.00
1.00 2.00 2.00 2.00 4.00 5.00 4.00 1.00 4.00 5.00 1.00
2.00 1.00 2.00 4.00 4.00 5.00 4.00 4.00 4.00 5.00 4.00
1.00 1.00 2.00 2.00 4.00 1.00 2.00 2.00 3.00 3.00 2.00
2.00 1.00 3.00 4.00 4.00 3.00 4.00 2.00 4.00 5.00 4.00
2.00 1.00 4.00 4.00 4.00 3.00 2.00 3.00 5.00 5.00 3.00
1.00 2.00 4.00 3.00 4.00 4.00 4.00 3.00 5.00 5.00 4.00
2.00 2.00 4.00 4.00 5.00 3.00 3.00 2.00 5.00 5.00 3.00
Fig. 6.1 Sample SPSS spreadsheet

134
A new screen will come up including the box in Fig. 6.2:
This tells us that the Alpha reliability is 0.776. This is not a high level
of reliability (in the 0.9 range is best), but for such a relatively small
number of participants (N = 35) it is not bad, and anything above 0.7
(or sometimes even in the 0.6 range) is usually considered to be OK (e.g.
Dörnyei 2007). The chances are that if we carried on and gathered more
data from more participants, reliability levels would rise, since, when
numbers are small “outliers” (i.e. participants whose response is consider-
ably different from the majority) have more of an effect than if it were
part of a much larger sample.
If, however, we would really like to get a higher level of reliability from
this study, we might wonder if there may be one item or a group of items
bringing the reliability co-efficient down. In order to check if this is the
case:
• Go back into the main matrix, re-click ANALYZE, go down to
SCALE, across to RELIABILITY ANALYSIS
• Before you click OK, click STATISTICS, then SCALE IF ITEM
DELETED
• Click CONTINUE, then OK
The screen will include the grid in Fig. 6.3:
From the far right-hand column of this grid we can see that there is
no item which, if it were deleted, would raise the reliability substantially,
although if we removed Q6 (about eating and drinking in class), the reli-
ability would go up to 0.791, which would round to 0.80, which actually
sounds much better. So, is this worth doing or not? Perhaps if we try a
factor analysis, we might get some more information to help us make up
our minds.
Reliability Statistics
Cronbach's
Alpha
N of
Items
.776 9
Fig. 6.2 Sample reliability statistics

135
Factor Analysis
A factor analysis is designed to explore whether certain items “hang
together”, and they can be daunting both to conduct and later to inter-
pret. And there are many different ways they can be done in order to
produce different results. But a simple factor analysis does not have to be
so complicated, so let us see how we can perform a factor analysis on the
data obtained for our classroom culture survey.
• Click ANALYZE, down to DIMENSION REDUCTION, across to
FACTOR
• Transfer items Q1–9 to the variable box
• Click OK
The next screen includes the grid in Fig. 6.4:
From this we can see that the programme’s default settings have
divided our questionnaire into three factors. If we look for the high-
est loadings (the largest numbers), we can see that by far the majority
load most highly onto factor 1. Of the rest, there is only one (Q6)
which loads most highly onto factor 2, and only two (Qs 4 & 9)
Item-Total Statistics
Scale Mean
if Item
Deleted
Scale
Variance if
Item Deleted
Corrected
Item-Total
Correlation
Cronbach's
Alpha if Item
Deleted
Q1 28.0000 31.412 .549 .744
Q2 27.5143 30.257 .553 .740
Q3 27.1143 30.457 .616 .734
Q4 28.1714 32.440 .357 .769
Q5 27.8571 30.303 .473 .752
Q6 28.3714 34.534 .198 .791
Q7 27.0000 28.765 .607 .730
Q8 26.7714 29.829 .531 .743
Q9 28.1143 33.575 .288 .778
Fig. 6.3 Grid for reliability if item deleted

136
which load most highly onto factor 3. These do not seem to provide
us with anything very useful by way of grouping the items into any-
thing at all thematically coherent. Indeed, we always saw this small
questionnaire as a coherent whole in itself, designed to measure ideas
of polite classroom behaviour, and we never really intended to divide
it. So perhaps we should see what we get if we try for a one-factor
solution:
• Click ANALYZE, down to DIMENSION REDUCTION, across to
FACTOR
• Transfer items Q1–9 to the variable box
• Before you click OK, click EXTRACTION, then, in the EXTRACT
box, click FIXED NUMBER OF FACTORS, and put 1 in the box
• CONTINUE, OK
You will get a page which includes the grid in Fig. 6.5
From this we can see that all of the items do, in fact, load onto one fac-
tor, although the loading for our slightly troublesome Q6 is the lowest.
Even this, however, is only slightly under 0.3 (often considered a cut-off
Component Matrixa
Component
1 2 3
Q1 .691 .379 -.256
Q2 .670 .538 .216
Q3 .767 -.150 -.350
Q4 .472 -.484 .528
Q5 .617 .313 .237
Q6 .297 .449 -.375
Q7 .765 -.352 -.151
Q8 .695 -.597 -.149
Q9 .369 .213 .639
Extraction method: Principal Component Analysis.
a. 3 components extracted
Fig. 6.4 Sample factor analysis grid (three-factor solution)

137
point), and it would actually round to 0.30 if we chose to work with two
places of decimals.
We are therefore left with the decision as to whether to leave Q6 in
place, and live with slightly lower reliability, or to remove it (the machine
cannot make all our decisions for us). In fact, as the researcher, I had
found myself quite interested in some of the responses to Q6 about eat-
ing or drinking in class, so I felt that this outweighed the possible threat
to reliability posed by a slightly low-loading item.
Normality of Distribution
Another test which should be performed before proceeding with further
tests (e.g. tests of central tendency such as mean, median, mode, or cor-
relation/difference tests) is for normality of distribution. This refers to the
well-known bell curve, where means are more or less evenly distributed
around a central axis. Parametric tests (e.g. Pearson’s product–moment
correlation, t-tests, ANOVAs) operate on an assumption of normality.
If, therefore, the data are not normally distributed, nonparametric tests,
Component Matrixa
Component
1
Q1 .691
Q2 .670
Q3 .767
Q4 .472
Q5 .617
Q6 .297
Q7 .765
Q8 .695
Q9 .369
Extraction method: Principal Component Analysis
a. 1 component extracted
Fig. 6.5 Sample one-factor solution grid

138
which do not assume normality, should be used. The normality of distri-
bution can be determined as follows:
• Click ANALYZE, then DESCRIPTIVES, then EXPLORE
• Move the variables you want to test (in this case, Q1–9) to the depen-
dent list
• Click PLOTS, then NORMALITY PLOTS WITH TESTS
• CONTINUE, OK
A grid that is something like in Fig. 6.6 will appear among others on
the page.
If the Sig. figure for these tests is less than 0.05, the distribution is not
normal. As we can see, the significance figures for both of these tests for
all items are less than 0.05. In other words, none of the distributions are
normal, meaning that we need to use nonparametric tests of correlation
and difference. As we can see, the results for both tests are similar, but, if
we need to choose one or the other, the Kolmogorov–Smirnov is usually
considered the more appropriate one for smaller samples (less than 100).
In fact, this is a very usual result for ordinal data, meaning that non-
parametric tests are by far the most commonly appropriate ones.
Tests of Normality
Kolmogorov-Smirnova
Shapiro-Wilk
Statistic df Sig. Statistic df Sig.
Q1 .271 35 .000 .803 35 .000
Q2 .265 35 .000 .871 35 .001
Q3 .271 35 .000 .793 35 .000
Q4 .205 35 .001 .913 35 .009
Q5 .207 35 .001 .904 35 .005
Q6 .199 35 .001 .913 35 .009
Q7 .303 35 .000 .723 35 .000
Q8 .417 35 .000 .593 35 .000
Q9 .243 35 .000 .889 35 .002
a. Lilliefors Significance Correction
Fig. 6.6 Sample tests for normality of distribution

139
Mean
Statistics such as the mean, median, mode, frequencies, and percentages
are called descriptive statistics, since they are used to describe the results
obtained from a study of a particular sample. The mean actually applies
to numerical data which are normally distributed. Since we have ordinal
data, and we have already found the distribution is not normal, means are
not appropriate for this study. We could, perhaps, have been interested in
a mean if we had included test scores (which are numerical) in our data
collection (and we had found them to be normally distributed). Since
this is actually a very common thing to do, for the record:
• Go to ANALYZE, go down to DESCRIPTIVE STATISTICS, go
across to DESCRIPTIVES
• Transfer the item/s you want to analyse to the VARIABLE box
• Click OK
The programme will produce a box with the ITEMS, the MINIMUM
andMAXIMUMvalues,theMEAN,andtheSTANDARDDEVIATION
(which indicates the average distance of the scores from the mean).
Frequencies
If we want to know how often a particular value occurs, SPSS will calcu-
late frequencies for us:
• Go to ANALYZE, go down to DESCRIPTIVE STATISTICS, go
across to FREQUENCIES
• Move target items (in this case, Q1–9) to VARIABLE box
• Click OK
A page will appear which includes a matrix for each of the items
(Fig. 6.7).
From the “Frequency” column of this matrix we can see that for Item
1, there were 3 students who gave a rating of 1, 6 with a rating of 2, 10

140
with a rating of 3, 16 with a rating of 4, and 0 with a rating of 5, adding
up to a total of 35 (the number of our participants). The SPSS page will
give similar matrices for each item.
Percentages
The same matrix gives us percentages. From the “Percent” column, we
can see that 8.6% of the participants gave a rating of 1, 17.1% a rating of
2, 28.6% a rating of 3, 45.7% a rating of 4, adding up to a total of 100%.
All of the percentages are valid, meaning we do not have to go look-
ing for problems with the data entry or elsewhere, and the “Cumulative
Percent” simply progressively adds the percentages to 100%.
Median
The median indicates the midpoint of a set of numbers, and is appropri-
ate for use with non-numerical data and/or data which are not normally
distributed. In order to calculate the median:
• Go to ANALYZE, down to DESCRIPTIVE STATISTICS, across to
FREQUENCIES
• Transfer target items to VARIABLE BOX
• Click STATISTICS, then MEDIAN
• Then CONTINUE, then OK
Q1
Frequency Percent Valid
Percent
Cumulative
Percent
Valid
1.00 3 8.6 8.6 8.6
2.00 6 17.1 17.1 25.7
3.00 10 28.6 28.6 54.3
4.00 16 45.7 45.7 100.0
Total 35 100.0 100.0
Fig. 6.7 Sample frequency matrix

141
A page will appear which includes a matrix like the one given in
Fig. 6.8
On the last line, this matrix gives us the medians for each item. We
can see that the items with the highest medians are Q7 and Q8 (using
inappropriate terms of address, and using bad language); in other words,
these are the items of which most students disapproved most strongly.
Items 2 and 3 (using a phone and talking while someone else is talking)
received medians of 4, indicating a high level of perceived impoliteness.
The remainder of the items received medians of 3, suggesting that, over-
all, these did not arouse strong opinions among the students. None of the
medians, however, were less than 3, indicating that none of the surveyed
behaviours were considered actually polite.
Mode
The mode indicates the most frequent “popular” or “fashionable” rating.
To get this,
• Go to ANALYZE, down to DESCRIPTIVE STATISTICS, across to
FREQUENCIES
• Transfer target items to VARIABLE BOX
• Click STATISTICS, then MODE
A page will appear which includes the matrix provided in Fig. 6.9.
Statistics
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9
Valid 35 35 35 35 35 35 35 35 35
Missing 0 0 0 0 0 0 0 0 0
Median
3.00 4.000 4.000 3.000 3.000 3.000 5.000 5.0000 3.0000
Fig. 6.8 Sample median matrix

142
From this we can see, that although Item 1 received a median of 3, the
most common rating was actually 4, suggesting that a large number of
the students consider coming late to be impolite. Interestingly also, the
mode for Item 6 (about eating or drinking in class) was 2, indicating that
many students actually do not have too much of a problem with this,
although the median for this item was 3.
Correlations
Correlations are a way of finding the relationship between variables,
from which we can infer relationships among a wider population (hence
these kinds of procedures are called inferential statistics). There are dif-
ferent tests of correlation which might be used. The Pearson product–
moment test is used for numerical data, such as exam scores and ages
(assuming these are normally distributed). For non-numerical data
(such as from Likert-type questionnaires) a nonparametric test (such as
Spearman’s rank order correlation) should be used. Correlations can be
used with numerical or ordinal data, but not with nominal data (such
as gender or nationality).
In the case of the small-scale exploration of student perceptions of
classroom politeness we have been using to illustrate research procedures
in this volume, there was no exam score available at the time of the study
which might have been used to correlate exam success with perceptions
of politeness (possibly quite an interesting question!). Furthermore, since
the questionnaire was completed anonymously in order to promote hon-
est rating, it would have been impossible to match the ratings and the
scores, even if they had been available. So the best we can do in the case
Statistics
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9
Valid 35 35 35 35 35 35 35 35 35
Missing 0 0 0 0 0 0 0 0 0
Mode
4.00 4.00 4.00 3.00 4.00 2.00 5.00 5.00 3.00
Fig. 6.9 Sample mode matrix

143
of this study is correlate the items with each other, which might tell us,
for instance, whether students who disapprove of coming late also disap-
prove of various other behaviours. In order to do this:
• Go to ANALYZE, then down to correlate, than across to bivariate
• Transfer target items (Q1–9) to VARIABLE box
• Unclick PEARSON (the default) and click SPEARMAN
• Click OK
The matrix in Fig. 6.10 will be produced (note that, except for col-
umn 1, reproduced in full for the sake of exemplification, only significant
results have been reproduced here because of space constraints).
So, what does it mean? First of all, it is necessary to understand that the
asterisks (e.g. .367*, .493**) indicate probability, which is the likelihood
that the relationship between the two variables in question is more than
would be expected merely by chance. In the first case, .367* indicates
that this probability is at the p < .05 level; in other words, there is more
than a 95% likelihood (also referred to as the confidence interval) that
the relationship is more than chance. In the second case, .493** indicates
a probability rate of p < .01, or a 99% likelihood that the relationship is
more than chance.
Note also that a correlation coefficient of 1.00 goes diagonally from
top left to bottom right. This is because these boxes are correlating items
with the same item; therefore the correlation is perfect (=1).
From the matrix we can see that there are significant relationships at
the p < .01 (99% probability) level between
• Q1 and Q2. In other words, there is a strong probability that students
who think that coming late is impolite will also think that using a
phone in class is impolite.
• Q2 and Q5. In other words, there is a strong probability that students
who think that using a phone in class is impolite will also think that
sleeping in class is impolite.
• Q7 and Q8. In other words, students who think that it is impolite to
use inappropriate terms of address will think the same about the use of
bad language.

144
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9
Q1 Correlation
coefficient
Sig (2-tailed)
N
1.00
-
35
.493**
.003
35
Q2 Correlation
coefficient
Sig (2-tailed)
N
.493**
.003
35
1.00 .538**
.001
35
.367*
.030
35
Q3 Correlation
coefficient
Sig (2-tailed)
N
.325
.057
35
1.00
Q4 Correlation
coefficient
Sig (2-tailed)
N
.028
.875
35
1.00
Q5 Correlation
coefficient
Sig (2-tailed)
N
321
.060
35
.538**
.001
35
1.00
Q6 Correlation
coefficient
Sig (2-tailed)
N
.332
.051
35
1.00
Q7 Correlation
coefficient
Sig (2-tailed)
N
328
.054
35
1.00 .604**
.000
35
Q8 Correlation
coefficient
Sig (2-tailed)
N
.134
.443
35
.604**
.000
35
1.00
Q9 Correlation
coefficient
Sig (2-tailed)
N
.171
.326
35
.367**
.030
35
1.00
**. Correlation is significant at the 0.01 level (2-tailed).
*. Correlation is significant at the 0.05 level (2-tailed).
Fig. 6.10 Sample Spearman’s correlation matrix

145
We can also see that there is a relationship at the p < .05 level between
Q2 and Q9. In other words, there is a 95% likelihood that the relation-
ship between perceptions of using a phone in class and wearing inappro-
priate clothing is more than just chance.
So we have established that there are significant relationships between
some of the variables, but it is now the researcher’s job to decide whether
these relationships are at all important, and, therefore, whether they are
worth reporting and discussing. Is it at all important for us to know that
Q1 is related to Q2, or Q7 to Q8? What use is this likely to be to any-
body? This is a decision that only the researcher can make based on the
aim of the study and the intended audience.
Differences
Differences are another kind of inferential statistic which can be used
to generalize beyond the immediate sample. Nominal categories, such
as gender or nationality, where arbitrary numbers are assigned to par-
ticular groups for the sake of convenience for entering them into a com-
puter programme, cannot be analysed by means of correlations. But such
categories can be used as a grouping mechanism for the purpose of exam-
ining differences.
In the case of our sample study of politeness perceptions in the lan-
guage classroom, there are two variables (gender and nationality) which
might provide interesting insights into different expectations of polite
behaviour. Since we have already found that the data from this study are
not normally distributed, we cannot use t-tests, so we will choose non-
parametric tests. If we look first of all at gender:
• Click ANALYZE, go down to NONPARAMETRIC TESTS, go across
to LEGACY DIALOGUES, then 2 INDEPENDENT SAMPLES
• Move target variables (Q1–9) across to the TEST VARIABLE LIST
box
• Move “gender” into the GROUPING VARIABLE box
• Click DEFINE GROUPS, and put 1 (male) in the first box and 2
(female) in the second

146
• Choose MANN–WHITNEY U if it is not already selected
• Click CONTINUE, then OK
The page produced will include the matrix provided in Fig. 6.11.
Bearing in mind that a probability value of anything over 0.05 in
not usually considered significant, we can see from the Asymp. Sig line
that none of these differences is significant. In other words, although
there might, perhaps, be some differences according to gender (maybe
a higher or lower median), such differences as there might be are not
large enough to be considered more than might be expected merely by
chance.
But it should be remembered that this may, in itself, be an interesting
finding, since gender is often believed to affect perceptions of behaviour
such as politeness. From this point of view, to discover that there is actu-
ally no significant difference is as interesting as if we had found some-
thing significant.
Test Statisticsa
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9
Mann-
Whitney
U
139.00
0
92.500 139.50
0
142.50
0
117.00
0
94.000 102.00
0
128.50
0
116.00
0
Wilcoxo
n W
392.00
0
183.50
0
230.50
0
233.50
0
208.00
0
347.00
0
193.00
0
219.50
0
207.00
0
Z -.146 -1.803 -.128 -.018 -.913 -1.728 -1.540 -.622 -.967
Asymp.
Sig. (2-
tailed)
.884 .071 .898 .986 .362 .084 .124 .534 .334
Exact
Sig.
[2*(1-
tailed
Sig.)]
.906b
.085b
.906b
.987b
.389b
.098b
.169b
.625b
.371b
a. Grouping Variable: Gender
Fig. 6.11 Sample matrix for Mann–Whitney U test of difference—gender

147
If we want to find out if there are any differences according to nation-
ality (Turkish and “others”), we follow the same procedure through to
the point of defining the grouping variable, when we select the “nation-
ality” rather than the “gender” variable. The matrix in Fig. 6.12 will be
produced:
In the case of this analysis, we can see that Item 1 (coming late to
class) shows a significant difference according to nationality (p = .029).
But which group is it that considers unpunctuality more impolite? If
we look at the matrix above the “Test Statistics” matrix, we find this
(Fig. 6.13).
From this matrix we can see that the mean rank for Item 1 for
Turkish students (group 1) is only 15.60, while the mean rank for
other nationalities (group 2) is 23.23. In other words, international
students consider unpunctuality considerably more impolite than
Turkish students.
Test Statisticsa
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9
Mann-
Whitney
U
74.500 99.500 94.000 98.500 120.00
0
117.50
0
127.50
0
124.50
0
129.00
0
Wilcoxo
n W
374.50
0
399.50
0
394.00
0
398.50
0
420.00
0
417.50
0
193.50
0
190.50
0
195.00
0
Z -2.183 -1.208 -1.442 -1.235 -.438 -.532 -.176 -.335 -.112
Asymp.
Sig. (2-
tailed)
.029 .227 .149 .217 .661 .595 .860 .738 .911
Exact
Sig.
[2*(1-
tailed
Sig.)]
.040b
.252b
.186b
.238b
.687b
.612b
.875b
.793b
.930b
a. Grouping Variable: Nationality
Fig. 6.12 Sample matrix for Mann–Whitney U test of difference—nationality

148
This small-scale study identified only two groups according to nation-
ality, since to have divided the “other nationalities” according to their
actual origins (e.g. Greece, Turkmenistan) would have produced unviably
small groups and emphasized even more the numerical dominance of the
Turkish students. If, however, there were sufficient numbers to support
Ranks
Nat N Mean
Rank
Sum of
Ranks
Q1
1.00 24 15.60 374.50
2.00 11 23.23 255.50
Total 35
Q2
1.00 24 16.65 399.50
2.00 11 20.95 230.50
Total 35
Q3
1.00 24 16.42 394.00
2.00 11 21.45 236.00
Total 35
Q4
1.00 24 16.60 398.50
2.00 11 21.05 231.50
Total 35
Q5
1.00 24 17.50 420.00
2.00 11 19.09 210.00
Total 35
Q6
1.00 24 17.40 417.50
2.00 11 19.32 212.50
Total 35
Q7
1.00 24 18.19 436.50
2.00 11 17.59 193.50
Total 35
Q8
1.00 24 18.31 439.50
2.00 11 17.32 190.50
Total 35
Q9
1.00 24 18.13 435.00
2.00 11 17.73 195.00
Total 35
Fig. 6.13 Sample mean ranks grid

149
analysis of more than two groups, this can be done with a Kruskal–Wallis
H test, which is, effectively, the nonparametric equivalent of an ANOVA
with parametric data. To do this:
• Follow the above procedure to LEGACY DIALOGUES, then go
across to K INDEPENDENT SAMPLES
• Move the target variables to the TEST VARIABLE LIST, and the
grouping variable to the GROUPING VARIABLE box
• In the DEFINE RANGE BOX, identify the MAXIMUM and the
MINIMUM values
• Then CONTINUE and OK
Something similar to the “Test Statistics” matrix above will be pro-
duced which will tell you whether there is a significant difference accord-
ing to the grouping variable (p less than .05). To determine which of the
groups rated the item more or less highly, refer to the “Ranks” matrix on
the same page.
Effect Size
Increasingly, journals are requiring reporting of effect size, since this is
not dependent on sample size as other statistical procedures are. For those
who aspire to publication, knowing how to produce effect sizes may be
necessary.
Confusingly, there seem to be numerous ways of calculating and inter-
preting effect sizes, but a common and relatively straightforward way is
to calculate the Eta value for parametric data:
• Go to ANALYZE, then down to DESCRIPTIVE STATISTICS, then
across to CROSS TABS
• Transfer the target dependent variables into the ROW/S box
• Transfer the target independent variable into the COLUMN/S box
• Click STATISTICS, then ETA

150
The programme will produce Eta values for each of the dependent
variables.
And what does it mean? According to Cohen (1988), the strength of
an Eta value can be assessed according to the following “rule-of-thumb”
thresholds (Table 6.1).
When calculating effect size from a Mann-Whitney, divide the Z figure
by the square root of the number of participants (Yanati, K. http://yatani.
jp/teaching/doku.php?id=hcistats:mannwhitney#effect_size) When cal-
culating from a Kruskall-Wallis, you can use Wilson, D. Practical Meta-
Analysis Effect Size calculator: https://www.campbellcollaboration.org/
escalc/html/EffectSizeCalculator-R5.php.
It is important to bear in mind, however, that “the main short-
coming of effect size… [is] that there are no universally accepted and
straightforward indices to describe it” (Dörnyei 2007, p. 212). Indeed,
the APA Publication Manual itself lists more than a dozen different ways
of estimating effect size.
Types of Data: Qualitative
Increasingly, qualitative research methods are gaining recognition for the
human dimension that they can add to quantitative methods. Perhaps
quantitative methods may be especially useful in a mixed methods
approach, where qualitative interpretations of data provided by par-
ticipants can add useful insights to otherwise de-personalized sets of
numbers.
There are two major ways of approaching a qualitative data set:
deductive and inductive. In the former, the researcher analyses against
Table 6.1 Approximate effect size thresholds for Eta values
Size Eta value
Small .1
Medium .24
Large .38

151
pre-set categories or themes. This is usually a practice in studies in
which particular dimensions are designed in a close-ended question-
naire. The verbal responses are evaluated around these dimensions
without inducing any new category from the data. Such data analysis
practices are confirmatory in the sense that the researcher checks the
data and confirms whether the category is evident in the data. Themes
and categories are gathered from the existing relevant literature through
careful synthesis of different views, or the dimensions of the borrowed
questionnaires may already contain themes and categories that allow for
categorization of the data.
In an inductive approach to data analysis, the researcher induces
emerging themes from the data as suggested by grounded theory through
an exploratory perspective. The researcher starts the analysis with no prior
selection of categories and themes. Rather than test hypotheses or theo-
ries, this kind of data analysis is used to build potential theories about
the phenomenon under investigation. Inductive analysis requires careful
reading and critical thinking to bring out themes and build them into a
meaningfully related whole. The characteristics of the two approaches to
data analysis are displayed in Table 6.2.
Analysing Interview Responses
The data collected and recorded through interviews should be transcribed
carefully. However, rather than transcribing all the data, it is possible
Table 6.2 Characteristics of deductive and inductive approaches to data analysis
Deductive Inductive
Follows a top-down strategy Follows a bottom-up strategy
Depends on external themes and
categories
Is data-driven
Tests external themes and categories Induces themes and categories
Categorizes content mechanically Synthesizes content creatively
Is confirmatory and static Is exploratory and dynamic
Supports existing literature Interprets new meanings

152
to select just the necessary information from what the participants have
said. Three basic steps could be:
• Read these manuscripts again and again until no more meanings can
be drawn (saturation)
• Identify and underline words or groups of words that have negative or
positive meanings relating to the theme being researched (open coding)
• Seek meaning relations among all these words (emerging themes) and
look for thematic groupings (axial coding)
• Look for an overall theme which will unify the study thematically
(selective coding)
Analysis of Observational Data
Analysing observational data depends on how data were collected:
through structured (checklists or schemes) or unstructured (open-ended,
thick notes) methods.
Structured observations can be analysed through ranking, percentages,
mean scores of behaviours, interactions, or practices. Such analysis could
help researchers present a set of data quantified from qualitative data.
For example, Fig. 6.14 shows how many times the teacher used different
types of error correction strategies in three different lessons.
0
2
4
6
8
10
Explicit Recast Metalingisc Elicitaon Clariﬁcaon
Error correcon types
Lesson 1 Lesson 2 Lesson 3
Fig. 6.14 Teacher’s error correction strategies

153
This is an example of how qualitative data is quantified in a simple
manner and shown in a bar chart. The numbers indicate that the teachers’
error correction practices are usually explicitly made and involve metalin-
guistic error correction. According to the results of this observation, the
teacher does not frequently resort to implicit correction strategies such as
elicitation, clarification, and recast. This is at least the case with the three
lessons observed.
Unstructuredobservationisanalysedthroughdifferentways.Itdepends
on how the data were collected: time-oriented or theme-oriented. If it is
collected on the basis of the particular periods such as the first 10 min-
utes, the second 10 minutes, and so on, then whatever happened during
these particular intervals is grouped together. On the other hand, if it
is theme-based, then each theme is exemplified and described in detail
with reference to how it happened in the classroom. For example, if the
research question is
• How does the teacher correct students’ verbal mistakes?
a checklist containing types of error correction can be prepared and
ticked as the teacher practises each type. The frequency of error correc-
tion that happens is analysed quantitatively. In order to understand the
nature of the mistakes, the conversation and interaction pattern between
the teacher and the student might need to be written down; or a video-
tape of the lesson will help to identify the error correction and how cor-
rections are made.
Reliability of Qualitative Results
During the analysis of the qualitative data, there are particular strategies
to follow which could enhance the trustworthiness of the analysed data.
The findings from the analysis of verbal or written data are often criticized
for being too subjective in that they are drawn from the researcher’s own
perspectives. To minimize the risk of drawing subjective results, research-
ers need to consult a second view about the appropriateness of the catego-
ries and themes that they induced. If this is done, it is possible to report
inter-rater reliability by calculating the number of categories identified by

154
one researcher, the number of categories identified by another researcher,
and expressing the relationship as a percentage. This practice is often
skipped or not reported properly, which makes readers suspicious of the
accuracy and authenticity of the data collection and analysis process.
Another way of safeguarding reliability is by means of member-checking,
which refers to the process of getting approval from the research partic-
ipants for the data transcribed, analysed, and reported in the paper. At
each stage, it is necessary to work closely with the participants and share
with them the analysed data. The participants might approve or ask for
modifications or changes in the particular statements which do not express
properly what they wanted to say. This process adds to the results’ trustwor-
thiness, and needs to be explicitly documented in the data analysis section.
Debriefing is another way of developing the level of trustworthiness of
the data. This time colleagues or other researchers or co-authors of the
paper can be consulted to check the accuracy and appropriateness of the
categories and themes. While selecting such people, it is necessary to pay
attention to whether they know about the research topic and have experi-
ence in qualitative data coding. Otherwise, inappropriate confirmation
may result which could spoil the findings. The ultimate goal is to find out
the level of inter-coder reliability. Again, all this process should be
reported in the methodology section with a detailed discussion of the
confirmed and disconfirmed categories and themes.
Questions to Consider
1. What is the difference between parametric and nonparametric data?
2. What is the difference between numerical, ordinal, and nominal data?
3. To what does reliability refer?
4. What does a factor analysis tell us?
5. If we say data is/not normally distributed, what do we mean?
6. How are mean, median, and mode different?
7. What do correlations tell us?
8. What are two common nonparametric tests of difference?
9. To what does the Eta value refer?
10. Why can qualitative data be useful?
11. What are the three coding stages?
12. What are some ways of safeguarding reliability?

155
Table 6.3 Median ratings for impolite classroom behaviour
No. Item Median
1 Coming late 3
2 Using a phone 4
3 Talking while someone else is talking 4
4 Not paying attention 3
5 Sleeping 3
6 Eating or drinking 3
7 Using inappropriate terms of address 5
8 Using bad language 5
9 Wearing inappropriate clothing 3
Example: An Action Research Study on Classroom Culture: Data
Analysis
Quantitative data
Since the quantitative data obtained from the Likert-type questionnaire
used in this study are ordinal, the ratings were analysed for medians, as
shown in Table 6.3
We can see that the items with the highest medians are Q7 and Q8 (using
inappropriate terms of address and using bad language); in other words,
these are the items of which most students disapproved most strongly.
Items 2 and 3 (using a phone and talking while someone else is talking)
received medians of 4, indicating a high level of perceived impoliteness.
The remainder of the items received medians of 3, suggesting that, over-
all, these did not arouse strong opinions among the students. None of the
medians, however, were less than 3, indicating that none of the surveyed
behaviours were considered actually polite.
Qualitative data
In order to analyse the comments that students were asked to add to
explain their ratings, a grounded approach was adopted. This involved
overviewing the data and noting down all the themes presented (open-
coding stage). Next, these sometimes rather fragmented ideas were orga-
nized around a central theme (axial coding). Finally, an overarching theme
was identified, which unified all the ideas presented (selective coding). In

156
order to check reliability, a member-checking procedure was adopted; in
other words, the results were taken back to the class and discussed, and
some changes/additions were made according to the students’ feedback
before the final coding classifications were decided.
Open coding
At this stage, the comments made by the students were reviewed (not all
students made comments about every item) and noted down under the
relevant questionnaire item:
1. Coming late
• Being on time is your responsibility
• Causes lack of concentration
• If it is temporary OK, but if they keep coming late it is rude
2. Using a phone
• It is disrespectful
• It disturbs others
• OK for emergency, but it causes students to lose their concentration
3. Talking while someone else is talking
• We should behave as we want to be treated
• Plain rude
• Everybody needs to respect one another
4. Not paying attention
• This is disrespectful
• It distracts others
• Don’t attend the class if you won’t pay attention anyway

157
5. Sleeping
• There is always coffee
• It is disrespectful
• Bedrooms are for sleeping, not classes
6. Eating or drinking
• Sometimes we don’t get time for lunch or breakfast
• OK unless loud or smells bad
• Maybe he/she has an illness
7. Using inappropriate terms of address
• This is very rude
• It may cause unfriendly situation or arguments
• It can be very embarrassing, but it depends on the culture
8. Using bad language
• This is impolite not just in class but everywhere
• Can cause anger, losing concentration, and so on
• Unrespect for the teacher
9. Wearing inappropriate clothing
• It may be distracting. This is a school
• It is a personal choice
• It is important to be comfortable
Axial coding
Further examination of these comments seemed to suggest that they fell
into two groups: those that were accepting of the behaviour in question,
and those that were unaccepting. These might be set out as follows:

158
Item Behaviour Unacceptable Acceptable
1 Coming late Irresponsible
Distracting
OK if not habitual
2 Using a phone Disrespectful
Disturbing
OK for an emergency
3 Talking while
someone
else is talking
Rude
Disrespectful
4 Not paying attention Disrespectful
Distracting
5 Sleeping Disrespectful
Inappropriate in a
classroom
6 Eating or drinking Sometimes necessary
because of time or
health issues
7 Using inappropriate
terms of address
Rude
Embarrassing
Confrontational
8 Using bad language Causes anger
Disrespectful
9 Wearing
inappropriate
clothing
Distracting Personal choice
Need for comfort
From this analysis, we can see that a number of the behaviours (Items
3, 4, 5, 7, 8) received entirely negative comments, suggesting that there
was strong opposition among these students to talking while someone
else is talking, not paying attention, sleeping, using inappropriate terms
of address, and using bad language. One (Item 6) received only positive
comments, suggesting that eating and drinking in the classroom is accept-
able under some circumstances. Reactions to other behaviours (Items 1,
2, 9) were more balanced, suggesting a degree of tolerance towards com-
ing late, using a phone and standards of dress, according to circumstances
or personal choice.
Selective coding
When it comes to selecting an overarching theme, the topic is clearly
about behaviour, the context is a university classroom, and the partici-
pants are university students, so this might perhaps be expressed as:
Student perceptions of acceptable behaviour in a university classroom

159
Task Deductive analysis
Collect some written or verbal learner reflections on a particular aspect of
your lesson.
Transcribe them on a word document.
Prepare a set of categories from relevant studies.
Read the transcribed data carefully and code it against the pre-set catego-
ries identified.
Identify quotes from the data that would best represent each theme.
Summary
The information provided in this chapter is, of necessity, limited and
selective. For those who would like to know more about data analysis and
interpretation, we recommend a specialist text such as Dörnyei (2007).
Alternatively, taking a data analysis course can be time and money well
spent.
Although it is a common practice to hand one’s data over to a statisti-
cian to analyse, it is very empowering to be able to do it oneself, since
nobody else really understands what we have done, how or why we have
done it, and what we are trying to achieve. In other words, nobody will
analyse our data as well as we could ourselves once we have empowered
ourselves with the knowledge and experience of how to do it.
Task Inductive analysis
Collect some written or verbal learner reflections on a particular aspect of
your lesson.
Transcribe them on a word document.
Begin open coding and create as many categories as you can.
Identify major themes among these categories.
Select quotes from your data that would best represent each theme.

160
Task Analysis of observation and interview data
Decide on a particular focus for observing some of your learners in the class-
room: either through a video or in another teacher’s lesson.
Create a checklist or a semi-structured protocol as an observation tool.
For after the observation, prepare some questions that need further elabo-
ration by the learners.
Reconsider each observation focus with what they said in the post-
observation interview.
Analyse the interview responses inductively.
Task Analysis of quantitative data
Decide whether the data are numerical, ordinal, or nominal.
Calculate whether distribution is normal or not.
Calculate means or medians as appropriate.
Calculate reliability.
Calculate correlations and/or differences as appropriate.
Conduct any other appropriate tests.

Analysing The Data

More Related Content

Similar to Analysing The Data (20)

More from Angel Evans (20)

Recently uploaded (20)

Analysing The Data