An Online Questionnaire For Evaluating Students And Teachers Perceptions Of Constructivist Multimedia Learning Environments.

Research in Science Education (2005) 35: 221–244 © Springer 2005
DOI: 10.1007/s11165-005-2148-3
An Online Questionnaire for Evaluating Students’ and Teachers’ Perceptions
of Constructivist Multimedia Learning Environments
Dorit Maor1 and Barry J. Fraser2
1Murdoch University
2Curtin University of Technology
Abstract
The purpose of this article is to describe the development, validation and use of the Constructivist
Multimedia Learning Environment Survey (CMLES). This questionnaire assesses teachers’ and stu-
dents’ perceptions of the learning environment when students use online multimedia programs while
teachers use constructivism as a referent for their teaching. The design of the questionnaire was based
on a constructivist approach to learning and focused on the process of learning with the multimedia
program and on the nature of that program. Before the use of the CMLES becomes widespread, it
was important to determine whether it is valid and reliable. Therefore, a study involving 221 students
in 12 high school classrooms into statistical validation and interpretive validation of the questionnaire
was undertaken. For this sample of Grade 10 and 11 students who completed the actual and preferred
forms of the questionnaire, the CMLES scales demonstrated a high degree of internal consistency
reliability (with alpha reliability coefficients ranging from .73 to .82), as well as satisfactory factorial
validity and discriminant validity. Therefore, the study supports the reliability and validity of the
CMLES for assessing students’ and teachers’ perceptions of one important aspect in evaluating
learning environments which promote the use of multimedia programs and constructivist learning
approaches.
Key Words: constructivist multimedia learning environment, high school, online media inquiry learn-
ing
The growing use of computers in schools for promoting higher-order learning
(Daniel, 1996; Orion, Dubowski, & Dodick, 2000) and the attempt to use construc-
tivist approaches to learning in classrooms (Cobern, 1995; Hand & Prain, 1995; Maor
& Taylor, 1995) led to a need for an instrument to assess this unique learning environ-
ment. For this purpose, the Constructivist Multimedia Learning Environment Survey
(CMLES) was developed, used and validated for assessing teachers’ and students’
perceptions of their experiences. This article specifically describes the development
and validation of this instrument in order to provide teachers and educators with a
tool for assessing students’ perceptions of the extent to which they are implement-
ing a constructivist approach in their teaching involving the use of multimedia. In
particular, this questionnaire can help to assess students’ learning environments that
combine the use of multimedia programs with inquiry learning.
In order to promote students’ development of investigative skills, inquiry-oriented
multimedia learning programs are increasingly being developed (e.g., Edleson, 2001;

222 DORIT MAOR AND BARRY J. FRASER
Songer, Lee, & Ham, 2002), but they are still scarce. These programs should in-
volve real-world contexts in which users practice problem-solving of authentic tasks,
as well as a variety of experiences and multiple perspectives in order to promote
higher-level thinking skills and embedded learning in a social context (Bonk &
Cunningham, 1998; Nelson, 1994).
A central goal of science education is to help students to develop higher-order
thinking skills (Bybee & DeBoer, 1994), which are a set of intellectual skills that
enable students to think critically, ask critical questions, reason and solve problems.
However, observational studies in science classrooms consistently have revealed an
emphasis on memorisation and lower-order learning (Tobin & Gallagher, 1987), as
well as the existence of numerous barriers to achieving learning environments that
emphasise higher-order learning, such as teachers maintaining a didactic approach in
the classroom (Songer et al., 2002; Tobin, 1993). Higher-order learning can be facil-
itated by implementing a constructivist-oriented classroom pedagogy (Tobin, 1998)
that builds on the theory of social constructivism which positions the learner as an ac-
tive constructor of knowledge within a socially-interactive environment (O’Connor,
1998; Tobin & Tippins, 1993). Social constructivism suggests that learning is a
socially-mediated experience for which individuals construct knowledge based on
interactions with their social and cultural environments (Hirumi, 2002).
One way of promoting higher-order thinking skills (Horak, 1991) involves engag-
ing students in investigations using interactive multimedia (Barnea & Dori, 1996;
Edelson, 2001; Kozma, 2000; Kozma & Russell, 1997; Krajcik, Simmons, & Lunetta,
1988) within socially-interactive classrooms. In a study of the use and perceived
usefulness of educational multimedia resources and communication technologies in
a biology program for first-year university students, communication technologies
were used more frequently for social interactions rather than for course-specific
needs (Peat, Franklin, & Lewis, 2003). In a socially-interactive classroom, students
can use selected interactive multimedia packages and Web-based programs to fa-
cilitate the use of inquiry-based and constructivist-oriented approaches to learning
science. Throughout this process, students can develop their higher-order thinking
skills by interacting in pairs with the multimedia programs. They are involved in
asking creative questions, designing investigations based on the multimedia software,
negotiating ideas and solving problems during group discussions.
The focus of the teaching in this environment is student learning using the com-
puter as a tool to facilitate understanding (Maor & Taylor, 1995). Previously, the role
of computer software was perceived as assisting students’ learning as that of a ‘tutor,
tool and tutee’ (Taylor, 1980). More emphasis is given to the role of the computer
in classrooms as a cognitive tool and to the constructivist learning approach that
has been adopted to promote critical thinking and higher-order learning (Jonassen,
1994; Jonassen & Reeves, 1996). In a study involving students in their own learning
by providing a curiosity-driven learning environment (Parril & Gervay, 1997), the
teacher used movies presented on the Web. In a different multimedia environment,
students worked in pairs to compose written collaborative responses to questions
embedded in the multimedia program Cview (Daniel, 1996). Criticisms about the
use of multimedia to teach science concepts suggest that

CONSTRUCTIVIST MULTIMEDIA LEARNING ENVIRONMENTS 223
although most of the students enjoyed using the multimedia program, there was no evidence to support
the assumption that it contributed to knowledge acquisition. In fact, much of the time invested in multi-
media authoring was devoted to producing decorative effects, reducing the time available for meaningful
learning. (Orion et al., 2000, p. 121)
Within the National School Framework (Newman, 1993), high school science
teachers collaborated in an attempt to use computer simulation of phenomena such
as gravity, relativity, photosynthesis and population ecology to share the output of
their investigations. One such a program is the Collaborative Visualisation Project
(CoVis), which provides an opportunity to re-think science education in the light
of the new pedagogy and emerging technologies (Harasim, Hiltz, Teles, & Turoff,
1996). Through the CoVis project, students are able to join with other students at
remote locations in collaborative group work and also to communicate with univer-
sity researchers and other scientific experts using scientific visualisation software.
Another Australian project for science classrooms is VisChem, which provides mul-
timedia resources to teach introductory chemistry (http://vischem.cadre.com.
au). Although it uses computer animation to illustrate the molecular world, Tasker
et al. (2002) warns us about the visual misrepresentation that this animation can
create.
Also, in the area of chemistry, Wu, Krajcik, and Soloway (2001) investigated
students’ understanding of chemical representations with the aid of computer-based
visualisation tool, eChem, that allows students to build virtual molecular models
and view multiple representations simultaneously. This study revealed that students
who were highly engaged in discussions while using the multimedia made better
links between visual and conceptual aspects of molecular representation (Wu et al.,
2001). The ChemSense Knowledge Building Environment (Schank & Kozma, 2002)
allows students and teachers to collaborate in investigating of chemical phenomena
and develop their inquiry skills. Edelson (2001) attempted to integrate the content
of science and the process of inquiry using a multimedia database program. He de-
scribed a curriculum, called the Create-a-World project, in which students engage in
open-ended earth science investigations using WorldWatcher (Gordin & Pea, 1995),
a geographic visualisation and data analysis environment that enables students to
develop inquiry skills, both general and domain-specific, and provides opportunities
to develop and apply those skills.
Because of the increasing use of multimedia programs in science education, the
development of a new tool that provides a range of indicators of students’ and teach-
ers’ perceptions of a multimedia learning environment was considered both impor-
tant and timely.
The particular educational programs selected for our study were considered to
support a constructivist-oriented approach to learning. In the present study, 12 Grade
10 and 11 students were engaged in interaction with a multimedia program and an
online program during a two-year period in an attempt to promote inquiry learning.
In 1999, the program Birds of Antarctica (Maor, 2001) was used by Grade 11 stu-
dents. This multimedia program involves an authentic database, based on a scientific
expedition to Antarctica, which enables students to conduct scientific investigation

by asking questions, designing investigations, and retrieving the answers from the
database. Logal online interactive software (http://www.riverdeep.net/demos/
logal_science_demo.jhtml/) was also selected for use in 1999. This program
takes students on a virtual journey to explore science topics. For example, in the
biology gateways, students go on a virtual field trip to explore population dynamics
or the principles of human genetics or photosynthesis.
In a later stage of the study, Exploring the Nardoo (Harper & Hedberg, 1997)
provided a rich information landscape of resources based on a geographic metaphor
which incorporated a Water Research Centre and a complex navigable river envi-
ronment. The students are challenged to become active participants when presented
with a range of media forms upon entry to the river environment. During the use of
these programs, the teacher simultaneously incorporates a constructivist approach to
the learning environment by encouraging students to work in pairs, promoting class
discussions, and asking students to reflect frequently on their learning.
The research reported in this article first involved a pilot study to trial our new
questionnaire with the multimedia programs and, at the second stage, students worked
with the multimedia program while the researchers evaluated the learning environ-
ment and validated the questionnaire.
Methods
A Pilot Study – Use of Constructivist Multimedia Program
A classroom-based pilot study, partly reported in this article, involved investi-
gating an attempt at developing students’ higher-order thinking skills while using
a constructivist approach to teaching and learning and an interactive multimedia
program. Initially, for the pilot study, two classes of 38 students engaged with the
interactive multimedia program Birds of Antarctica (Maor, 2001) in a constructivist-
oriented learning environment. Twice each week during the month following the
initial investigation, students interacted with each other and the multimedia program.
The two classes were drawn from a larger sample of Grade 10 and 11 information
technology classes within a boys-only secondary school in Perth, Western Australia.
Our research was conducted with the students as part of their IT course.
Students’ perceptions of their actual and preferred learning environments were
obtained by administrating our newly-developed Constructivist Multimedia Learning
Environment Survey (CMLES) designed especially for this study. The purpose of this
instrument is to assess the degree to which students and teachers perceive that their
classroom environment involves students in negotiation, inquiry learning and reflec-
tive thinking. The first part of the questionnaire assesses students’ perceptions of
the process of learning with the multimedia program and contains three scales: Stu-
dent Negotiation based on the Constructivist Learning Environment Survey (Taylor,
Fraser, & Fisher, 1997) and Inquiry Learning and Reflective Thinking based on the
Computer Classroom Environment Inventory (Maor & Fraser, 1996). Items in this

questionnaire reflect a constructivist-oriented approach to teaching and learning. The
second part of the CMLES assesses students’ reactions to the interactive multimedia
program and contains the two new scales of Authenticity and Complexity.
There were 25 items in this preliminary version of the CMLES, with five items
in each scale. For each item, the frequency response alternatives range from Almost
Never to Always. The CMLES consists of two forms, actual and preferred, which
were administrated to both students and teachers. The actual form assesses partici-
pants’ perceptions of the currently-prevailing classroom learning environment, while
the preferred form allows participants to give opinions about their ideal classroom
learning environment.
Qualitative data were gathered from an open-ended questionnaire which was ad-
ministered to the students in order to ascertain their perceptions about social learning
in the classroom. Additional qualitative data were obtained from a group of teachers
who were using the CMLES in their classrooms as an operational framework for
their classroom practice. This group of teachers was engaged in an online discussion
of the strengths and weaknesses of the questionnaire as part of the pilot study.
Following a small-scale qualitative study and initial validation of the instrument
during this pilot study, a revised version of the CMLES was finalised. Some of the
changes from the original version involved making the wording in the questionnaire
more user-friendly for students. The name of the scale Student Negotiation was
changed to Learning to Communicate, Inquiry Learning was changed to Learning
to Investigate, and Reflective Thinking was changed to Learning to Think. In the
second part of the questionnaire, the scale names were changed from Authenticity
to Relevance, and from Complexity to Ease of Use. A sixth scale of Challenge was
added to assess what the researchers believe to be an important characteristic of the
constructivist framework. Thus, the refined version of the questionnaire contained a
total of 30 items (see Appendix A for the actual form of the CMLES). A description
and a sample item for the revised form for each scale is presented later in Table 1.
Use of Revised CMLES
The next stage involved administering the revised version of the CMLES to a
group of five classes involving 100 Grade 10 and 11 students and three teachers
using the two multimedia programs in an all-boys school. The students interacted
with online science programs (Logal, 1999) four times a week during a period of
three weeks. The interactive online science program includes topics such as explor-
ing populations, photosynthesis and genetics. During that time, students investigated
in greater depth a topic of their choice, reported their investigation back to the class,
and evaluated their learning within the topic of their choice. The teacher encouraged
students to work in pairs and promoted interactions and dialogue. Students were also
asked to summarise what they had learned in each lesson.
Later, seven classes involving 121 students from the same school used the pro-
gram Exploring the Nardoo in order to develop their investigation skills in a social

Table 1
Descriptive Information for Each Scale of the CMLES.
Scale name Description Sample item
Negotiation Extent to which students
have opportunities to discuss
their questions and their so-
lutions to questions
In this class, I get the chance
to talk to other students
Inquiry Learning Extent to which students are
encouraged to engage in in-
quiry learning
In this class, I find out an-
swers to question by inves-
tigation
Reflective Thinking Extent to which students
have opportunities to reflect
on their own learning and
thinking
In this class, I think deeply
about how I learn
Relevance Extent to which the informa-
tion in the program is au-
thentic and representative of
real-life situations
Working with this multime-
dia program, I find that I am
presented with realistic tasks
Complexity Extent to which the program
is complex and represents
data in a variety of ways
dia program, I find it easy to
navigate
Challenge Extent to which the
program challenges and
stimulates students to think
dia program, I find that it
makes me think
constructivist learning environment created by the teacher. Using the multimedia
program, students explored the ecology of the Nardoo (an imaginary region) us-
ing a comprehensive set of multimedia resources. Exploring the Nardoo supports
the study of interactions between living organisms and their physical and chemical
environments, with particular emphasis on the role and impact of humans at both
a macro- and micro-level (Harper & Hedberg, 1997). These classroom activities
enabled the researchers to evaluate further the use of the CMLES and to examine
students’ development of higher-level thinking skills.
All classes involved in the research were taken to a computer laboratory with 16
networked PCs running Microsoft NT Workstation version 4. The network was con-
nected to the Internet via a two-megabyte-per-second PAPL modem. The computer
desks were arranged around the walls of the classroom, with tables in the middle for
group work and whole-class discussion.

During 1999, students were asked to work in pairs at one computer and to be
active in their discussions about the activities presented to them by the Logal Web site
(http://www.riverdeep.net/demos/logal_science_demo.jhtml) from which
they could select a science program. The students were also asked to prepare a
presentation for the class about what they had found interesting and what they had
learned from the program. They used screen captures (snapshots from the online
program) for their presentation.
2. Following an extensive introduction to the study by the teacher and the re-
searcher, the students were asked to treat this three-week experience as (2) research
into what constitutes a good Web design (including evaluating the use of the Logal
Web site as part of their computing course) and an investigation of the use of online
interactive multimedia for the development of higher-level thinking skills.
Initially, the students experienced some difficulties with down-loading the pro-
gram. The Internet connection was very slow because of an unknown setting at the
time. This caused frustration to the students, the teachers and the researcher involved
in the study. Overcoming the problem with the Internet connection with the Logal
server was the major activity during the first two periods.
On average, students had 20 minutes using the Logal Web site each lesson, with
the rest of the time being spent logging onto the Web site, undertaking their reports
and participating in class discussion. One pair of students in each class was video-
taped along with its activities on the Logal Web site. A major weakness of the Logal
program is the repetitive nature of its style of questioning. Students found themselves
doing the same type of activities in different content areas.
The second stage of the study was conducted in a similar way. Students worked
in pairs while they evaluated the multimedia program while solving social and eco-
logical problems of the Nardoo region. To overcome the Web connection problems
experienced previously, a different program was selected which did not require In-
ternet connection and could be used from a CD Rom. However, in a fashion similar
to the Logal program, Exploring the Nardoo involves repetitive approaches, which
can create problems with student motivation. Seven IT classes and three teachers
participated in this part of the study.
The CMLES home page was designed (http://www.smec.curtin.edu.au/
forms/cmles.html) and the CMLES was administered electronically to the stu-
dents in their classes via the Web. The data from all students were collated into a
spreadsheet and then analysed using SPSS.
Results
In this section, results are presented from different perspectives. First, teachers’
feedback on their use of the CMLES in their classrooms during online discussions
is presented. Second, analyses of quantitative data derived from students’ responses
to the online version of the CMLES are presented together with some evidence from
classroom observations. The results are presented in both tabular and graphical form.

As well, the perceptions of two students who were videotaped during their interaction
with the program are compared in terms of the mean scores for their classroom.
Teachers’ Feedback
Using the CMLES encouraged the teachers to think about the extent to which
they had used a constructivist-oriented teaching approach relative to what they had
initially set out to do. According to one of the teachers, the items in the first part of
the CMLES reflect a constructivist learning environment which:
. . .encourages the construction of knowledge through: negotiation (both internal and social); collaboration
(with both students and teacher); exploration and invention of real-world environments and questions;
independent and interdependent learning; reflection on your own learning and understanding; and an
exploration, questioning and testing of solutions. (Teacher)
This insight from the teacher identifies a major strength of the questionnaire as
being that teachers and students can identify what is/is not occurring in the learning
process. However, the teacher’s major concern based on her classroom experience
was the extent to which we equip students and/or teachers to recognise that this is
actually occurring, or should occur, in an interactive learning environment. Another
teacher suggested that the questionnaire enabled students to focus their attention on
what to expect, how to learn and the nature of their activities.
In the process of self-reflective teaching, teachers came to realise the value of
social learning through negotiation and reflective thinking. One of the teachers sug-
gested:
I think that reflecting for a moment (at least) on how the students, as well as ourselves as tutors, come to
terms with the learning process and the aspects of the learning environment is valuable, even if only to
formalise goals and recognise sign posts along the pathway of the learning process.
According to some of the teachers, having separate preferred and actual forms of
the CMLES made them aware of the opportunity to determine why goals might or
might not be met in the learning environment:
I felt that it was useful for me to see how I did, relative to what I set out to do. It was interesting to see
how the students perceive their learning.
By using the questionnaire, teachers became aware that, when actual and preferred
perceptions do not support one another, it is important to scrutinise the components
in the learning environment. Or it could suggest that the goals of the teachers and the
learners are not synonymous and thus the situation needs to be resolved.
Experience with the questionnaire also emphasised to teachers that the learning
process needed to be put ahead of the technology. This was reflected by the comment
that:

. . .a major strength of the survey is its convergence effect, bringing students and teachers closer in their
perceptions of the need to promote active learning.
The teachers also gave some practical advice concerning how to improve the ques-
tionnaire. Although lower-ability students had a few problems with the readability of
some of the words, overall the students handled the survey well. Another teacher’s
valuable comment related to the scale of Negotiation. This teacher complained that
Negotiation referred only to interactions among students and not to interactions
between the teacher and students, which are important in a constructivist learning
environment (Maor & Taylor, 1995). In relation to the range of responses of Almost
Never, Seldom, Sometimes, Often and Always, one of the teachers commented:
While such responses are obviously easy to categorise and gather for statistical purposes, I question
whether the picture created by such responses is truly and completely indicative of the true landscape.
A questionnaire cannot describe the rich interrelationships and multitude of outcomes arising from the
interaction between the student and tutor, student and student, student and groups of students, and the
students and the software.
This suggests that combining the questionnaire results and qualitative data is de-
sirable for gaining a more meaningful sense of the learning environment (Tobin &
Fraser, 1998). Based on discussions with teachers, further modifications were made
to the questionnaire before it was used with another group of high school students.
Analyses of Students’ Questionnaire Responses
Following the online discussion with teachers about the educational value of the
questionnaire, the validity and reliability of the CMLES were investigated by analys-
ing data collected from the whole sample of 221 students in 12 classes. In order to
determine the internal structure of the 30 items in the instrument, we conducted factor
analysis. Table 2 shows the factor loadings obtained separately for the actual and pre-
ferred versions of the questionnaire. Principal components factor analysis followed
by varimax rotation confirmed the a priori structure of the instrument comprising 30
items with five items in each of the six scales.
Table 2 shows that the a priori factor structure was replicated perfectly for the
actual form of the CMLES except for Item 18. The factor loading for every item of
the actual version is at least .4 for its own scale for every item, and is less than .4 for
all other scales except for Item 18 for the Inquiry Learning scale.
For the preferred form, 26 items (with the exception of Items 10, 18, 21 and 23)
have a loading of at least .4 with their a priori scale (Table 2). Also, all items have a
loading of less than .4 with other scales, with the exception of Item 1 for Relevance,
Item 18 for both Inquiry Learning and Challenge, and Item 23 for Challenge.
The percentage of the total variance extracted for each factor is also recorded at
the bottom of Table 2. The total variance accounted for by the six factors is 63.40%
for the actual form and 62.21% for the preferred form.

230
DORIT
MAOR
AND
BARRY
J.
FRASER
Table 2
Factor Loadings for Actual and Preferred Forms of the Constructivist Multimedia Learning Environment Survey (CMLES).
Item Factor loading
No. Negotiation Inquiry learning Reflective thinking Relevance Complexity Challenge
Actual Prefer Actual Prefer Actual Prefer Actual Prefer Actual Prefer Actual Prefer
1 .51 .66 .41
2 .77 .76
3 .88 .74
4 .74 .79
5 .70 .67
6 .59 .45
7 .77 .70
8 .77 .72
9 .58 .44
10 .49 – .51
11 .76 .72
12 .81 .85
13 .78 .83
14 .80 .69
15 .82 .74
16 .76 .75
17 .70 .61

CONSTRUCTIVIST
MULTIMEDIA
LEARNING
ENVIRONMENTS
231
Table 2
Continued.
Item Factor loading
No. Negotiation Inquiry learning Reflective thinking Relevance Complexity Challenge
Actual Prefer Actual Prefer Actual Prefer Actual Prefer Actual Prefer Actual Prefer
18 .41 .43 .65 – .40
19 .77 .45
20 .62 .73
21 .58 .69 –
22 .83 .76
23 .81 – .46
24 .81 .86
25 .76 .82
26 .68 .59
27 .56 .53
28 .81 .70
29 .53 .80
30 .62 .77
% Variance 5.93 6.54 4.09 4.23 27.09 30.54 9.19 7.38 12.33 4.61 4.77 8.92
Alpha reliability .78 .82 .76 .73 .89 .87 .83 .77 .87 .78 .81 .83
Note: Loadings smaller than .4 omitted.

Table 2 also provides information about the reliability of both the actual and pre-
ferred forms of the CMLES based in its use with the 221 students in 12 classes.
Cronbach’s alpha reliability coefficient was used as an index of scale for internal
consistency. As shown at the bottom Table 2, the data collected support the reliability
of all scales for both actual and preferred forms of the CMLES. With the individual
student as a unit of analysis, the alpha reliability ranges from .73 to .87 for the actual
form and from .76 to .89 for the preferred form. This suggests that all scales of the
CMLES possess satisfactory internal consistency.
Descriptive Information for Whole Sample
To investigate differences in students’ perceptions of the preferred and actual
classroom environment, t-tests for matched pairs were conducted. Average item
means (i.e., scale means divided by the number of items in a scale) and average
item standard deviations are reported separately for the actual and preferred admin-
istrations for the six scales of the CMLES in Table 3. As in past research (Fisher &
Table 3
Average Item Mean, Average Item Standard Deviation, and Difference between
Actual and Preferred Forms (Effect Size and Paired t-Test Results) for Each CMLES
Scale (N = 221).
Scale Version Mean SD Effect size t
Negotiation Preferred 3.63 .65 .80 9.52∗
Actual 3.04 .80
Inquiry Learning Preferred 3.31 .69 .57 5.96∗
Actual 2.91 .72
Reflective Thinking Preferred 3.55 .90 .46 7.16∗
Actual 3.13 .89
Relevance Preferred 3.90 .79 .67 9.01∗
Actual 3.38 .76
Complexity Preferred 4.36 .79 1.19 13.69∗
Actual 3.38 .85
Challenge Preferred 3.58 .80 .79 9.67∗
Actual 2.90 .89
∗p < .001.
Effect size is (Preferred–Actual)/pooled SD.

Figure 1: Average item mean for actual and preferred version of each CMLES scale
(N = 221).
Fraser, 1983), the preferred mean was higher than the actual mean for each of these
six scales.
The results of the t-tests suggest that a statistically significant difference (p <
.001) between preferred and actual scores occurred for all CMLES. Effect sizes were
calculated using Cohen’s (1977) d formula involving the difference between two
group means for a scale divided by the pooled standard deviation. The effect sizes in
Table 3 suggest a substantial difference between actual and preferred on all learning
environment scales ranging from .46 to 1.19 standard deviations. Mean scores for
each scale of both the actual and preferred versions of the CMLES are presented
graphically in Figure 1.
An inspection of the scale means for the actual and the preferred forms indicates
that, although students generally perceive a high level of Negotiation, Inquiry Learn-
ing and Reflective Thinking, they still prefer an even higher level of these dimensions
(Figure 1).
In relation to the learning processes identified by the first three scales during the
interaction with the multimedia program, there was a general perception amongst the
students that only Sometimes were they engaged in Negotiation, Inquiry Learning
and Reflective Thinking (mean of approximately 3.05). However, students preferred
to be engaged between Often and Sometimes in discussion with other students on
how to conduct investigations (mean = 3.63) (see Figure 1) and preferred more
opportunities to find out answers to questions by investigations (mean = 3.32).
Likewise, students perceived that they Sometimes had opportunities for Reflective
Thinking (i.e., thinking deeply about how they learn) (mean = 3.14), but they still
preferred more opportunities (mean = 3.55) to think deeply about their own learning
and own ideas.

This pattern emerged also in the second part of the questionnaire which involves
students’ perceptions of their work with the online multimedia program. Students
generally indicated very high preferences for Relevance Complexity and Challenge.
The frequency of the average responses for Relevance and Complexity items are
close to and above Often, suggesting that students expected the multimedia program
Often to present information in a relevant way (mean = 3.91) and to be easy to
navigate (mean = 4.36). In practice, students perceive that more than Sometimes
the program initially presented the information in a relevant way (mean = 3.39) and
was easy to navigate (mean = 3.38). Their expectation from the program was quite
frequently (mean = 3.58) to make students think and to challenge their ideas, but in
practice students perceived that this happened only Sometimes (mean = 2.91).
Qualitative data based on students’ work and interviews with students, which did
not always agree with the CMLES results, suggest that students perceived working in
pairs as rewarding and, as a result, they were able to exchange ideas and thus enrich
their learning experiences.
Perceptions of Two Selected Students Relative to Their Class Mean Results
The first researcher worked closely with one of the teachers who enthusiastically
used the multimedia program while adopting a constructivist approach in his teach-
ing. He encouraged students to work in pairs, discuss their investigations and present
their findings to the class. From this group of Grade 10 students, two students, David
and Gerry, volunteered to work with the video camera and to express their views
to the researcher while interacting with the Logal science Web site while initially
working on the ecology topic.
Figure 2 compares the scores obtained on the CMLES by David and Gerry with
their class mean average. As illustrated in Figure 2, Grade 10 students in this class in
general perceived that only Sometimes did they have opportunities for Negotiation
and Inquiry Learning, and that more frequently they had opportunities for Reflective
Thinking. An interesting picture emerged when the class mean was compared with
the score of the two students.
Gerry perceived the learning environment in a similar way to his classmates,
whereas David perceived a higher degree of Negotiation and Inquiry Learning oc-
curring in the classroom. Both students perceived a high level of Reflective Thinking
(mean = 4.20), while their fellow students felt that they less than Often (mean =
3.40) reflect on their learning and think critically about their own and others ideas.
Observations in the classroom suggested that the students quite frequently are in-
volved in Negotiation. David appeared slightly more in control throughout the lesson,
directing its pace and sequence, and asking Gerry questions. David appeared to ask
more questions and to raised more uncertainties than Gerry. Prompted by David,
Gerry would then answer, and this frequently led David to ask more questions. The
following classroom observations supported David’s and Gerry’s perceptions of a
high level of the CMLES scales of Negotiation and Reflective compared to their
classmates:

Figure 2: Comparison of average item means for a class actual form of the CMLES
for the class as a whole and for two individual students.
Gerry and David both maintain high interest throughout the lesson. Whenever Gerry appears unsure,
David would confidently give him an answer, but Gerry wouldn’t agree immediately to David’s answer.
He would remain quiet for a while (giving an impression of being more reflective, I think), and then would
say “Ya, but why. . .?”, or “OK, but. . . .” Then, this leads on to more discussion among them. On the other
hand, when David appears unsure, Gerry wouldn’t give an immediate confident answer. Instead, he tends
either to give a tentative answer (in a sort of questioning manner), or to ask a question in return (relating to
David’s question). This style of interaction among these two boys seems to engage them in an interactive
manner of communication throughout the lesson, with the effect that they don’t seem distracted by the
noise from their classmates. But the interruption in the network sometimes causes them momentarily to
loose focus.
The two students perceived that only Sometimes were they engaged in Inquiry
Learning (e.g., “I find out answers to questions by investigation”) (Figure 2). They
also engaged in prediction and briefly discussed their answers before writing them
down.
These two students also reported that they more frequently engaged in Reflective
Thinking (e.g., “I get to think deeply about how I learn”). They were both engaged in
what they were doing. Gerry constantly verbalised his thought processes throughout
the lesson, while David asked more questions. Gerry appeared to be more reflective,
often taking time to think before giving an answer. This was consistent with the
questionnaire results from the CMLES.
Students’ reflective writing at the end of each session suggested that the science
program proved to be very informative. A different topic on population growth was
highly informative as well, but David suggested that he found it repetitive. The two
students discussed the Logal Web site and found some parts more challenging than
the others, which made the discussion more productive. As the students continued to
explore different topics on the Logal site, they found the teaching methods repetitive:
“Although the program is well put together, the teaching methods, I believe, should
be altered to make things a little more interesting.”

This supported the results of the CMLES and explains why the students perceived
the program to provide Challenge only Sometimes (Figure 2). During this activity,
the students were less engaged in discussion “as everything was quite explanatory
and fairly uninteresting.” For most of the activity session, Gerry was controlling the
mouse and David was typing. This could help to explain why Gerry perceived the
program to be easy to use almost Always (mean = 4.60), whereas David only Often
perceived the program to be easy to use and to navigate. They both perceived more
frequently (mean = 3.8) than their fellow students (mean = 3.4) that the program
presents relevant data in a meaningful way. It is interesting to note the small but
meaningful differences in relation to Challenge. As illustrated in the CMLES results,
both Gerry and David experienced more Challenge than other students. However,
David suggested that, although he learned a lot about the topic, the program didn’t
make him think or help him to generate new ideas.
Conclusion
This article reports the development and validation of a questionnaire (the Con-
structivist Multimedia Learning Environment Survey, CMLES) for assessing stu-
dents’ and teachers’ perceptions of constructivist multimedia learning environments.
The questionnaire also can be used by teachers for assessing the extent to which they
are successfully implementing a constructivist-oriented learning environment when
using multimedia in their classes. Information from administering the CMLES gives
an indication of whether or not students and teachers perceived opportunities for
social learning while they interact with inquiry-based online multimedia programs.
For a sample of 221 Grade 10 and 11 students who completed the actual and
preferred forms of the questionnaire, the six CMLES scales displayed satisfactory
factorial validity for both forms. Also the CMLES scales demonstrated a high degree
of internal consistency (with alpha reliability coefficients ranging from .73 to .87) for
both its actual and preferred forms. Overall, these findings support the validity and
reliability of the CMLES.
The results of paired t-tests for differences between the actual and preferred learn-
ing environment revealed statistically significant differences between the way in
which students perceived their actual and their preferred learning environment. Find-
ings for the CMLES were consistent with past research (Fraser, 1998a) in that stu-
dents’ preferred scores were higher than their actual scores. This suggests a need to
focus on changing pedagogical practices. Where lack of congruence is indicated,
sound educational criteria should be used to determine whether students should
reconstruct their expectations, or whether teachers should improve the learning en-
vironment, including examining carefully the quality of the multimedia before using
it in the classroom (McDougall & Squires, 1995).
However, these discrepancies between actual and preferred environment also partly
could be due to the repetitive nature of the specific multimedia program used in our
study. A major weakness that the teachers perceived with the CMLES was the lack of

open-ended questions that would enable them to elaborate on their responses. How-
ever, our study tried to overcome this problem partially by triangulating qualitative
data with quantitative data from the CMLES (see Erickson, 1986, 1998) to provide
an enriched framework for evaluating the effectiveness of the learning and teaching
processes in the classrooms.
In the study described in this article, results from the CMLES suggested that gen-
erally the students perceived that only Sometimes were they engaged in Negotiation,
Reflective Thinking and, to a slightly lesser degree, Inquiry Learning. However,
this contradicted qualitative data from a pair of students who suggested that quite
frequently they engaged in discussions with each other for the purpose of asking
questions and conducting investigations. This pair of students was also in agreement
that they were quite frequently involved in reflective thinking while interacting with
the online program and each other. Based on the whole sample, students believed
that they should be engaged more frequently in Negotiation, Inquiry Learning and
Reflective Thinking. There was also an apparent gap between students’ experiences
of and their expectations for the online program.
These results might be discouraging, but they also prompted us to reflect critically
on our pedagogical framework and the implementation of the online program in
the classroom. Problems such as immediate connection to the online programs via
the Internet, repetitive modules of teaching and a lack of challenge in the content
could explain some of the discrepancies between experiences and expectations as re-
flected in the CMLES results. This suggests that, although we are promoting a highly
constructivist learning environment by increasing opportunities for negotiation and
reflective thinking, the ease of use, the relevance of the program and a reasonably
higher level of challenge for the students are essential for promoting higher-level
learning.
In conclusion, the results from administering the CMLES, together with the qual-
itative data, suggest several further challenges for creating constructivist multimedia
learning environments. Students’ expectations and their perceptions have pointed
consistently to a need for still further improvement. In particular, improvement of
the technical aspects of the learning environment and the quality of the online mul-
timedia (especially in terms of relationships to real-life problems) also are likely to
enable students to engage in higher-order thinking skills and become more reflective
learners. The challenge for teachers in creating the new learning environment is to
use the computer as a cognitive tool (Jonassen & Reeves, 1996) by combining good-
quality multimedia programs with opportunities for critical thinking and higher-order
learning (Garbinger, 1996).
This study supports the potential of the CMLES for use in evaluating construc-
tivist-oriented multimedia learning environments. Further action research using the
CMLES could be aimed at identifying progress towards that goal and the use of
online programs. Furthermore, in order to enhance teachers’ use of a constructivist-
oriented approach to teaching and learning using computers in the classroom, the
CMLES can be used as a reflective tool. However, it would be desirable also to use
additional scales which focus on the specific quality and role of the multimedia, as

well as using the online version of the CMLES. Having validated the CMLES, it now
can be used in evaluating students’ perceptions of their social constructivist learning
environment while they use multimedia and/or online educational programs. It is
recommended that data obtained from using such questionnaires be combined with
qualitative data and outcomes of students’ learning (although it was not the purpose
of our study to examine students’ outcomes).
Because the Constructivist Multimedia Learning Environment Survey (CMLES)
provides a basis to judge quantitatively, as well as qualitatively, the effectiveness
of educational software in promoting inquiry learning, it is potentially valuable for
teachers and students for reflecting on their classroom environment. It can also be
used by teachers to help them to evaluate how close they are to achieving their
teaching goals and by students to provide a framework for evaluating their classroom
learning experiences.
Acknowledgements
This research was funded by the Australian Research Council (ARC) during the
period when the first author worked at Curtin University of Technology. The authors
thank Richard Cackett, the teacher who was involved in the study, the participating
school, students and teachers.
Correspondence: Dorit Maor, School of Education, Murdoch University, Murdoch,
WA, 6150, Australia
E-mail: dmaor@murdoch.edu.au
References
Barnea, N., & Dori, Y. J. (1996). Computerized molecular modelling as a tool to
improve chemistry teaching. Journal of Chemical Information and Computer
Science, 36, 629–636.
Bonk, C. J., & Cunningham, D. J. (1998). Searching for learner-centred, construc-
tivist, and sociocultural components of collaborative educational learning tools.
In C. J. Bonk & K. S. King (Eds.), Electronic collaborators: Learner-centred
technologies for literacy, apprenticeship and discourse (pp. 25–50). Mahwah,
NJ: Lawrence Erlbaum.
Bybee, R., & DeBoer, G. (1994). Research on goals for the science curriculum. In
D. Gabel (Ed.), Handbook of research on science teaching and learning (pp. 357–
387). New York: Macmillan.
Cobern, W. W. (1995). Constructivism for science teachers. Science Education
International, 6(3), 8–12.
Cohen, J. (1977). Statistical power analysis for the behavioral sciences. New York:
Academic Press.

Daniel, P. (1996). Helping beginning teachers link theory to practice: An interac-
tive multimedia environment for mathematics and science teacher preparation.
Journal of Teacher Education, 47, 197–209.
Edelson, D. C. (2001). Learning-for-use: A framework for the design of technology
supported inquiry activities. Journal of Research in Science Teaching, 38, 355–
385.
Erickson, F. (1986). Qualitative methods for the research on teaching. In M. C. Wit-
trock (Ed.), Handbook of research on teaching (3rd ed., pp. 119–160). New York:
Macmillan.
Erickson, F. (1998). Qualitative research methods for science education. In B. J.
Fraser & K. G. Tobin (Eds.), International handbook of science education
(pp. 1157–1173). Dordrecht, The Netherlands: Kluwer Academic.
Fisher, D. L., & Fraser, B. J. (1983). A comparison of actual and preferred classroom
environment as perceived by science teachers and students. Journal of Research
in Science Teaching, 20, 55–61.
Fraser, B. J. (1998a). Science learning environments: Assessment, effects and deter-
minants. In B. J. Fraser & K. G. Tobin (Eds.), International handbook of science
education (pp. 527–565). Dordrecht, The Netherlands: Kluwer Academic.
Garbinger, S. R. (1996). Rich environments for active learning. In D. H. Jonassen
(Ed.), Handbook of research on educational communications and technology
(pp. 665–693). New York: Macmillan.
Gordin, D. N., & Pea, R. D. (1995). Prospects for scientific visualization as an
educational technology. Journal of the Learning Sciences, 4, 249–279.
Hand, B., & Prain, V. (1995). Teaching and learning in science: The constructivist
classroom. Sydney: Harcourt Brace.
Harasim, L., Hiltz, S. R., Teles, L., & Turoff, M. (1996). Learning networks. London:
The MIT Press.
Harper, B., & Hedberg, J. (1997, December). Creating motivating interactive
learning environments: A constructivist view. Paper presented in the ASCILITE
conference, Perth, Western Australia. Retrieved January 19, 2004, from
http://www.ascilite.org.au/conferences/perth97/papers/Harper/
Harper.html
Hirumi, A. (2002). Student-centred, technology-rich learning environments (SCen-
TRLE): Operationalizing constructivist approaches to teaching and learning.
Journal of Technology and Teacher Education, 10, 497–537.
Horak, W. J. (1991, April). An analysis of metacognitive skills utilised by students
during computer simulation activities. Paper presented at the annual meeting of
the National Association for Research in Science Teaching, Fontane, WI.
Jonassen, D. H. (1994). Towards a constructivist design model. Educational Tech-
nology, 34, 34–37.
Jonassen, D. H., & Reeves, T. C. (1996). Learning with technology: Using computers
as cognitive tools. In D. H. Jonassen (Ed.), Handbook of research for educational
communications and technology (pp. 693–720). New York: Macmillan.

Kozma, R. B. (2000). Students collaborating with computer models and physical
experiments. In I. C. Hoadley (Ed.), Computer support for collaborative learning
(pp. 314–322). Mahwah, NJ: Erlbaum Associates.
Kozma, R. B., & Russell, J. (1997). Multimedia and understanding: Expert and
novice responses to different representations of chemical phenomena. Journal
of Research in Science Teaching, 34, 949–968.
Krajcik, J. S., Simmons, P. E., & Lunetta, V. N. (1988). A research strategy for the
dynamic study of students’ concepts and problem solving strategies using science
software. Journal of Research in Science Teaching, 25, 147–155.
Logal. (1999). Logal high school science gateways. [On-line]. Available: http://
www.riverdeep.net/demos/logal_science_demo.jhtml
Maor, D. (2001). Development of formative evaluation of a multimedia programme
using interpretive research methodology. Journal of Computers in Mathematics
and Science Teaching, 20(1), 75–97.
Maor, D., & Fraser, B. J. (1996). Use of classroom environment perceptions in evalu-
ating inquiry-based computer assisted learning. International Journal of Science
Education, 18, 401–421.
Maor, D., & Taylor, P. C. (1995). Teacher epistemology and scientific inquiry in
computerised classroom environments. Journal of Research in Science Teaching,
32, 839–854.
McDougall, A., & Squires, D. (1995). An empirical study of a new paradigm for
choosing educational software. Computer Education, 25(3), 93–103.
Nelson, W. (1994). Efforts to improve computer-based instructions: The role of
knowledge representation and knowledge construction in hypermedia systems.
Computers in the Schools, 10, 371–399.
Newman, D. (1993). School networks: Delivery or access. Communications of the
ACM, 36(5), 49–51.
O’Connor, M. C. (1998). Can we trace the efficacy of social constructivism? Review
of Educational Research, 23, 25–71.
Orion, N., Dubowski, Y., & Dodick, J. (2000). The educational potential of multime-
dia authoring as a part of the earth science curriculum – A case study. Journal of
Research in Science Teaching, 37, 1121–1153.
Parril, A., & Gervay, J. (1997). Fostering curiosity-driven learning through inter-
active multimedia representations of biological molecules. Journal of Chemical
Education, 74, 1141–1142.
Peat, M., Franklin, S., & Lewis, A. (2003). Learning-centred evaluation of CFL
projects in higher education. Retrieved January 19, 2004, from http://www.
tlc.murdoch.edu.au/project/cutsd01.html
Schank, P., & Kozma, R. (2002). Learning chemistry through the use of a
representation-based knowledge building environment. Journal of Computers in
Mathematics and Science Teaching, 21, 253–279.
Songer, N. B., Lee, H., & Kam, R. (2002). Technology-rich inquiry science in urban
classroom: What are the barriers to inquiry pedagogy? Journal of Research in
Science Teaching, 39, 128–150.

Tasker, R., Daiton, R., Sleet, R., Bucat, B., Chia, W., & Corrigan, D. (2002).
Description of VisChem: Visualising chemical structures and reactions at the
molecular level to develop a deep understanding of chemistry concepts. Retriev-
able from the Australian Universities Teaching Committee (AUTC) Learning
Designs Web site: http://www.learningdesigns.uow.edu.au/exemplars/
info/LD9/index.html Last accessed on Tuesday, 16 November 2004.
Taylor, P. C., Fraser, B., & Fisher, D. (1997). Monitoring constructivist classroom
learning environments. International Journal of Educational Research, 27, 293–
302.
Taylor, R. (Ed.). (1980). The computer in the school: Tutor, tool, tutee. New York:
Teachers College Press.
Tobin, K. G. (Ed.). (1993). The practice of constructivism in science education.
Washington, DC: AAAS Publications.
Tobin, K. G. (1998). Issues and trends in the teaching of science. In B. J. Fraser &
K. G. Tobin (Eds.), International handbook of science education (pp. 129–151).
Dordrecht, The Netherlands: Kluwer Academic.
Tobin, K., & Fraser, B. J. (1998). Qualitative and quantitative landscapes of class-
room learning environments. In B. J. Fraser & K. G. Tobin (Eds.), International
handbook of science education (pp. 623–640). Dordrecht, The Netherlands:
Kluwer Academic.
Tobin, K. G., & Gallagher, J. (1987). What happens in high school science
classrooms? Journal of Curriculum Studies, 19, 549–560.
Tobin, K. G., & Tippins, D. (1993). Constructivism as a referent for teaching
and learning. In K. G. Tobin (Ed.), The practice of constructivism in science
education (pp. 3–23). Washington, DC: AAAS Publications.
Wu, H., Krajcik, J., & Soloway, E. (2001). Promoting understanding of chemical
representations: Students’ use of a visualization tool in the classroom. Journal of
Research in Science Teaching, 38, 821–842.
Appendix A
Constructivist Multimedia Learning Environment Survey (CMLES)
Student Actual Form
What actually happens in my classroom
DIRECTIONS
1. Purpose of the Questionnaire
This questionnaire asks you to describe important aspects of the classroom which
you are in right now. There are no right or wrong answers. Your opinion is what
is wanted. Your answers will enable us to improve future science teaching.

2. How to Answer Each Question
On the next few pages you will find 30 sentences. For each sentence, circle
only one number corresponding to your answer. For example:
Almost Seldom Some- Often Always
Never times
In this class. . .
8 I ask the students questions 1 2 3 4 5
• If you think that you always ask the students questions, circle the 5.
• If you think that you almost never ask the students questions, circle the 1.
• Or you can choose the number 2, 3 or 4 if one of these seems like a more
accurate answer.
3. How to Change Your Answer
If you want to change your answer, cross it out and circle a new number. For
example:
8 I ask the students questions 1 2 3 4
5

×
4. Course Information
Please provide information in the box below. Please be assured that your answers
to this questionnaire will be treated confidentially.
a. Name: b. School:
c. Grade/Year-level: d. Sex: male/female
(please circle one)
5. Completing the Questionnaire
Now turn the page and please give an answer for every question.
Part I: The Process of Learning with the Multimedia Program
Please select how often the following learning activities actually DO occur in your
classroom.

Learning to Communicate Almost Seldom Some- Often Always
Never times
In this class. . .
1 I get the chance to talk to other 1 2 3 4 5
students
2 I discuss with other students 1 2 3 4 5
how to conduct investigations
3 I ask other students to explain 1 2 3 4 5
their ideas
4 other students ask me to explain 1 2 3 4 5
my ideas
5 other students discuss their ideas 1 2 3 4 5
with me
Learning to Investigate
In this class. . .
6 I find out answers to questions 1 2 3 4 5
by investigation
7 I carry out investigations 1 2 3 4 5
to test my own ideas
8 I conduct follow-up investigations 1 2 3 4 5
to answer new questions
9 I design my own ways 1 2 3 4 5
of investigating problems
10 I approach a problem 1 2 3 4 5
from more than one perspective
Learning to Think
In this class. . .
11 I get to think deeply about how I learn 1 2 3 4 5
12 I get to think deeply about my own 1 2 3 4 5
ideas
13 I get to think deeply about new ideas 1 2 3 4 5
14 I get to think deeply how to become 1 2 3 4 5
a better learner
15 I get to think deeply about 1 2 3 4 5
my own understandings

Part II: The Multimedia Program
Please select how often each of the following statements actually IS experienced in
your classroom.
Relevance Almost Seldom Some- Often Always
Never times
Working with the multimedia program,
I find that it. . .
16 shows how complex real-life 1 2 3 4 5
environments are
17 presents data in meaningful ways 1 2 3 4 5
18 presents information that is relevant 1 2 3 4 5
to me
19 presents realistic tasks 1 2 3 4 5
20 H as a wide range of information 1 2 3 4 5
Ease of Use
I find that it. . .
21 has an interesting screen design 1 2 3 4 5
22 is easy to navigate 1 2 3 4 5
23 is fun to use 1 2 3 4 5
24 is easy to use 1 2 3 4 5
25 takes only a short time to learn how 1 2 3 4 5
to use
Challenge
I find that it. . .
26 makes me think 1 2 3 4 5
27 is complex but clear 1 2 3 4 5
28 is challenging to use 1 2 3 4 5
29 helps me to generate new ideas 1 2 3 4 5
30 helps me to generate new questions 1 2 3 4 5

An Online Questionnaire For Evaluating Students And Teachers Perceptions Of Constructivist Multimedia Learning Environments.

More Related Content

Similar to An Online Questionnaire For Evaluating Students And Teachers Perceptions Of Constructivist Multimedia Learning Environments. (20)

More from Simar Neasy (20)

Recently uploaded (20)

An Online Questionnaire For Evaluating Students And Teachers Perceptions Of Constructivist Multimedia Learning Environments.