Models and instruments for assessing Technology Enhanced Learning Environments in higher education

In-depth
Models and instruments for assessing Technology
Enhanced Learning Environments in higher
education
Authors The Bologna Process calls for a substantive change in the pedagogical model of teach-
ing and learning in higher education, focusing on the acquisition of skills by students
Sérgio André Teixeira
and not the mere accumulation of knowledge. Technology Enhanced Learning Environ-
Ferreira
ments (TELE) are seen as a fundamental support in teaching reengineering, and may
Teacher at Escola Básica e
Secundária das Flores support a more effective approach to constructive educational philosophies.
sergioandreferreira@gmail. The evaluation of TELE, as a means of certifying its quality, is giving rise to several
com
initiatives and European experiences. However, the mechanisms for defining quality
António Manuel Valente parameters vary according to different contexts. If assessment aims to function as a
de Andrade management tool, it should seek specific criteria and indicators that would allow it to
Senior Lecturer at the respond to questions of well-defined contexts. In this study, which stems from a litera-
School of Economics and ture review, we present basic guidelines for TELE continuous assessment (as a manage-
Management of the Catholic ment tool). Throughout this article the importance of ongoing, in-context evaluation
University of Portugal
is emphasized. Models, methods and tools to collect data that permit institutions to
aandrade@porto.ucp.pt
develop a properly contextualized assessment process are presented.

Tags

technology enhanced 1. Technology Enhanced Learning Environments (TELE)
learning environment,
quality guidelines, evaluation
delimitation of the concept
methods The concept Technology Enhanced Learning Environments (TELE) is comprehensive and,
therefore, it is not easy to define nor does it have a single definition. In the Report of the
Technology Enhanced Learning Committee, University of Texas (2004, p. 182) it is defined as
follows:

“Technology enhanced learning leverages technology to maximize learning within
an environment of sound course design that can offer students the options of time,
place, and pace and emphasizes different learning styles.

There is no one definition for the look or feel of a technology enhanced course;
instead, this effort occurs along a very broad spectrum that at one end can include
a course with only minimal technology enhancement such as a Web site with an
electronic syllabus, while at the opposite end is found a robust, multimedia rich,
interactive, collaborative, fully online course” (Armstrong, et al., 2004).

In defining the concept of TELE rather than seek formulations or rigid definitions, we sought
to establish common views that facilitate communication. Thus, following the approach in
the University of Texas’ Report, we understand that a TELE goes beyond technology related
issues and focuses on building learning environments supported, in more or less detail, and
more or less integrated by technology.

ing
earn
eLearning Papers • ISSN: 1887-1542 • www.elearningpapers.eu
eL ers
24
u
ers.e
gpap
www
.elea
rnin n.º 24 • April 2011
Pap
1

In-depth

2. The growing importance of the a more constructive approach. The TELE based on LMS are
Technology Enhanced Learning advocated as being able to support more effectively the
Environments (TELE) in higher education knowledge construction in higher education;
• The TELE as support in pedagogical reengineering. The ICT,
The development of TELE efficient assumes a fundamental stra-
as educational technologies, are advocated as being able
tegic importance in the competitiveness of Higher Education In-
to support more effectively the construction of knowledge
stitutions (HEIs). Figure 1 represents a gear, with the main social
and the promotion of meaningful learning, particularly if
factors, which are pressing the HEIs for change, through tech-
used as cognitive tools to expand mental capacities and not,
nology. The capacity of the institutions understand the change
as traditionally happens, just as a means of transmission of
and formulate strategies to adapt to the new environment is
information, as communicators of knowledge or guardians
crucial to its success and survival. This situation is similar to the
of students;
Darwinian Theory of Evolution by Natural Selection: “It is not
the strongest of the species that survives, nor the most intel- • The growing importance of social networks and collabora-
ligent that survives. It is the one that is the most adaptable to tive work performed by collective intelligence has height-
change” (quote attributed to Darwin that summarizes the theo- ened the value of Social Learning Networks (Tapscott & Wil-
ry of evolution by natural selection). liams, 2008). As such it has become necessary for IES’s to
evolve from LMS architecture to Hybrid Institutional Per-
sonal Learning Environment (HIPLE) architecture, thus
serving as a bridge between the vision held by the
institution and by the student.
• With the implementation in Higher
Education of the Bologna reform,
there will predictably be, a signifi-
cant part of the student population
that will tend to be less available to
have a face to face full-time education
after completion of primary school.
The HEIs will be able to reap dividends if
they offer combined ways of learning (b-
learning) and the distance which enables
extending the range of potential candidates
for middle and secondary school. Even in
classroom learning, the online availability of
some of the syllabus or the virtual extension of
the classroom, may allow a greater monitoring
of the teaching activity by students, who for professional or
Figure 1: Social factors that require the change of HEIs
through technology other compelling reasons, see their presence in the class
compromised;
• The deepening of inter-institutional cooperation of HEIs for
• The Bologna Process calls for a substantive change in the
different countries, the projection of movement of people
pedagogical model of teaching and learning in higher edu-
(teachers, researchers, students and administrative staff)
cation, focusing on skills acquisition by students and not the
and integrated programs of study, training and research,
mere accumulation of knowledge. In other words, it is not
foreseen by the Bologna declaration may be favored if there
just about learning concepts, which will then be assessed;
is a basis for work online to facilitate them;
the students will have to acquire skills themselves, and
• The decline in birth rate, which is reflected in the decreas-
therefore they will be co-responsible for their own educa-
ing number of students, coupled with the growing mobility
tion. This philosophy leads us to pedagogical models with

ing
earn
eL ers
24
u
ers.e
gpap
www
.elea
Pap
2

In-depth

of the student population throughout the course of their examples in literature that seek to contribute to the quality as-
training, fosters competition between IES’s. The develop- surance of TELEs. Over the next two chapters, we will briefly
ment of efficient TELE´s may be a determining factor in the review some relevant points.
attractiveness of the institutions;
• Considering that the globalization of the sources of knowl- 3. Quality evaluation and improvement of
edge takes the world to school, it can additionally be con- practices: a growing concern
sidered that notebooks, netbooks, tablets, smartphones
The issue of TELE, supported by ICT, education is one of the
and other alike can take school anywhere in the world. In
most relevant topics in discussion. Many hopes have been de-
this fashion a new frontier of Personal Learning Environ-
posited in its potential to serve education, but many have also
ments (PLE) has been opened, enabling students to control
been the failures in terms of results. In the words of (2006, p.
and manage their own learning experience;
434): “Trillions of dollars are annually spent on the develop-
• The increasing need for lifelong training requires models of
ment and implementation of information technology within the
distance learning or a combination of classroom/distance
United States and around the world. On average, roughly 50%
learning;
of such systems are considered failures or fall short of meeting
• The penetration of technology into all human activities is a the expectations set forth by management.”
reality, and IES’s that fail to make the change via technology
will be less competitive. In view of the American Society for Quality1, quality is what
defines and guides the individual success of organizations and
If it is true that an investment in technology does not necessar- communities without interrupting their process of evolution.
ily lead to the construction of efficient TELE, then it is also true This is an ongoing process of development and pursuit of the
that an investment in technology does not necessarily signify best practices. Quality can only be improved if there is an evalu-
the construction of an intelligent TELE (Lippert & Davis, 2006). ation of practices
Rosenberg (2006) considers that the lack of assessment of vari-
ous aspects of TELEs is one of the factors responsible for the According to the European Quality Observatory2:
failure of these initiatives.
“Quality in e-learning has a twofold significance in Europe:
Khan and Granato (2007) argue that, in order to understand a first, e-learning is associated in many discussion papers and
TELE, it is necessary to consider multiple dimensions: people, plans with an increase in the quality of educational oppor-
processes and products. In his own words: “To understand tunities, ensuring that the shift to the information society
online learning environment, we need to have a comprehen- is more successful. We call this context ‘quality through e-
sive picture of people, process and product involved in it, and learning’. Second, there is a separate but associated debate
also study critical issues encompassing its various dimensions” about ways of improving the quality of e learning itself.
(Khan & Granato, 2007). We term this context ‘quality for e-learnin.”(Ehlers, Goertz,
Hildebrandt, & Pawlowski, 2005, p. 1).
Following the same line of thought, Ehlers and Goertz (2005)
wrote that: “It is necessary to regard all factors of influence – The evaluation of the quality and the improvement of practices
the learner, the subject, the intended results, the technologi- are the goals of the European Foundation for Quality in E-Learn-
cal and social surrounding (work place, learning culture in the ing (EFQUEL)3, a European organization whose main objective
company, private learning situation etc.). An effective quality is to promote the quality of e-learning in Europe through sup-
assurance has to cover the whole process from the first plan up port services to institutions and agents involved in e-learning
to the development and implementation until the assurance of in general. This foundation has created a European observatory
transfer” (Ehlers & Goertz, 2005, p. 169) of quality in e-learning, projects, workgroups and publications
of interest, as well as the creation of a European prize for the
Many European and international institutions emerged to try
and meet this challenge of change which has been imposed 1 http://www.asq.org/
upon Institutions. It is essential that the process of change is 2 http://www.virtualcampuses.eu/index.php/European_Quality_Ob-
accompanied by quality assurance initiatives. There are several servatory
3 www.efquel.org

ing
earn
eL ers
24
u
ers.e
gpap
www
.elea
Pap
3

In-depth

eQuality. Among the EFQUEL projects, we would like to high- 4. Evaluation Models
light eUnique4, which involves several research centers and Eu-
In literature there have been many proposals for the global eval-
ropean universities, and aims to create a tool that will help Eu-
uation of the use of technology in education. Under e-learning,
ropean universities to maximize the educational use they make
the classic model of Kirkpatrick is widely applied. Kruse (2002)
of technology.
recovers the classical model of the four levels of evaluation of
There are some other European associations promoting Kirkpatrick (1975) and adapts it to the evaluation in e-learning
projects, seminars and specific publications about the quality of (table 1).
e-learning, making some proposals about how to evaluate the
Criteria Indicators of quality
TELE, such as: The European Distance and online-learning Net-
work (EDEN)5 and the European Association for Distance Teach- • Quality indicator 1: How are the
objectives of a training program
ing Universities (EADTU)6.
prepared?
Criteria I
• Quality Indicator 2: How can we
Although in literature there is a consensus that evaluation is Teaching Program
facilitate the curricular flexibility
an aspect of capital importance in the development of TELE, in while answering to the objectives
particular in e-learning courses, the way how we assess them of the training program?
and what aspects should be examined are issues of great con- • Quality Indicator 3: How to
troversy: “How to evaluate e-learning appropriately is thus the plan actions for continuous
Criteria II
improvement?
crucial question for researchers trying to understand the impact Teaching and learning
• Quality Indicator 4: How
and effectiveness of e-learning in a business or academic envi- organization
to establish effective
ronment” (C. Voigt & Swatman, 2004). communication?
• Quality Indicator 5: How to
engage teachers in research,
Level Evaluation Parameters Criteria III
development and innovation?
Human resources
• Are the course objectives • Quality Indicator 6: How to value
relevant? teaching?
• Does the course have the ability • Quality Indicator 7: How
to maintain interest? to articulate the library and
• Is the Syllabus appropriate and document databases with the
Level 1 Criteria IV
do they have the interactivity educational process?
Satisfaction Material resources
needed? • Quality Indicator 8: Are the means
• Is it easy to navigate? and resources adapted to the
• Are the value and the possibility training program?
of transferring knowledge to the
workplace perceived? • Quality Indicator 9: How to
develop students’ skills in the
• Did the students achieve the teaching and learning process?
Level 2
objectives (knowledge, skills and • Quality Indicator 10: What is the
Learning Criteria V
attitudes) required in the program? methodology of the teaching and
Training process
• To what extent is the new learning process?
Level 3 • Quality Indicator 11: How does
knowledge and skills applied in
Skills the tutor guide and motivates the
the workplace?
student during the training?
• Does the knowledge acquired in
Level4 • Quality Indicator 12: How do you
training have an impact on the Criteria VI
Results measure student satisfaction in
company’s business? Results
the training process?
Table 1: Evaluation in e-learning: Introduction to the Kirkpatrick
model (Kruse, K. 2002) Table 2: Criteria and indicators of MEPFL quality (Model of
Excellence Programs for Online Training) (Rosa & Angulo 2007)

4 http://www.qualityfoundation.org/
5 http://www.eden-online.org/ The academic environments still in transition of processes, from
6 http://www.eadtu.eu/ one centered on classroom learning and teaching, to a mixed

ing
earn
eL ers
24
u
ers.e
gpap
www
.elea
Pap
4

In-depth

Axes Levels
• One access per semester
Access Dynamics • One access per month
What is the necessary frequency of access to • One access per week
succeed in the course? • Two or three accesses a week
• Daily
• None
• Reduced
Evaluation
• About half
What is the volume of online assessment?
• Majority
• All assessment
• Reduced
• Less than half
Communication
• More than half
What is the volume seen in online communication?
• Most
• All communication
• Information about CU (program, evaluation, objectives, teaching staff, etc.)
Syllabus (elements) • Book, or supporting text
What is the content of the Curriculum Unit (CU) • Lectures material from lectures (theoretical and practical)
available online? • Exercises (cases, solutions, exams, etc.)
• Additional information (one point for each)
• Simulators (games and animations)
Syllabus (Digital wealth) • Video (specific, youtube)
Does the content take advantage of the digital • Audio (podcast)
environment? • Slides (class slides)
• Additional Sites (one point for each)
• Essentially dependent on presence
Independence • Significant lessons, but online materials, exercises and evaluation
What is the degree of independence of the CU • Limited regular face-to-face contact
model in comparison with the traditional model? • Sporadic face-to-face contact
• No face-to-face contact

Table 3: Adaptation of the Kaczynski, Wood and Harding’s (2008) Model

education and with increasing use of technology, combined uum between diagnostic assessment (Where are we? The sta-
with a survey of pedagogical changes, require a quick and easy tus) and prognostic (where do we want to go? objectives to be
to apply model for monitoring. Table 3 identifies in detail a pos- achieved). This is a process of continuous assessment-training in
sible approach to this simple system of monitoring the migra- which the different actors involved participate. (Gomes, Silva, &
tion from a traditional system to a system that incorporates Silva, 2004, pp. 1,2)
technology in teaching and learning.
The Matrix Integration Technology (MIT) developed by the Flor-
For everything that was said, the assessment is quite a relevant ida Center for Instructional Technology (2009), illustrates the
topic and essential in creating a participatory dynamic of con- five stages of organizational change to achieve the highest level
struction, adaptation and continuous appreciation of TELE. The of change in a TELE. Ideally, a TELE will evolve from an introduc-
process of assessment will therefore have to consider the par- tory phase (entry) to a stage of transformation (transformation),
ticipation and negotiations between the various actors involved which is reached when the technology is used effectively by eve-
in that same process (designers, trainees, trainers...) and should ryone in the organization and the culture of knowledge sharing
result in a continuous reflection on the objectives, if they were is the rule. In the transformation phase, the whole TELE is im-
met or not, about what is necessary to redefine, what is impor- mersed in technology and does not work without it. A produc-
tant to keep and what may be important to change, in a contin- tion of participatory evaluation is essential in the construction,

ing
earn
eL ers
24
u
ers.e
gpap
www
.elea
Pap
5

In-depth

adaptation and evolution of TELE to growing levels of integra- indicators of a given institution with difficulty, so its effective-
tion. In Figure 2, we have constructed the plan of the evolution ness as an instrument of management and of improvement of
of technology integration in the curriculum, based on the MIT, practices in the institution, will be limited.
proposed by the Florida Center for Instructional Technology.
In evaluating TELE it is possible to distinguish models of partial
and global focus.

In models of global focus, we can
discriminate trends:

• The evaluation of systems
focusing on models and / or
quality standards;
• Systems based on the practice
of benchmarking.

The models of partial focus are
centered on issues such as:

• Training activity;
• Training syllabus;
• Technologic platforms;
• Cost/benefit analysis

Although we can integrate con-
tinuous evaluation as a manage-
ment tool into the category of
partial focus evaluation, its goal is
to examine in depth all the com-
ponents of TELE from the per-
spective that all actors somehow
participate in the environment. To
Figure 2: Levels of technology integration in the curriculum meet this goal, the evaluation should have the following char-
(Florida Center for Instructional, 2009) acteristics:

• Circular evaluation (360⁰) - All players and components of
the environment are evaluated by all participants. Each ac-
5. Evaluation methodology tor evaluates the responsibility aspects of the other actors,
The evaluation of the TELE, as a means of certification of its and simultaneously is evaluated by peers. This methodology
quality, is giving rise to several initiatives and European experi- involves the development of a multidirectional evaluation
ences, some already mentioned. However, the mechanisms for that overcomes the limitations of a one-way evaluation, as
parameterization of quality vary according to each context (Tait, it allows you to compare various aspects of the environ-
1997), so if the evaluation has the aim to perform functions ment, from different perspectives. (Acuña & Aymes, 2010;
of a management tool, it will have to seek specific criteria and Fernández, 1997)
indicators, which respond to questions of delimited contexts • Evaluation of procedure - During the process, as we contin-
(Rubio, 2003). In other words, the import and direct application ue to develop the different modules of the courses, there is
of an evaluation model will contemplate variables, criteria and a collection of data and from the evaluation that is made de-

ing
earn
eL ers
24
u
ers.e
gpap
www
.elea
Pap
6

In-depth

cisions must be taken to improve the TELE. Training Evalua- • Comparative Studies - Comparison between different sub-
tion. jects, courses and schools to identify potentials and bottle-
• Final Evaluation – The training evaluation prepared during necks of the TELE.
the process aims not only to contribute to the improvement • Students’ evaluation results – They can offer a big variety of
of the TELE, but also to enable the final results at the course information on learning outcomes.
and organization levels (summative assessment) as an aid to • User tracking – Tracking the students’ activities enables us
decision making. to capture differences in participation rates and in use of
• Mixed focus evaluation - Resorting to qualitative instru- resources.
ments (interviews, focus group) we expect to deepen and • Charts of direct observation - Comparison between the
get contextual information that enables us to identify the characteristics of the course criticism strands (tutoring, ma-
dimensions and factors of the integration of the TELE. The terials, activities) and the ones desired.
information extracted from the qualitative analysis will
• Online Learning Environment Survey (OLES) - In the perspec-
serve as input for the construction of quantitative instru-
tive of research the OLES evaluation system (http://www.
ments (questionnaires). The data is collected, analyzed and
monochrome.com.au/oles/survey.htm) is anchored to the
processed in the quantitative phase (dynamic of access,
environment assessment of the classroom, in dynamics of
communication volume, nature and wealth of digital con-
distance education and the contributions of technology in
tent, ...).
teaching and learning. This method comprises nine dimen-
sions (1 - Computer Usage, 2 - Teacher Support, 3 - Student
6. Data collection tool Interaction and Collaboration; 4 - Personal Relevance, 5 -
In a continuous evaluation, which is intended to be a manage- Authentic Learning, 6 - Student Autonomy; 7 - Equity EQ;
ment tool, data collection should be made according to a pre- 8 - Enjoyment and 9 - Asynchronicity ), perfectly framed in
established plan. We have already mentioned the potential of constructivist philosophy, and fifty-four factors on a scale
a participatory and multidirectional assessment. If the aim is to of five levels that explains not only the current perception
provide decision makers with accurate and detailed informa- of students in relation to the system under study, but also
tion, it will be important to take a holistic view of reality. Data seeks to project its expected ideal (Pearson & Trinidad,
collection should be done in a real context, and given the com- 2005). This way, it is possible to foresee the differences be-
plexity of the phenomenon, it requires the collection of varied tween the current state and the desired one. This system
data, hence this type of approach is similar to a case study when can be used online, by creating an account for the effect,
seen as research methodology (Yin, 2009). which consists of the generation of an address for the ac-
cess of teachers and students, addressing both perspec-
Harvey, Cathy Higgison and Gunn (2000) list a number of tools tives (teacher and student.) The OLES latest version can be
to collect data for the evaluation of TELE: entirely administered online by the teacher, with charts to
be automatically produced for discussion and comparison.
• Questionnaires – They are recording instruments planned
This view of assessment is perfectly aligned with the multi-
to search for data about subjects, through questions about
directional methodologies of evaluation participated by all
knowledge, attitudes, beliefs and feelings (Wood. & Haber,
actors, allowing, at each moment, to the gathering of rele-
2001).They can be useful tools to obtain feedback from all
vant information on the operation of the course, becoming,
the participants in the TELE.
therefore, great management tools.
• Online discussions - online discussion forums on various as-
pects related to the operation of the course.
• Interviews and focus groups - Interviews and meetings in 7. Presentation of results
small or medium groups (between five and ten people), in The results of the evaluation process should have an impact on
which participants talk in a relaxed and informal environ- the educational environment, in relation to content, attitudes
ment (Sampiere, Collado, & Lucio, 2006). of students and teachers, in the way of content delivery and
• Peer review - Sharing of views between tutors on content, in a more efficient integration of technology (Moussiades & Ili-
activities and communication structures.

ing
earn
eL ers
24
u
ers.e
gpap
www
.elea
Pap
7

In-depth

opolou, 2006). Thus, regardless of how it is conducted, it is im- • Narrative accounts of the evaluation;
portant to present the evaluation results.
• Presentations, embedding oral descriptions;
In 1996, Leading Change Kotter writes, the best-selling book in • Poster of findings;
the last decade about change in organizations. Leading Change
• Research reports;
was based on extensive research, which identified and defined
a standard eight steps associated with change initiatives with a • Spreadsheets.
high success rate. In eight steps, a methodology capable of deal-
We have already mentioned the potential of tools such as OLES,
ing with big changes is defined: to create new and improved re-
which facilitate the collection of relevant information and the
lations, to encourage greater growth, to eliminate weaknesses,
producing of charts just in time. Thus, the results are immedi-
to improve the quality of products and services or to restructure
ately available for discussion and action.
the human resources and processes. Kotter (1996) considers
that disclosure of evaluation results is a key point in developing We have also highlighted the potential of the radar chart in the
the strategy for change, as it allows to: presentation of results. The American Society for Quality defines
radar chart as follows: “A graph with multiple scales to report
• Create some visible, unambiguous successes quickly, which
self-assessed knowledge or competence, often several points
will serve as incentive for the implementation of the strat-
in time. A Radar Chart is used to identify current level of self-
egy;
assessed knowledge or competence, and then monitor change
• Remove barriers, so that those who want to make the vision or growth across several factors” (American Society for Quality,
a reality can do so; 2006, p. 375). Radar charts due to their versatility in represent-
• Carry on on a more intense and faster way after the first ing knowledge, are often used in the analysis of organizational
successes. Successive changes enable the new production development and quality measurement (Kaczynski, et al., 2008).
to become a reality;
Thus, in an academic environment still in transition of processes
• The promotion of new forms of conduct and the certainty in which there is an increasing use of technology, combined
that the successful cases are published and become strong
enough to replace the traditional ones.

The publication of results is, according Moussiades and Ili-
opolou (2006, p. 182), a crucial factor in the success
of the evaluation. In his words: “A possible
reason for an evaluation method to be un-
successful is that the results of the evalua-
tion research are presented in a way that is
not comprehensible. Thus the interested
parties don’t bother to look at them and
to take any action to repair the malfunc-
tioning components of a learning envi-
ronment”:

Moussiades and Iliopolou (2006) sum-
marize some of the most common
ways for presentation of results:

• Data sets;
• Executive summary of the activ- Figure 3: Radar Charts - Relationship between the
ity to be taken; characteristics of the learning environment and the Levels
of Technology Integration

ing
earn
eL ers
24
u
ers.e
gpap
www
.elea
Pap
8

In-depth

with a survey of pedagogical changes, the radar charts are ap- We believe that continuous assessment, as a management tool,
propriate and adaptable. In Figure 3, is shown the versatility of should examine in depth all the components of the TELE, from a
the radar charts: here are represented the characteristics of the multidirectional perspective to 3600, in which each player gives
learning environment (Active, Collaborative, Constructive, and an opinion about the various components of the TELE, evaluates
Goal Directed Authentic) as well as the levels of technology in- and is evaluated by other actors. We recognize it as virtues of a
tegration in the curriculum for each of the characteristics (Entry, model of cross-sectional evaluation and participated by all: the
Adoption, Constructive, Infusion and Transformation), available availability of current information, contextualized and meaning-
under the Technology Integration Matrix (TIM) developed by ful for all participants.
the Florida Center for Instructional Technology (2009).
The model ALACT (Korthagen & Vasalos, 2005) is a concept of
the reflection process, in which a restructuring dialectic of ex-
Conclusions perience and knowledge is visible. The restructuring of the ex-
The TELE has a major strategic importance in the competitive- periences and knowledge is a cyclical process that results from
ness of Institutions of Higher Education and is a central sup- a process and multifaceted evaluation of the reality and, in this
port to pedagogic reengineering. However, the implementation sense, it synthesizes the main conclusions of this study (Fig. 4)
of TELE does not necessarily mean a gain or a break with the
After the action (1), steps 2, 3 and 4 illustrate how the continu-
educational models of the past, and therefore it is important
ous assessment can work as a tool for management and im-
to develop evaluation models that are the guarantor of quality
provement of practices:
systems and promote the improvement of practices.
• Step 2: Looking back of the action (What is happening?)
Several institutions and researchers have proposed assessment
models that focus on the problem of TELE assessment. These • Step 3: Awareness of essential aspects (What are the posi-
contributions are important assets in quality control. However, tive and negative aspects of the TELE?)
the quality parameterization mechanisms vary according to • Step 4: What will I determine for the next time? (What are
context. It is therefore important that HEIs are able to imple- the alternatives for improvement? What advantages / dis-
ment these contributions, choose the best tools to collect data advantages have they got?)
and present the data effectively. Starting with a revision of the
existing literature, in this study we seek to address these issues. The evaluation will be a management tool, an effective promot-
er in improving practices, if it is based on notions of progress,
change, adaptation and rationalization.

References
Acuña, S., & Aymes, G. (2010). Evaluación de la calidad en
educación virtual: aportes para una metodología. Paper presented
at the I Congresso Iberoamericano sobre Calidade de la Fomación
virtual – CAFVIR, Alcalá de Henares, España.

Armstrong, N., Ashcroft, J. C., Bruce, R. G., Buskirk, R.,
Cullingford, E., Davis, P., et al (2004). Report of the Technology
Enhanced Learning Committee. Austin: The University of Texas at
Austin.

C. Voigt, & Swatman, P. (2004). Contextual e-learning
evaluation: a preliminary framework. Learning, Media and
Technology, 29(3), 175-187.

Ehlers, U.-D., & Goertz, L. (2005). Quality in e-learning: Use
Figure 4: The ALACT model (Korthagen & Vasalos, 2005) and Dissemination of Quality Strategies. A Study by the European

ing
earn
eL ers
24
u
ers.e
gpap
www
.elea
Pap
9

In-depth

Quality Observatory (pp. 157-169). Luxembourg: Office for e-Learning%20Introduction%20to%20the%20Kirkpatrick%20
Official Publications of the European Communities. Model.htm

Ehlers, U.-D., Goertz, L., Hildebrandt, B., & Pawlowski, Lippert, S. K., & Davis, M. (2006). A conceptual model
J. M. (2005). Quality in e-learning: Use and dissemination of integrating trust into planned change activities to enhance
quality approaches in European e-learning. Luxembourg: Office for technology adoption behavior. Journal of Information Science,
Official Publications of the European Communities. 32(5), 434-448.

Fernández, J. A. (1997). Spanish evaluation model in higher Moussiades, L., & Iliopolou, A. (2006). Guidelines for
education: circular evaluation. Higher Education Management, evaluating e-learning Environments. Interactive Technology &
9(1), 71-84. Smart Education, 3(3), 173-184.

Florida Center for Instructional, T. (2009). Technology Pearson, J., & Trinidad, S. (2005). OLES: an instrument
Integration Matrix Retrieved Dez, 2010, from http://fcit.usf.edu/ for refining the design of e-learning environments. Journal of
matrix/index.html Computer Assisted learning, 21, 396-404.

Gomes, M. J., Silva, B. D., & Silva, A. M. (2004). Avaliação de Quality, A. S. f. (2006). Radar Chart Vol. 2010. Retrieved from
cursos em e-learning. Paper presented at the Actas da Conferência http://www.asq.org/education/docs/radarchart.pdf
oLES`04, Aveiro, Portugal.
Rosa, O., & Angulo, L. (2007). Modelo de Excelencia de
Harvey, J., Higgison, C., & Gunn, C. (2000). Evaluation. In C. Programas Formativos en Línea (MEPFL). Revista Iberoamericana
Higgison (Ed.), Online Tutoring e-book (pp. 5.1-5.9). Edinburgh: de Educación, 42(5), 1-14.
The Institute for Computer Based Learning of the Heriot-Watt
University. Rosenberg, M. (2006). Beyond e-learning. San Francisco: Pfeiffer.

Kaczynski, D., Wood, L., & Harding, L. (2008). Using radar Rubio, M. J. (2003). Focus and models of evaluation of the
charts with qualitative evaluation: Techniques to assess change in e-learning. 9(2), 101-120. Retrieved from http://www.uv.es/
blended learning. Active Learning in Higher Education, 9(1), 23- RELIEVE/v9n2/RELIEVEv9n2_1.htm
41.
Sampiere, R., Collado, C., & Lucio, P. (2006). Metodologia de
Khan, B. H., & Granato, L. A. (2007). Program evaluation in Pesquisa (Vol. 3). São Paulo: McGrawHill.
E-learning. Retrieved from http://asianvu.com/digital-library/
Tait, A. (1997). Quality Assurance in Higher Education: Selected
elearning/elearning_program_evaluation_by_khan_and_Granato.
Case Studies.Vancouver: The Commonwealth of Learning.
pdf
Tapscott, D., & Williams, A. D. (2008). Wikinomics: A Nova
Korthagen, F., & Vasalos, A. (2005). Levels in reflection: core
Economia dasMultidões Inteligentes. Lisboa: Quidnovi.
reflection as a means to enhance professional growth. eachers and
Teaching: theory and practice, 11(1), 47-71. Wood., G., & Haber, J. (2001). Métodos, avaliação crítica e
utilização (4.th ed.). Rio de Janeiro: Guanabara Koogan.
Kotter, J. (1996). Leading change. Massachusetts: Harvard Business
School Press. Yin, R. K. (2009). Case study research – design and methods (4th
ed.). California: Sage.
Kruse, K. (2002). Evaluating e-Learning: Introduction to
the Kirkpatrick Model. Retrieved May, 2010, from http://
www.mizanis.net/edu3105/bacaan/design_L/Evaluating%20

Edition and production
Name of the publication: eLearning Papers Copyrights
ISSN: 1887-1542
The texts published in this journal, unless otherwise indicated, are subject
Publisher: elearningeuropa.info
to a Creative Commons Attribution-Noncommercial-NoDerivativeWorks
Edited by: P.A.U. Education, S.L. 3.0 Unported licence. They may be copied, distributed and broadcast pro-
Postal address: c/Muntaner 262, 3r, 08021 Barcelona (Spain) vided that the author and the e-journal that publishes them, eLearning
Phone: +34 933 670 400 Papers, are cited. Commercial use and derivative works are not permitted.
Email: editorial@elearningeuropa.info The full licence can be consulted on http://creativecommons.org/licens-
Internet: www.elearningpapers.eu es/by-nc-nd/3.0/

ing
earn
eL ers
24
u
ers.e
gpap
www
.elea
Pap
10

Models and instruments for assessing Technology Enhanced Learning Environments in higher education

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Models and instruments for assessing Technology Enhanced Learning Environments in higher education (20)

Recently uploaded (20)

Models and instruments for assessing Technology Enhanced Learning Environments in higher education