Quality evaluation of eLearning through an international peer-review community

Quality evaluation of eLearning
through an international peer-review community
Ulf-Daniel Ehlers and Jochen Joosten
University Duisburg-Essen, Essen, Germany

Summary

The purpose of the paper is to present the Open ECBCheck, a new certification and quality
improvement scheme for eLearning in the field of Capacity Building. The model supports
Capacity Building Organisations to measure how successful their eLearning programmes are
and allows for continuous improvement though peer collaboration and benchlearning. Open
ECBCheck forms a participative quality environment which allows its members to benefit in a
variety of ways by having access to tools and guidelines for their own practice on the one hand,
and being able to obtain a community based label on the other hand.

The certification scheme has been developed throughout a participatory process with
organisations in the field of Capacity Building, who were extensively interviewed regarding their
demands towards a new certification scheme. In a second step, existing standards and
certificates like UNIQUE, EFMD CEL or ISO/IEC 19796-1 were assessed to see if they meet
these demands or if parts of the existing standards and certificates could be incorporated into
the scheme. Finally, as presented in this article, a blueprint for a certification has been
proposed, including a description of the architecture, the definition of quality criteria and related
indicators as well as a definition of a certification process.

The results of extensive analysis suggest that organisations in the field of Capacity Building
have in general a high interest to obtain a quality label. Mostly, organisations are interested in a
tool which supports measuring and improving impact efficiency, helping to ensure the success
of eLearning programs and allowing to benchmark with other organisations.

The analysis we present in this article also shows that organisations differ in the diffusion of
eLearning within their Capacity Building activities. To incorporate these differences a label with
two options has been developed: one for single programs and another one for institutions.

Keywords: Capacity building, quality certification, elearning, quality label, benchlearning,
efficiency tool, quality management, Open ECBCheck, peer review

Original version of the paper presented at ICDE World Conference in Maastricht, 2009

1 Introduction
One potential for e-learning in Capacity Building which has been identified in a recent study
(Ehlers et at. 2007, p. 58) is the field of quality management. So far, there is neither
transparency about the quality of e-learning programmes for Capacity Building nor about the
quality of the organisations that offer e-learning in Capacity Building according to Ehlers et al..
The study argues that quality certification, developed in consensus with a network of Capacity
Building Organisations, could lead to higher trust in the still developing market of e-learning for
Capacity Building. Observations by the Operations Evaluation Department of World Bank

eLearning Papers • www.elearningpapers.eu • 1
Nº 17 • December 2009 • ISSN 1887-1542

(OED) and Whyte support this view: In a report the OED states that “many projects have
capacity building activities embedded in their major operational components, but the objectives
of these activities tend to be ill defined, and their achievement is poorly tracked and reported”
(OED 2005, p. viii). Whyte (2004, p. 9) who conducted a so called landscape analysis of donor
trends, also agrees that Capacity Building activities „are often embedded in other programs and
are not tracked separately“.

This paper argues that for certain requirements none of the existing approaches for quality
management or quality certification can be applied one-to-one to e-learning in the field of
Capacity Building. To fill this gap and to support the application of e-learning in the field of
Capacity Building the aim of this paper is to develop a certification model based on existing
quality approaches that fits the special requirements in Capacity Building. This could lead to
higher trust in e-learning for Capacity Building as well as to higher effectiveness of e-learning
programmes and thus to a broader application of e-learning in Capacity Building programmes
with a higher impact on economic development.

2 Open ECBCheck
Open ECBCheck is a new certification and quality improvement scheme for E-Learning
programmes and institutions in international Capacity Building. It supports Capacity Building
Organisations to measure how successful their e-learning programmes are and allows for
continuous improvement though peer collaboration and benchlearning. Open ECBCheck forms
a participative quality environment which allows its members to benefit in a variety of ways by
having access to tools and guidelines for their own practice on the one hand, and being able to
obtain a community based label on the other hand. Three stages to quality are suggested:
1. Members of the Open ECBCheck professional community document their commitment
to quality by joining
2. The Open ECBCheck professional community provides access to and allows sharing of
guidelines, tools as well as experiences for quality development for its members
3. On basis of a detailed self-assessment process, members can enter into mutual peer-
review partnerships to improve the quality of their e-learning offers.

Open ECBCheck is developed from the community of organisations through an innovative and
participative process which has been initiated by InWent – Capacity Building International,
Germany and the European Foundation for Quality in E-Learning (EFQUEL). Over 20
organisations have meanwhile showed their interest in joining the Open ECBCheck community
which will form a professional network for quality and innovation.

3 Requirements for a Quality Label for e-learning in Capacity Building
Between June and September 2008 an extensive consultation process involving 15
international Capacity Building Organisations in the field of e-learning took place. All
organisations were asked to contribute information on their activities in the field of e-learning,
as well as their interests and requirements for the development of a quality label for e-learning.
The information was collected through both a questionnaire and an extensive interview series.
The study provides precise information on requirements for the development of a quality label
for e-learning in Capacity Building. It shows a general high interest into the topic of quality
development and suggests that apart from a quality label, the launch of a professional network
which gives its members access to guidelines, tools and best practices is of urgent interest to
participating stakeholders. Therefore it is suggested to develop Open ECBCheck as a
combination of a self commitment (declaration of intent) (stage 1) which allows organisations to
join a professional network where they get access to guidelines, tools and practices (stage 2),
and where they can enter into a self assessment and peer-review process which allows them to
obtain a quality label (stage 3). The quality label is thus constructed as a community based
peer-label, focussing on the improvement of practices through a peer-reviewed self-
assessment.

Nº 17 • December 2009 • ISSN 1887-1542

In a context analysis as well as the study on requirements and constraints of Capacity Building
Organisations towards a quality label for e-learning in Capacity Building the distinct
characteristics of the field could be elaborated and a number of unique requirements could be
derived. These requirements can be systematised in two major domains. The first is concerned
with the fundamental architecture of the label, concerning the quality dimensions, criteria and
methods of assessing and validating them and the second with the governance system and
processes, concerning the way the label is awarded. An overview of the main aspects in both
areas is given in figure 1.

Governance System
Architecture
and Processes

• Coverage of e-Learning • Quality Certification as an Open,
Programmes as well as e- Community Driven Process
Learning Institutions
• Governance of the Quality Label
• Broad Coverage and
Applicability through Minimum • Low Cost Certification System
and Excellence Criteria
• Learning for Improvement
• Educational Territories

Figure 1: Overall Requirements for a Quality Label for e-Learning in Capacity Building

Architecture of the quality label

1. Coverage of e-learning programmes as well as e-learning institutions: The label should
offer the possibility to cover both single programmes and institutions as certification
entity to consider different e-learning strategies of organisations. A quality label for
single programmes is considered of more interest for organisations that are in the stage
of integrating e-learning rather project oriented and a quality label on the institutional
level is considered to be of more interest for organisations that have advanced to
integrate e-learning strategically in internal organisational processes as well as all
Capacity Building activities.

2. Broad coverage and applicability through minimum and excellence criteria: The label
should offer the distinction between minimum criteria that indicate solid quality of a
programme or institution and are relevant to all organisations and excellence criteria
that demonstrate exceptional quality achievements of an organisation or programme.
While all organisations would be required to meet at least the minimum criteria in order
to guarantee that they are conformant to the set standards, they can create their own
excellence profile through scoring high on the excellence criteria. The employed
methodology for evaluation is going back to the method of Qualitative Weighting and
Summation, described for the evaluation of learning software for the first time by
Baumgartner et al. (2002).

3. Educational territories: For the study and the development of the label the concept of
educational territories rather than educational segments has proven valid. It emphasises
that e-learning is not an own educational territory but a transversal component which
creates many different educational contexts, some covering the traditional distinction

Nº 17 • December 2009 • ISSN 1887-1542

between educational segments, some creating new learning spaces. Considering the
educational territories, Capacity Building Organisations focus their e-learning activities
largely on activities in the educational fields e-Learning at the Workplace and Evolved
Distance Education (see MENON Network 2007, pp. 34 or Delrio and Fischer 2007, p. 4
for the terminology used). Further important activities are Virtual Professional Networks
and Non-professional e-Learning Communities. While the quality criteria which will be
developed for the quality label can potentially be used for internal assessment and as
guidelines by organisations or individual organisation actors, the quality label will cover
especially those activities which are explicit and formal e-learning programmes.
However, informal activities like professional networks and community oriented learning
processes are more and more important in Capacity Building Organisations. The way
employees deal with technology in order to form learning relevant contexts like
communities then plays an important role for the assessment of the institutional
readiness of Capacity Building Organisations

Governance of the quality label

4. Quality certification as an open, community driven process: The special characteristics
of the Capacity Building community suggests an open, community and learning based
approach of mutual recognition of quality with a bottom-up agreement on criteria and the
governance system as the fundamental architecture of Open ECBCheck. While this is
sometimes perceived as a contradiction because quality certification appears often as
instrument of competitive distinction and not as open, community and consensus
oriented concept, the analysis’ results show a clear preference for open models,
oriented towards peer-review processes. Transparency of the evaluation processes on
basis of peer-review has been stated as an important factor for acceptance.

5. Governance of the quality label: The specific context of Capacity Building and of
Capacity Building Organisations has to be taken into account. It is not following primarily
market logic with free flowing capital in which customers have to be attracted, but rather
a closed market structure which follows clear rules and regulations. This has
consequences for a governance system of a quality label. Capacity Building
Organisations on the one hand have a clearly identified need for proving their
effectiveness and efficiency and need to work with transparent quality standards,
however, these quality standards can not be easily imposed on them from the outside.
Due to the specific constraints of this group of organisations quality rules and
regulations have to be developed in a consensus process and a certification processes
as well. This requires developing a certification system around a community of
organisations in which Capacity Building Organisations should be represented. The
representation of those who want to be certified within the system of certification is
posing specific constraints towards the governance system to be developed. In order to
avoid conflict of interest, the different acting bodies of a quality certification system have
to be clearly identified and separated from each other, so that decisions are based on
consensus of many rather than on interests of only few. Such a certification system then
would ensure high acceptance of the system within the community of Capacity Building
Organisations. For Open ECBCheck therefore a three-stage approach will be adopted:
a. Organisations become part of a professional network, which has been launched
in a common meeting. They are asked to declare their willingness to advance
quality development for their e-learning activities in a declaration of intent.
b. Organisations will have access to the Open ECBCheck guidelines, criteria and
tools for e-learning quality for use in their organisations and networks.
c. Organisations have the possibility to undergo a self-assessment process to
document how they are assuring and promoting quality in e-learning in their
organisations or their programs which will be reviewed by two members of the
community in form of a peer-review.
This open, three stage process which is based on a professional network allows
organisations to advance over time in their quality practices and – in case they want – to

Nº 17 • December 2009 • ISSN 1887-1542

get recognition from other organisations by obtaining the peer-review based label. This
concept is also supported by a number of organisations that are not offering e-learning
themselves but work together with partner organisations who offer e-learning and want
to use the tools and guidelines to support these partners in quality development. It
allows to demonstrate (self-)commitment towards quality (code of practice), make use of
existing guidelines and tools, and obtain a community based label through self-
assessment and peer-review.

6. Low cost certification system: Furthermore, the certification system has to be
constructed in a way that it can be operated on a low cost level to be affordable for the
majority of Capacity Building Organisations and their clients in developing countries.
Developing a low cost system demands an exchange of services between the
stakeholders rather than a flow of capital for buying those services. For Open
ECBCheck a thorough conceptualisation of the costs factors of a certification process
has to be taken into account. Within the development of open ECBCheck two concepts
will therefore be explored. First, for assessment purposes a combination of self-
assessment and peer-review of a self-assessment report will be developed and
secondly the peer-review services will be exchanged between those organisations
undergoing certification processes.

7. Learning for improvement: The interview results show that the certification process
should not only lead to a certification of a programme or institution but should be
conceptualised as a learning activity. Also in this case, self-assessment and peer-review
of a self-assessment report will be explored. First, the self-assessment allows an
organisation a systematic analysis of strengths and weaknesses of the institution or a
programme. Furthermore, the peer-review of the resulting self-assessment report will
not only provide the organisation with a feedback on the self-assessment and provided
information, but also include a learning report that focuses on the possibilities for
improvement that have been identified during the review.

The results of the analysis of requirements and constraints of Capacity Building Organisations
towards a quality label for e-Learning in Capacity Building show the distinct characteristics of
the field and a number of unique requirements in the two major domains architecture and
governance system and processes. These requirements have been the basis for the
development of the quality label architecture. In conclusion the study shows that organisations
are interested in an open, community of practice based approach for mutual recognition of
quality within the international community of Capacity Building Organisations that is based on a
bottom up process of agreeing on framework criteria that allow to incorporate the multitude of
quality initiatives that are already in place in these organisations and a strong self-commitment
to demonstrate the seriousness of the interest and effort put into quality.

4 Conceptual Frame for the Open ECB Check Label
Based on the requirements that Capacity Building Organisations have towards a quality label in
e-learning for Capacity Building and on the characteristics that can be derived from a
systematisation of the use of e-learning in Capacity Building, the following architecture for a
quality label is proposed.

Nº 17 • December 2009 • ISSN 1887-1542

Declaration of Intent

Community of Practice around Open ECB Check

Community and Guidelines
Program level Institutional level

e.g. pedagogy e.g. e.g. strategy e.g. processes
instructional
design

Label and Process of Certification

Program level Institutional level

e.g. pedagogy e.g. e.g. strategy e.g. processes
instructional
design

self-assessment self-assessment

peer-review peer-review

Figure 2: Architecture of Open ECBCheck

The basic element of the Open ECBCheck label will be a professional network consisting of
organisations that are interested into quality issues related to e-learning in Capacity Building.
This community allows on the one hand a bottom-up approach in developing and improving the
Open ECBCheck continuously as well as the sharing of best practice. On the other hand this
approach offers the chance for mutual recognition of quality within the international community
of Capacity Building Organisations. To join the community and gain access, interested
organisations are asked to sign a declaration of intent to demonstrate their interest to advance
quality to the community and their willingness to contribute. Organisations that are members of
the community do gain access to quality guidelines that support their work on quality issues and
are the reference group for a peer-review process that organisations have to undergo if they
want to obtain the Open ECBCheck quality label. Guidelines as well as the review process will
be available for both program and institutional level and information will be disseminated
through an online resources centre (Quality Centre) that also supports collaboration of the
member organisations.

Guidelines and framework criteria will focus on existing quality practices for e-learning in
Capacity Building Organisations. Thus, the focus will be on helping organisations to benchmark
their quality practices, improve their tools and learn from each other through a community

Nº 17 • December 2009 • ISSN 1887-1542

based exchange. In cases where no practices (yet) exist, a number of basic guidelines will be
provided for adoption by the respective organisation. Rather than being normative ridging, the
focus will be on recognising achievements and improving existing practices. Examples of
quality e-learning practices on program and institutional level will be elaborated and provided as
cases to benchmark own practices and learning from each other. This ensures that
organisations gain an overview on what criteria are of importance but they are open to decide
how to address them. Thus, already established procedures or guidelines can be integrated
and organisations are not forced to abandon these. Still, the guidelines will contain advice what
tools could be used to achieve quality in this area. Additionally this allows member
organisations to share best practice as support for each other. If a best practice (e.g. a checklist
how to assess learner’s needs) is accepted by the community it can be added to or referenced
by the guideline in the online resource centre.

For those organisations interested in obtaining the Open ECBCheck as a quality label, the tools
and guidelines will be applied to the certification process. This process can address either the
programme or the institutional level. Based on the criteria that are defined within the guideline
organisations are asked to carry out a self-assessment proceedure to assess their programmes
or institution. The self-assessment is supported by documents that systematise and structure
the process. As soon as an organisation has finished the self-assessment, the results of the
self-assessment are validated through a peer-review that is conducted by another member
organisation of the Open ECBCheck community which has already obtained the label. It will be
one of the requirements for obtaining the label to volunteer in a review process for another
candidate. Through this inter-changed collaboration, not only the assessment of own practices
but also a benchlearning process through assessment of other organisations’ practices will be
introduced. After the peer-review has been completed successfully, the Open ECBCheck label
is awarded to an organisation and can be used by them. After a certain period of time, the label
has to bee renewed if criteria have changed.

References

Baumgartner, Peter; Häfele, Hartmut; Maier-Häfele, Kornelia (2002): Evaluierung von Lernmanagement-
Systemen: Theorie - Durchführung - Ergebnisse. In: Hohenstein, Andreas; Wilbers Karl (Eds.) (2002):
Handbuch E-Learning, Fachverlag Deutscher Wirtschaftsdienst, Köln.
Delrio, Claudio; Fischer, Thomas (2007): HELIOS: Redefining e-Learning Territories. In: eLearning
Papers, 4, http://www.elearningeuropa.info/files/media/media12725.pdf
Ehlers, Ulf-Daniel; Aimard, Virginie; Gwardak, Lukas; Dembski, Sven (2007): Potentiale von E-Learning
für Capacity Building. Studie “E-Learning by InWEnt”, Universität Duisburg-Essen, Essen.
MENON Network (2007): e-Learning for Innovation. HELIOS Yearly Report 2007. MENON Network,
Brussels.
OED (2005): Capacity Building in Africa. An OED Evaluation for World Bank Support. The World Bank,
Washington, D.C..
Whyte, Anne (2004): Landscape Analysis of Donor Trends in International Development. The Rockefeller
Foundation, New York.

Nº 17 • December 2009 • ISSN 1887-1542

Authors

Ulf-Daniel Ehlers
Assistant Professor
University Duisburg-Essen, Germany
http://www.ulf-ehlers.de/

Jochen Joosten
Student of business administration
University Duisburg-Essen, Germany / Robert Bosch GmbH

Copyrights

The texts published in this journal, unless otherwise indicated, are subject to a
Creative Commons Attribution-Noncommercial-NoDerivativeWorks 3.0
Unported licence. They may be copied, distributed and broadcast provided that
the author and the e-journal that publishes them, eLearning Papers, are cited. Commercial use
and derivative works are not permitted. The full licence can be consulted on
http://creativecommons.org/licenses/by-nc-nd/3.0/

Edition and production

Name of the publication: eLearning Papers
ISSN: 1887-1542
Publisher: elearningeuropa.info
Edited by: P.A.U. Education, S.L.
Postal address: C/ Muntaner 262, 3º, 08021 Barcelona, Spain
Telephone: +34 933 670 400
Email: editorial@elearningeuropa.info
Internet: www.elearningpapers.eu

Nº 17 • December 2009 • ISSN 1887-1542

Quality evaluation of eLearning through an international peer-review community

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Quality evaluation of eLearning through an international peer-review community (20)

More from eLearning Papers (20)

Recently uploaded (20)

Quality evaluation of eLearning through an international peer-review community