LAK13 linkedup tutorial_evaluation_framework

Using Linked Data in Learning Analytics
LAK 2013 tutorial

EvaluaHon
of
Linked
Data
tools
for
Learning
AnalyHcs

Hendrik
Drachsler
(@hdrachsler,
drachsler.de)

(CELSTEC,
Open
Universiteit
Nederland,
NL)

Eelco
Herder

(L3S
Research
Center,
DE)

Mathieu
d’Aquin
(@mdaquin,
mdaquin.net)

(Knowledge
Media
InsHtute,
The
Open
University,
UK)

Stefan
Dietze

(L3S
Research
Center,
DE)

Example of scientific competitions
What are the evaluation criteria of Robot Wars?

Criteria:
•  Damage
•  Aggression
probabilistic combination of
– Item-based method •  Control
– User-based method
– Matrix Factorization
•  Applause
– (May be) content-based method

2

RecSysTEL Evaluation criteria
1. Accuracy
1. Accuracy
2. Coverage
2. Coverage
3. Precision
4. Recall
3. Precision
4. Recall

1. Effectiveness of learning 1. Reaction of learner
2. Efficiency of learning 2. Learning improved
3. Drop out rate 3. Behaviour
4. Satisfaction 4. Results

Combine approach by Kirkpatrick model by
Drachsler et al. 2008 Manouselis et al. 2010
3

TEL RecSys::Review study

Conclusions:

Half of the systems (11/20) still at design or prototyping
stage only 9 systems evaluated through trials with human
users.

Manouselis, N., Drachsler, H., Vuorikari, R., Hummel, H. G. K., & Koper, R. (2011).
Recommender Systems in Technology Enhanced Learning. In P. B. Kantor, F. Ricci,
L. Rokach, & B. Shapira (Eds.), Recommender Systems Handbook (pp. 387-415).
Berlin: Springer. 4

The TEL recommender
research is a bit like this...
We need to design for each domain an appropriate
recommender system that ﬁts the goals and tasks"

5

TEL recommender
experiments lack results
“The performance
transparency and
of different research
standardization.
efforts in recommender
They need tohardly
systems are be
repeatable to test:
comparable.”

•  Validity
(Manouselis et al., 2010)
•  Verification Kaptain Kobold 
http://www.ﬂickr.com/photos/
•  Compare results kaptainkobold/3203311346/

6

Data-driven Research and Learning Analytics"

EATEL-
Hendrik Drachsler (a), Katrien Verbert (b)"
"
(a) CELSTEC, Open University of the Netherlands"
(b) Dept. Computer Science, K.U.Leuven, Belgium"
"

7

7

TEL RecSys::Evaluation/datasets

"
Drachsler, H., Bogers, T., Vuorikari, R., Verbert, K., Duval, E., Manouselis, N., Beham, G.,
Lindstaedt, S., Stern, H., Friedrich, M., & Wolpers, M. (2010). Issues and Considerations
regarding Sharable Data Sets for Recommender Systems in Technology Enhanced Learning.
Presentation at the 1st Workshop Recommnder Systems in Technology Enhanced Learning
(RecSysTEL) in conjunction with 5th European Conference on Technology Enhanced Learning
(EC-TEL 2010): Sustaining TEL: From Innovation to Learning and Practice. September, 28,
2010, Barcelona, Spain." 9
"

5. Dataset Framework
dataTEL evaluation model
Datasets
Formal Informal

Data A Data B Data C

Algorithms: Algorithms: Algorithms:
Algoritmen A Algoritmen D Algoritmen B
Algoritmen B Algoritmen E Algoritmen D
Algoritmen C

Models: Models: Models:
Learner Model A Learner Model C Learner Model A
Learner Model B Learner Model E Learner Model C

Measured attributes: Measured attributes: Measured attributes:
Attribute A Attribute A Attribute A
Attribute B Attribute B Attribute B
Attribute C Attribute C Attribute C
17

42
10

5. Dataset Framework
dataTEL evaluation model
Datasets
Formal Informal

In Data A
LinkedUp we have Data B opportunity to apply a
the Data C
structured approach to develop a
community accepted evaluation framework.
Algorithms: Algorithms: Algorithms:
Algoritmen A Algoritmen D Algoritmen B
Algoritmen B Algoritmen E Algoritmen D
1.  Top-Down by a literature study
Algoritmen C

2.  Bottom-up by Models: with experts in the field
Models: GCM Models:
Learner Model A Learner Model C Learner Model A
Learner Model B Learner Model E Learner Model C

Measured attributes: Measured attributes: Measured attributes:
Attribute A Attribute A Attribute A
Attribute B Attribute B Attribute B
Attribute C Attribute C Attribute C
17

42
11

Development
of
the
Evalua=on
Framework

P1: Initialisation P2: Establishment P3: Exit and
and Evaluation Sustainability

M0-M6: Preparation M7-M18: Competition cycle M18-M24: Finalising

Comp
etition

Revie Final
Expert
3x
Draft w of
EF proposal EF release
validation of EF

New Refin
versio ement
n of EF
Literature review Group Concept Documentation
Cognitive Mapping Mapping Dissemination
Practical experiences
and refinement

Hendrik Drachsler 25 February 2013 13

Group
Concept
Mapping

•  Group Concept Mapping resembles the
Post-it notes problem solving technique
and Delphi method

•  GCM involves participants in a few
simple activities (generating, sorting
and rating of ideas) that most people are
used to.

GCM is different in two substantial ways:
1. Robust analysis (MDS and HCA)
GCM takes up the original participants contribution and then quantitatively
aggregate it to show their collective view (as thematic clusters)

2. Visualisation
GCM presents the results from the analysis as conceptual maps and other graphical
representations (pattern matching and go-zones).


Group
Concept
Mapping

brainstorm
•  innovations in way network is delivered
•  (investigate) corporate/structural alignment
•  assist in the development of non-traditional partnerships (Rehab with the
Medicine Community)

sort
•  expand investigation and knowledge of PSN'S/PSO's
•  continue STHCS sponsored forums on public health issues (medicine
managed care forum)
•  inventory assets of all participating agencies (providers, Venn Diagrams)
•  access additional funds for telemedicine expansion
•  better utilization of current technological bridge
•  continued support by STHCS to member facilities
•  expand and encourage utilization of interface programs to strengthen the
viability and to improve the health care delivery system (ie teleconference)
•  discussion with CCHN

Decide how to
manage multiple
tasks.
20 Manage resources effectively.
4

Work quickly
and effectively
under pressure
49

Organize the work
when directions are
not specific.
39

e t ive
ly
ly.
he eff
e ct tive
gS ime eff
ec
tin g et rc es
Ra
a ou ic.
an n s.
M res atio a sk ecif
1 na
ge rm t.
info tan le t t sp
Ma no
of por ultip

...organize the
5 e a re
4 2 ud is im age m ns
ultit hat an ctio
3 5 am ew om ire
an id wt nd
2 4 Sc dec ho he
1 d e r kw
3 5 3 an cid wo nd
De he ely na
2 4 et
1 4 niz ctiv atio
3 ga ffe orm

issues...
5 Or
2 ee inf nt.
4 5 tim of ta
1 ge de r
3 5 na itu impo
2 4 Ma ult
1 1 a m at is
3 5 an e wh
2 Sc cid
4 de
1 3
3 5
2 4
1 3
2
1

rate

Group
Concept
Mapping


Group
Concept
Mapping

Medicine Community)
managed care forum)

…”map” the issues...

organize
sort
Decide how to
manage multiple
tasks.
4

Work quickly and
effectively under
pressure
49

Organize the work
when directions are not
specific.
39

et
he effectiv effectively
ely .

g S time
tin Manage resource
s d
an

Ra ation .
ks. ecific

Technology
1 na
ge inform le tas t sp
5 Ma e of rtant. ltip no
4 2 ltitudimpo ge
mu ns
are
3 a muat is na ectio
5 an ma dir
2 Sc e wh w to en
4 cid wh
1 3 de e ho rk
3 5 cid wo
De d

Information Services
2 4 e the ely n an
4
1 3 5 ga
niz ectiv ma
tio
Or e eff or
2 4 5 tim inf
1 3 5 na
ge e of rtant.
2 Ma ltitudimpo
4
1 1 a muat is
3 5 an wh
2 Sc cide
4 3 de
1 3 5
2 4
1 3
2
1

rate Community & Consumer Views

Regionalization

Management STHCS as model
Financing


Group
Concept
Mapping

Medicine Community)
managed care forum)

Information Services Technology
organize
sort
Community & Consumer Views
Decide how to
manage multiple
tasks.
4

Work quickly and
effectively under
pressure
49

Organize the work
when directions are not
specific.
39

et
he effectiv effectively
ely .

g S time
tin Manage resource
s d
an

Ra ation .
ks. ecific
1 na
ge inform le tas t sp
5 Ma e of rtant. ltip no
4 2 ltitudimpo ge
mu ns
are
3 a muat is na ectio
5 an ma dir
2 Sc e wh w to en
4 cid wh
1 3 de e ho rk
3 5 cid wo
De d
2 4 e the ely n an
4
1 3 5 ga
niz ectiv ma
tio
Or e eff or
2 4 5 tim inf
1 3 5 na
ge e of rtant.
2 Ma ltitudimpo
4
1 1 a muat is
3 5 an wh
2 Sc cide
4 3 de
1 3 5
2 4
1 3
2

Regionalization
1

rate

map Information Services
Technology

Community & Consumer Views

Regionalization

Financing Management Mission & Ideology
Management STHCS as model
Financing

...prioritize the issues...


Group
Concept
Mapping

D2.1 Evaluation Criteria and Methods

•  Invited 122 external experts
•  56 experts contributed 212 indicators for the
evaluation framework
•  After cleaning -> 108 indicators remained
•  26 experts sorted on similarity in meaning
•  26 experts rated on priority and
applicability


Plus Minus Interesting rating

Look at and listen to the presentation
of the Evaluation Framework

Meanwhile…create notes on

P: Plus
M: Minus
I: Interesting

Write down everything that comes to your mind, generate
as many ideas as possible, do not filter your ideas.

Group
Concept
Mapping

A point map


Group
Concept
Mapping

A cluster map 15


Group
Concept
Mapping

A cluster map 6


Group
Concept
Mapping

Clusters’ labels


Group
Concept
Mapping

Rating Map Priority


Group
Concept
Mapping

Rating Map Applicability


Group
Concept
Mapping


WP2: Literature review
1. Literature review of suitable evaluation approaches and criteria
2. Review of comprising initiatives such as LinkedEducation, MULCE, E3FPLE and
the SIG dataTEL

Many thanks for your attention!
This silde is available at:
http://www.slideshare.com/Drachsler

Email: hendrik.drachsler@ou.nl
Skype: celstec-hendrik.drachsler
Blogging at: http://www.drachsler.de
Twittering at: http://twitter.com/HDrachsler


LAK13 linkedup tutorial_evaluation_framework

More Related Content

What's hot (19)

Viewers also liked (20)

Similar to LAK13 linkedup tutorial_evaluation_framework (20)

More from Hendrik Drachsler (20)

Recently uploaded (20)

LAK13 linkedup tutorial_evaluation_framework