Designing Pedagogical Interventions to Support Student Use of Learning Analytics

D E S I G N I N G P E D A G O G I C A L
I N T E R V E N T I O N S T O S U P P O R T
S T U D E N T U S E O F L E A R N I N G
A N A LY T I C S
ALYSSA FRIEND WISE
Supported by the Social Sciences and Humanities Research Council of Canada

L E A R N I N G A N A LY T I C S

0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
0
1
1
0
1
0
1
1
0
0
1
0
1
0
0
1
0
1
0
1
1
0
1
0
0
1
0
1
1
0
0
1
1
0
0
1
0
1
1
0
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
0
1
0
1
0
0
1
0
1
0
0
1
1
0
1
0
1
0
1
1
1
1
0
1
1
0
0
1
1
0
1
1
0
1
0
0
1
0
1
0
1
0
0
1
1
0
1
0
1
1
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
1
0
1
0
1
0
1
0
0
1
0
1
0
1
1
0
1
0
0
1
0
1
1
0
0
1
1
0
0
1
0
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
0
1
0
1
0
0
1
0
1
0
1
0
1
1
1
1
0
1
1
0
0
1
1
0
1
1
0
1
0
0
1
0
1
0
1
1
0
0
1
0
1
0
1
0
1
0
1
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
1
0
1
0
0
1
0
1
0
1
1
0
1
0
0
1
0
1
1
0
0
1
1
0
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
0
1
0
1
0
0
1
0
0
1
0
1
1
1
1
0
1
1
0
0
1
1
0
1
1
0
1
0
0
1
0
1
0
1
1
0
0
1
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
0
0
1
0
1
0
0
1
0
1
0
1
1
0
1
0
0
1
0
1
1
0
0
1
1
0
1
0
1
0
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
0
1
0
1
0
0
1
0
1
1
1
0
1
1
1
1
0
1
1
0
0
1
1
0
1
1
0
1
0
0
1
0
1
1
0
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
1
0
1
0
1
0
1
0
0
1
0
1
0
1
1
0
1
0
0
1
0
1
1
0
0
1
1
0
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
0
1
0
1
0
0
1
0
1
1
1
1
0
1
1
1
1
0
1
1
0
0
1
1
0
1
1
0
1
0
0
1
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
0
1
0
1
0
0
1
0
1
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
0
0
1
0
1
0
0
1
0
1
0
1
1
0
1
0
0
1
0
1
1
0
0
1
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
0
1
0
1
0
0
1
0
1
0
1
0
1
1
1
1
0
1
1
0
0
1
1
0
1
1
0
1
0
0
1
0
1
0
1
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
1
0
1
0
0
1
0
1
0
1
1
0
1
0
0
1
0
1
1
0
0
1
1
0
0
1
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
0
1
0
1
0
0
1
0
1
1
0
1
1
1
1
0
1
1
0
0
1
1
0
1
1
0
1
0
0
1
0
1
0
1
1
1
0
1
1
1
1
0
1
1
0
0
1
1
0
1
1
0
1
0
0
1
0
1
0
1
1
0
1
0
1
0
0
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
1
0
1
1
0
1
1
0
1
0
1
0
1
0
0
1
0
1
0
1
1
0
1
0
0
1
0
1
1
0
0
1
1
0
0
1
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
0
1
0
1
0
0
1
0
1
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
0
1
0
1
1
1
1
0
1
1
0
0
1
1
0
1
1
0
1
0
0
1
0
1
0
1
0
1
0
1
0
0
1
0
1
0
1
1
0
1
0
0
1
0
1
1
0
0
1
1
0
1
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
1
0
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
1
0
0
1
1
0
1
0
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
1
0
0
1
0
0
1
1
0
1
0
1
0
1
0
0
1
0
1
0
1
1
0
1
0
0
1
0
1
1
0
0
1
1
1
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
0
1
0
1
0
0
1
0
0
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
0
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
1
0
1
0
0
1
0
1
0
1
1
0
1
0
0
1
0
1
1
0
1
1
1
0
0
0
1
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
0
0
1
0
1
0
1
1
0
1
0
0
1
0
1
1
0
0
1
1
0
1
1
1
0
0
1
0
1
0
1
0
0
1
0
1
0
1
1
0
1
0
0
1
0
1
0
1
1
0
0
1
1
1
0
1
1
0
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
0
1
0
1
0
0
1
0
1
0
1
1
0
0
1
0
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
0
1
0
1
0
0
1
0
1
1
0
1
1
0
1
0
1
0
1
0
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
0
1
1
0
1
0
1
1
0
0
1
0
1
0
0
1
0
1
0
1
1
0
1
0
0
1
0
1
1
0
0
1
1
0
0
1
0
1
1
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
0
1
0
1
0
0
1
0
1
0
0
1
1
0
1
0
1
0
1
1
1
1
0
1
1
0
0
1
1
0
1
1
0
1
0
0
1
0
1
0
1
0
0
1
1
0
1
0
1
1
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
1
0
1
0
1
0
1
0
0
1
0
1
0
1
1
0
1
0
0
1
0
1
1
0
0
1
1
0
0
1
0
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
0
1
0
1
0
0
1
0
1
0
1
1
0
1
1
1
1
0
1
1
0
0
1
1
0
1
1
0
1
0
0
1
0
1
0
1
1
0
1
0
1
0
0
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
1
0
1
1
0
1
1
0
1
0
1
0
1
0
0
1
0
1
0
1
1
0
1
0
0
1
0
1
1
0
0
1
1
0
0
1
1
0
1
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
1
0
0
1
1
0
1
0
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
1
0
0
1
0
0
1
1
0
1
0
1
0
1
0
0
1
0
1
0
1
1
0
1
0
0
1
0
1
1
0
0
1
1
0
1
1
1
0
0
1
0
1
0
1
0
0
1
0
1
0
1
1
0
1
0
0
1
0
1
0
1
1
0
0
1
1
1
0
1
1
0
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
0
1
0
1
0
0
1
0
1
0
1
1
0
0
1
0
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
0
1
0
1
0
0
1
0
1
1
0
1
1
0
1
0
1
0
1
0
1
0
1
1
1
1
0
1
1
0
0
1
1
0
1
1
0
1
0
0
1
0
1
0
0
1
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
0
0
1
0
1
0
0
1
0
1
0
1
1
0
1
0
0
1
0
1
1
0
0
1
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
0
1
0
1
0
0
1
0
1
0
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
0
1
0
1
0
0
1
0
1
1
0
1
1
1
1
0
1
1
0
0
1
1
0
1
1
0
1
0
0
1
0
1
0
1
1

Sub UniquePostsRead()
For k = 1 To MaxUser Step 1
RowCount = Range("A1").CurrentRegion.Rows.Count
For w = 1 to MaxWeek Step 1
StartTime = Sheets("Week").Cells(w + 1, 2)
EndTime = Sheets("Week").Cells(w + 1, 3)
PostNum = 0
PostsIndex = 0
Do While Cells(i, datestamp) <= EndTime And i <= RowCount
If Cells(i, Source) = “Read" Then
If Cells(i, Message_Author) <> Val(ActiveSheet.Name)
And Cells(i, Scan) <> "X" Then
flag = 0
For j = 1 To PostsIndex Step 1
If Posts(j) = Cells(i, Message_Id) Then
flag = 1
j = PostsIndex
End If
Next j
If flag = 0 Then
PostsIndex = PostsIndex + 1
Posts(PostsIndex) = Cells(i, Message_Id)
End If
End If
End If
Sheets(“Stats").Cells(Line, 22) = PostsIndex
Next w
Next k
End Sub
PercentPostsRead = UniquePostsRead
TotalPostNumber

IF LEARNING ANALYTICS ARE TO TRULY
MAKE AN IMPACT ON UNIVERSITY
TEACHING, AND LEARNING AND MAYBE
EVEN REVOLUTIONIZE EDUCATION
WE NEED TO CONSIDER AND DESIGN
FOR WAYS IN WHICH THEY WILL IMPACT
THE LARGER ACTIVITY PATTERNS OF
INSTRUCTORS AND STUDENTS

S T U D E N T U S E O F
L E A R N I N G A N A LY T I C S

WHY FOCUS ON STUDENTS AS LEARNING
ANALYTICS USERS?

CHALLENGES & OPPORTUNITIES FOR
STUDENTS AS LEARNING ANALYTICS USERS

LEARNING
ANALYTICS
LEARNING
ANALYTICS
INTERVENTIONS
C A P T U R I N G A N D
C A L C U L AT I N G M E A N I N G F U L
T R A C E S O F L E A R N E R S ’
A C T I V I T Y A N D P R E S E N T I N G
T H E M I N A U S E F U L F O R M
S U P P O R T I N G I N T E R P R E TAT I O N
A N D U S E O F T H E A N A LY T I C S I N
D E C I S I O N M A K I N G T H R O U G H
F R A M I N G T H E P R O C E S S
T H R O U G H W H I C H L E A R N I N G
A N A LY T I C S A R E TA K E N U P B Y
P E O P L E A S PA R T O F S O M E
L A R G E R E D U C AT I O N A L A C T I V I T Y

( W H AT P O I N T S I N W H AT L E A R N I N G P R O C E S S E S , W / W H AT F R E Q )
( W H AT Q U E S T I O N S A R E T H E Y A N S W E R I N G )
( H O W S H O U L D I N F O B E I N T E R P R E T E D , U S E D I N T H I S C O N T E X T )
( W H AT D O N E D I F F E R E N T LY, H O W D O A C T I V I T I E S F I T T O G E T H E R )
LOCALLY CONTEXTUALIZED QUESTIONS
OF INTERPRETATION & ACTION

1 G O A L
2 F O U N D AT I O N S
3 P R O C E S S E S
4 PRINCIPLES

Learning Analytics
&
Learning Activities

INTEGRATION
• MAKE THE USE OF LEARNING ANALYTICS AN
ELEMENT OF THE LEARNING DESIGN
• POSITION THE USE OF ANALYTICS AS AN
INTEGRAL PART OF COURSE ACTIVITIES TIED
TO GOALS AND EXPECTATIONS
• PROVIDE A LOCAL CONTEXT FOR MAKING
SENSE OF THE DATA

CONCEPTUAL QUESTIONS
1. GIVEN THE GOALS OF THE
EDUCATIONAL ACTIVITY, WHAT METRICS
ARE IMPORTANT TO FOCUS ON?
2. WHAT DO PRODUCTIVE AND
UNPRODUCTIVE PATTERNS IN THESE
METRICS LOOK LIKE?

PRACTICAL QUESTIONS
1. HOW TO MAKE THIS THREAD BETWEEN
LEARNING GOALS, STUDENT ACTIONS
AND ANALYTICS FEEDBACK CLEAR
2. HOW TO MAKE ANALYTICS USE PART
OF COURSE ACTIVITY FLOW

Learning Analytics
&
Learning Activities
Grounding

Clear guidelines and discussion of
– the purpose of engaging in online discussion
– the instructor’s expectations for a productive process of
engaging in online discussions
– how the learning analytics provide indicators of this process
GROUNDING
articulating one’s ideas, being exposed to the ideas of
others, negotiating differences in perspective
attending deeply to a spectrum of others’ ideas, and
contributing comments that are responsive and rationaled,
percent of posts read introduced as a metric that has clear
meaning in the context of the activity

Discussion Participation Guidelines
Attending to Others Posts
Broad Listening: Try to read as many
posts as possible to consider everyone’s
ideas in the discussion. This can help
you examine and support your own ideas
more deeply. However, when time is
limited it is better to view a portion in
depth, then everything superficially.
Learning Analytics Guidelines
Attending to Others’ Posts
% of
posts
read
The proportion of posts
you read (not scanned)
at least once.
It is good to read as many posts as
possible to consider everyone’s
ideas in the discussion However,
when time is limited it is better to
view a portion in depth, then
everything superficially.
*The visual interface shows posts that
you have viewed in blue and new
ones in red to help you track this.
GROUNDING

STUDENT AGENCY
CAN GIVE STUDENTS THE OPPORTUNITY TO
– ESTA B L I S H P E RS O N A L G OA L S FO R T H E AC T I V I T Y
( I N R E L AT I O N TO T H E G I V E N I N ST R U C T I O N A L
G OA L S )
– H AV E ( S O M E ) AU T H O R I T Y I N I N T E R P R E T I N G W H AT
T H E A N A LY T I C S SAY A B O U T T H E I R P RO G R ES S
TOWA R D S T H E G OA L S
– P ROV I D E H U M A N CO N T E X T TO T H E DATA
– D EC I D E W H AT AC T I O N S TO TA K E A S A R ES U LT O F
T H E I N FO R M AT I O N P ROVI D E D

Learning Analytics
&
Learning Activities
Grounding
Goal Setting
Reflection

GOAL-SETTING
• INDIVIDUAL GOAL-SETTING ALLOWS FOR
MULTIPLE POSSIBLE PROFILES OF PRODUCTIVE
ACTIVITY AND IMPROVEMENT (RATHER THAN
A SINGLE PATH ALL MUST FOLLOW)
• (SELF-SET) GOALS MOTIVATE LEARNERS TO
PUT IN GREATER EFFORTS, SUPPORT SELF-
MONITORING AND INCREASE COMMITMENT
TO MEET CHALLENGES ENCOUNTERED

• Discussion guidelines present metrics as a starting point for
consideration, not as absolute arbiters of engagement
• Goal-setting is an explicit and structured part of the learning
activity as students set weekly goals for engaging in the online
discussions in an online reflection journal (in the LMS)
S A M P L E S T U D E N T G O A L S
“I aim to read all (most) posts *in the discussion+, and actively participate in two
threads in addition to any I create”
“Well, since I didn't hit last week’s goal really I *still+ need to do that, also keep the
length *of my posts+ down and get more interactive with the other kids.”
“As a goal for the next discussion, I will try to synthesize ideas from different
thread areas”
GOAL-SETTING

(DATA INFORMED) REFLECTION
• KEY ELEMENT OF LEARNING ANALYTICS USE
– CONNECTS THE INFORMATION TO THE CONTEXT
TO GENERATE MEANING
• DUAL DANGER OF OMNIPRESENT ANALYTICS
– ABILITY TO REVIEW “ANYPLACE /ANYTIME”
MEANS IT HAPPENS NOWHERE /NEVER
– AT TENTION TO CONSTANTLY AVAILABLE METRICS
DISTRACTS FROM ENGAGEMENT IN LEARNING

• Establish a rhythm for reflection
– Weekly cycle of reviewing the analytics
– Evaluate progress towards the goals
– Assess when the goals themselves need to be updated or revised
• Provide a dedicated space
– Online reflective journal (private wiki in the LMS)
– Supports examination of trajectory over time
SAMPLE STUDENT REFLECTION
“I found that I wanted the challenge of trying to up the % of overall posts that I
reviewed each week. This also meant slowing down my reading since the data
would not record a quick read of the information. The overall result was that I
think I learned more and was able to get a broader sense of opinion concerning the
readings.”
REFLECTION

DIALOGUE
• SPACE OF NEGOTIATION AROUND THE
INTERPRETATION OF THE ANALYTICS
• ANALYTICS AS A START, NOT THE END
– WHAT TO CHANGE IS NOT ALWAYS CLEAR
– STUDENTS MAY NEED HELP TAKING ACTION
• USE OF “NEUTRAL” DATA AS LEVERAGE

• Conversation between each student and the instructor about their
participation, grounded in the analytics
• Conducted thought the online reflective journal
• Simultaneously creates an audience for the reflection and allows for
feedback, suggestions etc.
S A M P L E D I A L O G I C C O M M E N T S
“This week I was out of town to renew my entry visa, so I went to the discussion forum later
than usual, as a result, my role was mainly to build on others' comments or answer
questions, studying more as a listener. Timing is very important for online discussion :) …I
hope I could ...do better next week”
“Despite your comment that you made fewer posts than in previous weeks I notice that you
are still way above the class average. I'm curious to know your thoughts on this - especially
in relation to your goal of wanting to focus more on quality rather than quantity.”
DIALOGUE

REFERENCE FRAME
• COMPARISON POINTS TO WHICH
STUDENTS ORIENT WHEN THEY EXAMINE
THEIR ANALYTICS
– THEORETICAL PAT TERNS
– PEER ACTIVITY
– THEIR OWN PRIOR ACTIVITY

• Continually reminding students of theoretical patterns
• Prompting reflection on individual progress and goals
• Value and danger of comparisons to peers
SAMPLE MENTIONS OF REFERENCE FRAMES
“I was surprised to see that most of classmates checked the forum more than I did…I also
did not expect that they referred [back to] their own post quite many times.”
“Since all my numbers are below the average so that makes me feel, ‘Oh my gosh, I’m kind
of jumping out of this class’ or something like that. It is kind of a little bit – sometimes
depressing.”
“Compared to the previous week, *my+ number of reviews of others’ posts has been hugely
increased … and I did spend more time to read and understand others’ posts.”
REFERENCE FRAME

Integration (technological and pedagogical) made analytics use an
coherent part of the learning process
Students embraced their agency in setting goals and evaluating their
progress, no “big brother” issues
Reflection on data a powerful starting place, concrete and proximal
goal-setting is harder, change happens slowly and can require
support
Reference frames were important for making sense of the data;
reactions can be both cognitive and emotional
Dialouge was powerful but presents challenges for scalability
PILOT WORK

E M B E D D I N G S T U D E N T U S E O F
A N A L Y T I C S A S P A R T O F
L E A R N I N G P R A C T I C E S I N A
P R I N C I P L E W A Y O F F E R S
E X C I T I N G O P P O R T U N I T I E S T O
H E L P S T U D E N T S T A K E C H A R G E
O F T H E I R O W N L E A R N I N G
B A S E D O N D A T A - I N F O R M E D
D E C I S I O N S

All images used and adapted under a creative commons 2.0 attribution license
(http://creativecommons.org/licenses/by/2.0/legalcode)
english106/4398730100; joshuarothhaas/3126559973; departmentofed/9610345404;
csessums/4781752262; daquellamanera/423150347; kb35/430976324; devdsp/6999839463
alyssa_wise@sfu.ca
http://www.sfu.ca/~afw3/research/e-listening/
http://www.slideshare.net/alywise
Supported by the Social Sciences and Humanities Research Council of Canada

Designing Pedagogical Interventions to Support Student Use of Learning Analytics

More Related Content

Similar to Designing Pedagogical Interventions to Support Student Use of Learning Analytics (20)

More from alywise (19)

Recently uploaded (20)

Designing Pedagogical Interventions to Support Student Use of Learning Analytics