Learning Analytics In Higher Education: Struggles & Successes (Part 2)

ACCELERATE LEARNING PERFORMANCE
Learning Analytics in Higher Education:
Struggles & Successes (Part 2)
Presented by Lambda Solutions and UBC

UBC Faculty of Education
Leah Macfadyen, PhD
Meet Our Presenter

What Learning Analytics Means
● learning analytics is
the measurement, collection, analysis and reporting of data about learners and their
contexts for purposes of understanding and optimizing learning and the environments in
which learning occurs.
http://www.solaresearch.org
Data - usage & application
● Beyond LMS clickstream data...learner and learning data of many kinds
● Investigated to offer insights into real teaching and learning questions
Benefits of Learning Analytics vis-a-vis education
● Potential to offer learners, educators, leaders, institutions and governments new tools to
improve learning
Main Takeaways from Session #1

Outline
This ‘Part 2’ session continues this introduction to learning analytics in
education by:
● Considering the current state of LA implementation in higher education.
● Reviewing some of the technical, organization and ethical struggles and
dilemmas that can make implementation of learning analytics projects
challenging.
● Showcasing some real learning analytics use cases (in higher education)

● Stage 1: Extraction and reporting of transaction-level data
● Stage 2: Analysis and monitoring of operational performance
● Stage 3: “What-if” decision support (such as scenario building)
● Stage 4: Predictive modeling & simulation
● Stage 5: Automatic triggers and alerts (interventions)
Poll 1
What stage of Learning Analytics is your organization at?

LA Implementation: Current state
● Stage 1: Extraction and reporting of transaction-level data
● Stage 2: Analysis and monitoring of operational performance
● Stage 3: “What-if” decision support (such as scenario building)
● Stage 4: Predictive modeling & simulation
● Stage 5: Automatic triggers and alerts (interventions)
Goldstein, P. & Katz, R. (2005)

Learning analytics (LA) offers the capacity to investigate the rising tide of
learner data with the goal of understanding the activities and behaviors
associated with effective learning, and to leverage this knowledge in
optimizing our educational systems (Bienkowski, Feng, & Means, 2012;
Campbell, DeBlois, & Oblinger, 2007).
Macfadyen et al., 2014
What is going on?

Cultural and organizational barriers
● Unwillingness to act on findings from other disciplines
● Disagreement over the relative merits of qualitative vs
quantitative approaches
● (Human) tendency to base decisions on anecdote
● Different language spoken by researchers and
decision-makers
● Lack of familiarity with statistical methods (probability)
● Researchers tendency to hedge conclusions
● Data not accessible to decision-makers
See:
● McIntosh (1979)
● Ferguson et al. (2014)

Strategic and structural barriers
1. Educational institutions are stable systems, resistant to change
2. Data struggles
3. Struggles with learning technologies & IT
4. HR needs: lack of relevant skills and resources
5. Leaving the troops behind…
6. “Analytics panic”
7. Lack of clarity about purpose and goals
8. Failure to identify and prioritize stakeholders
9. Failure to consider ethical questions and lack of
relevant data ethics policy.
10. Poor/absent data governance policy and tools
For more on LA strategy and policy, see:
● Ferguson et al. (2014)
● Macfadyen et al.(2014)

https://sheilaproject.eu/

It seems intuitively obvious that a greater understanding of a student cohort
and the learning designs and interventions they best respond to would benefit
students and, in turn, the institution’s retention and success rate. Yet collection
of data and their use face a number of ethical challenges, including location
and interpretation of data; informed consent, privacy, and de-identification of
data; and classification and management of data.
Slade & Prinsloo (2013)

Ethics of data research and review
● Purpose: Why is data being collected? To what end? (Financial?
Educational?)
● Ownership of data
● Informed consent, privacy, de-identification
● How will data be handled and protected? Who should have access to it?
● Ethics of surveillance - power imbalance between educators/institution
and learners
● …but in parallel with changing attitudes to privacy and self-disclosure

Ethics of LA tool and system deployment…
● Is there an obligation to act on new knowledge?
● How do we find the balance between potential “individual harms" and greater
scientific knowledge? Or greater good?
● Questions about the ethics of intervention…does this approach treat learners
as passive objects?
● With what confidence can we act upon findings from interpretation of
incomplete data sets?
● Use of collected data beyond educational purposes (e.g. fundraising)
● Data management and governance – diff. types need diff. levels of protection?

● What is a data policy?
● I have no idea.
● Definitely not.
● We are working on it.
● Yes, we have a well-designed policy in place.
Poll 2
Does your workplace, organization or institution have an
updated data policy sufficient for the learning analytics
era?

Learning analytics is primarily a moral and educational practice, serving
better and more successful learning. The inherent peril and promise of
having access to and analyzing “big data” (Bollier, 2010) necessitate a
careful consideration of the ethical dimensions and challenges of learning
analytics.

Principles for learning analytics
1. LA as a moral practice
2. Learners as agents (and collaborators), not just recipients
3. Recognition that learner identity and performance (and thus labels and
categories) are temporal constructs
4. Learner success is complex and multidimensional - recognition of the
incompleteness and biases of our data
5. Transparency of purpose
6. The necessity of using the data

INSTITUTIONAL LEARNING ANALYTICS POLICIES
1. The Open University, UK (approved in September 2014)
2. Nottingham Trent University, UK (approved in November 2015)
3. University of West London, UK (approved in September 2016)
4. Charles Sturt University (CSU), Australia (version 3.2, approved on 16 September 2015)
5. The University of Sydney (USyd), Australia (approved in April 2016)
6. The University of Edinburgh (policy principles), UK (approved on 2 May 2017); Full
policy (approved in May 2018)
7. The University of Wollongong (UOW), Australia (approved in December 2017)
OTHER LEARNING ANALYTICS POLICIES
1. Jisc (UK) – Code of Practice for Learning Analytics (published in June 2015)
2. National Union of Students (UK) – Learning Analytics: A Guide for Students’ Unions (published
in August 2015)
3. EU-funded LACE project – A DELICATE checklist (published in April 2016)
4. EU-funded LEA’s Box project – Privacy and Data Protection Policy (published in December 2014)

LA implementations that have failed…..

Course Signals (Purdue University, USA)
● “Early alert system” for instructors
● Predicted risk of failure using
pre-college preparation information
(high school GPA, standardized test
scores, and socioeconomic status),
and post-admission performance
data (in the form of grades, advising
visits, and use of the LMS).
● ‘Signals’ alert instructors (and
students, if enabled)
See: Caulfield, M. (2013)

Degree Compass
(Austin Peay State University, USA, later acquired by D2L/ Brightspace)
● ”Recommender system” (Netflix!)
● “Reverse Degree Audit” algorithm made
use of academic preparation data, final
grades, and course registration choices of
past students, to help current students by
recommending courses in which they are
more likely to succeed.
See:
● Denley, T. (2013)
● Johnson, S. (2018)

LA implementations that show promise...

Check my Activity
(University of Maryland, Baltimore County)
● Uses LMS (Blackboard) data metrics to
allows learners to see their own activity in
comparison with the class average.
● When the LMS gradebook is used in a
course, this tool also offers learners a
summary of activity levels of peers who
earned lower or high grades
See: Fritz (2010)

AcaWriter (University of Technology Sydney, Australia)
● A formative feedback app for
student writing.
● Natural Language Processing
(NLP) tool identifies concepts,
people, places, and the
meta-discourse corresponding to
rhetorical moves.
● Gives rapid formative feedback
on draft work.
Demo at: http://acawriter-demo.utscic.edu.au/demo

E2Coach(University of Michigan, USA)
● A recommendation engine designed to
help current students pass difficult
courses.
● Provides feedback and advice from peers
who performed “better than expected” in
a prior version of the same course.
● Struggling students can opt in to get
advice and tips from successful peers who
‘look like them’ demographically.
See: Michigan's World Class: Electronic coaching
offers tailored help.

OnTask(An ongoing multi-university
collaboration)
● A system that helps instructors select
and make use of data about student’s
activities throughout the semester
● Allows instructors to design
personalized feedback with
suggestions about their learning
strategies.
● Can accept data from various sources
such as LMS, video engagement,
assessments, student information
systems, electronic textbooks,
discussion forums, etc.
https://www.ontasklearning.org/

Threadz
(Eastern Washington University,
USA)
● A plugin tool for the Canvas LMS that visualizes learner
engagement.
● Assists in identifying specific behaviors and
characteristics within the course, such as: learner
isolation, non-integrated groups, instructor-centric
discussions, and key integration (power) users and
groups.
● Allows instructors to to see who is engaged, whose
responses are going unanswered, and whether students
are achieving the engagement goals that were set for
the course.
● Alerts instructors to learner isolation and
instructor-centric discussions. https://threadz.ewu.edu/

● Learning analytics is a new and emerging field; R&D is ongoing.
● Most institutions and organizations are only in the early and testing stages of
adoption and implementation.
● A variety of cultural/institutional and structural factors make implementation
challenging.
● Data ethics and governance policies are a vital precondition for success.
● LA use cases illustrate additional factors that can contribute to failure or
success.
Conclusion / Main Takeaways

Lambda Lab: Maximize Your eLearning
Effectiveness With Meaningful Connections
What’s Next
Tuesday, July 30th 2019
10:00 AM PT | 1:00 PM ET
https://go.lambdasolutions.net/maximize-your-elearning-effectiveness-with-mea
ningful-connections

TOLL FREE +1.877.700.1118
EMAIL SALES@LAMBDASOLUTIONS.NET
WWW.LAMBDASOLUTIONS.NET

Learning Analytics In Higher Education: Struggles & Successes (Part 2)

More Related Content

What's hot (19)

Similar to Learning Analytics In Higher Education: Struggles & Successes (Part 2) (20)

More from Lambda Solutions (20)

Recently uploaded (20)

Learning Analytics In Higher Education: Struggles & Successes (Part 2)