Measuring student engagement with learning technology

MEASURING STUDENT
ENGAGEMENT WITH
LEARNING TECHNOLOGY
David Havens
June 18, 2014
ABSTRACT
The nature of education, interaction, and engagement is rapidly changing as new modes of communication and
technologies enter the hands of learners. While teachers are the greatest lynchpin for keeping students and
classrooms engaged, there are many features tech tools can employ to help. For maximum engagement, technology
tools in learning must appeal to social motivation, have opportunities for creativity, personalize the content and
experience, engage a mentor or teacher, and provide interactivity and immediate feedback. Measuring this
engagement must combine insights from both qualitative and quantitative data.
OUTLINE
• Background
• Existing frameworks for student engagement
• Frameworks for student engagement while using technology
• Measuring engagement while using technology

NewSchools Venture Fund Page 2
©2014. NewSchools Venture Fund. All rights reserved. June 18, 2014
INTRO
In the age of mobile computing, attention and
engagement have become commodities. Advertisers
are willing to spend millions for just 30 seconds of a
viewer’s attention. With all the ads and apps buzzing
in students’ pockets in the 21st
century classroom, we
had to ask ourselves: how are learning technologies
(books, desktops, tablets, and otherwise) keeping up
with the rising bar of grabbing attention? What are
the best practices for engaging students today?
Specifically relevant to the NewSchools Seed Fund,
how can we measure the role technology can play in
student engagement?
In this paper I seek to explore how existing
frameworks for engagement in education can be
adapted to better account for new technology and to
suggest a framework for evaluating qualitative
engagement using technology. As consumer and
learning technologies continue to grow, improving
methods to capture and grow student engagement
becomes increasingly more challenging – and more
important.
BACKGROUND
While much has been written on engagement in the
classroom and on engagement with technology tools,
not much research has been done on the intersection
of the two. A roomful of students with tablets in their
hands tapping away at the screen, quiet and engaged,
is becoming an increasingly familiar scene (with 10
million iPads sold to US K12 schools at time of
publication). With use cases proliferating, how can a
teacher truly understand students’ thought processes
and level of intellectual engagement with the material?
To answer this question, let’s first reflect on what we
know about student engagement. Engagement is a
prerequisite to learning and has many benefits backed
up in the literature. Research suggests the depth of
engagement correlates to the depth of learning (Carini
2004). Moreover, engagement has been shown to be a
“protective factor” with respect to education risk
(Resnick 1997, cited in Christensen 2012). Third,
longitudinal studies have found that small changes in
measures of school engagement could dramatically
change a students likelihood to complete higher
education and go on to a professional or managerial
career, even overcoming longstanding socioeconomic
barriers (Abbott-Chapman 2013).
How we define student engagement has evolved over
time as well, from the observable (turning the pages
in a book) to the emotional and cognitive (grappling
with the content in the book). In other words,
researchers have progressed from tracking behaviors
such as participation and time on task to students’
investment in learning and use of deeper learning
strategies (Fredericks & McColsky 2011). As a
concrete example, in the early 1990’s, Apple Inc. felt
pressure to measure engagement by recording the
time students spent looking at the screen or instructor
(Sandholtz et al 1992). By comparison, the Apple iPad
is now analyzed for its contribution to “engagement,
collaboration, and perseverance” (Harrold 2012).
EXISTING FRAMEWORKS FOR ENGAGEMENT
Given that student engagement is both valuable and
complex, the next step is to measure and strengthen
it. A full review of engagement techniques is beyond
the scope of this white paper, but many of the
popular views all rely on one central point: the
teacher.
Indeed, in a classroom the teacher holds the most
effective tools for enhancing engagement. In his book
The Art and Science of Teaching, Robert Marzano lays out
the dimensions of engaged learning, noting that it is
the role of a teacher to supply the following through
instructional design and practice:
• High energy: through physical activity, pacing,
focus, and enthusiasm
• Missing information: games, fill in the blanks, and
other gaps
• Appeal to the self system: putting work in the
context of each individual learner
• Mild pressure: to focus student attention
• Mild controversy or competition: structured
debate or points

Marzano notes that each of these techniques is a tool
to be used at the right time by a teacher, who ought
to be constantly assessing the level of engagement in a
classroom and making a judgment call on which
technique to use.
Beyond Marzano’s toolkit, other frameworks for
understanding engagement have different insights and
uses. Daniel Pink has made himself famous for
breaking motivation into three categories: autonomy,
mastery and purpose (Pink, 2009). Strong, Silver and
Robinson put forth a similar theory in a piece in
Education Leadership, stating the keys to motivation
are SCORE, or:
• Success (the need for mastery),
• Curiosity (the need for understanding),
• Originality (the need for self-expression),
• Relationships (the need for involvement with
others).
Whatever model a teacher wants to use, it will still
need to be measured. While many tools exist, they all
center around three core techniques: self-report,
teacher-report, and observation.
FRAMEWORKS FOR ENGAGEMENT WITH
TECHNOLOGY
On the opposite end of the spectrum, the term
“engagement” is thrown around a lot in Silicon Valley
when talking about apps and websites. Engagement in
this sense often does not get beyond “did the user
login every day” or “how many pages did the user
visit” and “how long did the user stay?”
Educational apps like eSpark have done a great job
breaking down how to think about the value of an
individual app when it comes to engagement. You can
read more in their whitepaper Student Engagement in
Educational Apps where they lay out a rubric of app
quality (Lopuch 2013). According to Lopuch, the
quality of an app depends on the Alignment,
Authenticity of Task, Scaffolding of Learning,
Intuitiveness, Student Engagement, and Cost. In this
situation, engagement is measured simply by asking a
student the yes or no question “did you like this
activity?” Through this metric they can conclude that
animation, background music, and badges or rewards
have a positive effect on student engagement while
menus of activities, drill and kill problems, silent apps
have a negative effect.
This is where apps for use in the classroom play a
unique and un-researched role in the education
ecosystem. They keep the teacher in the center and
also have the benefits of measurement that traditional
direct to learner apps have.
To understand engagement with hybrid technology
tools that blend the digital and the physical worlds to
engage students from multiple angles, I propose a
new framework for looking at engagement as well as
an initial student- and teacher- report survey to dig
into the results. A good technology tool for use in the
classroom should include measurable elements of
social motivation, creativity, personalization, educator
engagement, and interactivity.
1. Social Motivation: put the learning in the context
of the student’s social environment. Can include
elements of competition, collaboration, and
gamification. For example:
• When a student gets stuck in math app Front
Row Education, they are connected to another
student in the classroom that has demonstrated
mastery on the concept to receive help
• When a students submits answers to closed or
open response questions on Socrative, the teacher
has the option of displaying that student’s name –
raising the stakes on the quality of her answer
2. Creativity within structure: tie the amazing
creation technology of tools in the classroom to
lessons to enhance autonomy, curiosity, and
originality. For example:
• Nearpod enables students to take pictures, remix
media, and draw ideas before submitting them for
the teacher to review or share
• Students are able to bring their imaginations to
life by using Tynker to visually code animations
and interactive programs
3. Personalization of content: keep students in their
Zone of Proximal Development with content that is
relevant to their lives and at their competency level
and modified for their learning profile. For example:
• Glean will feed students a different video based
off how long you engage with different styles of

online teachers (at a whiteboard, animated videos,
using PowerPoint, etc.)
• Newsela will level current events and news stories
to five different levels so all students can access
the content
• Knewton quizzes students and gives them
formative math problems that fill in their gaps in
understanding
4. Educator engagement: how well can a teacher or
mentor see what is going on or give live feedback. For
example:
• Classwork for iPad allows teachers to see exactly
what is happening on every screen in one mission
control view as students work through worksheets
or draw pictures on the iPad
• Teachers aren’t limited to multiple choice data,
either: FreshGrade allows teachers to capture
qualitative data in the classroom and share that
with students and parents, involving more
stakeholders in the learning process
5. Interactivity: provide immediate feedback, ability
to rewind or review, and checks for understanding.
For example:
• Students are able to dynamically interact with
sheet music on Pluto Media music apps allowing
them to receive multimodal stimulation through
sight, sound, and touch while studying music in
the absence of an instrument
• When students finish a quiz on Nearpod, they are
immediately shown their results on any
quantitative items so they can correct and
understand their mistakes
MEASURING ENGAGEMENT
As a first step towards understanding how students
engage with hybrid learning models we recommend
sending a survey to students using different products.
This survey is designed for students and can be used
for teachers by replacing “I” with “my student(s).”
These are starter questions for entrepreneurs and tech
companies to start thinking about what types of
questions will be useful when performing engagement
surveys for their users. The questions are intended to
be answered on a Likert scale of agreement, frequency
or quality.
• I have meaningful interactions with other students
while using EDUAPP
• Someone I care about will see the work I do in
EDUAPP
• I feel like I can improve my learning by working
on EDUAPP
• I try hard on assignments given to me on
EDUAPP
• I am able to express my creativity while using
EDUAPP
• How meaningful do you find the topics you study
while using EDUAPP
• My teacher knows how I am doing in this app
• EDUAPP helps my teacher or parent teach me
better
• My teacher plans lessons where EDUAPP is
embedded in a bigger unit
• The content and questions in EDUAPP challenge
me
• Using EDUAPP helps me understand the content
• I am personally interested in the topics I learn
about on EDUAPP
• How often do you go back and review material
from earlier lessons using EDUAPP
• How easy is it to get help while you are using
EDUAPP?
• Anything else you want to tell us about this app?
What were the best things? What could be better?
If you run a survey on student engagement with your
tool, please share it with NewSchools Venture Fund
(twitter: @nsvf).
REFERENCES:
Abbott-Chapman, J., Martin, K., Ollington, N., Venn,
A., Dwyer, T. and Gall, S. (2013), The longitudinal
association of childhood school engagement with
adult educational and occupational achievement:
findings from an Australian national study. British
Educational Research Journal. Retrieved June 18,
2014, from
http://onlinelibrary.wiley.com/doi/10.1002/berj.3031/ab
stract
Carini, R.M., Kuh, G.D., Klein, S.P. (2004). Student
Engagement and Student Learning: Testing the
Linkages. Research in Higher Education. Retrieved

June 18, 2014, from
http://nsse.iubres.edu/pdf/research_papers/testing_linka
ges.pdf
Christenson, S. (2012). Handbook of research on student
engagement. New York: Springer. (link)
Fijor, M. (2010, April 10). Defining Student
Engagement with Technology - New School
Technology. New School Technology. Retrieved June 18,
2014, from
http://www.newschooltechnology.org/2010/04/defining-
student-engagement-with-technology/
Fredericks, J., & McColsky, W. Measuring student
engagement in upper elementary through high school:
a description of 21 instruments . Regional Education
Laboratory at UNC, REL 2011. Retrieved June 18,
2014, from
http://ies.ed.gov/ncee/edlabs/regions/southeast/pdf/RE
L_2011098.pdf
Harrold, R. (2012). The iPad Effect: Leveraging
Engagement, Collaboration, and Perseverance. The
International Educator. Retrieved from:
http://www.tieonline.com/view_article.cfm?ArticleI
D=100
Lopuch, M. (2013). Student Engagement in
Educational Apps. eSpark Learning. Retrieved May
2014 from http://www.esparklearning.com/student-
engagement-educational-apps-research/
Pink, Daniel. (2009, July). The puzzle of motivation.
[Video file]. Retrieved from
http://www.ted.com/talks/dan_pink_on_motivation
Reading, M. (2014) The Three Foundational
Principles of Student Engagement and Student
Motivation. Retrieved June 18, 2014, from
http://teacherstraining.com.au/the-three-foundational-
principles-of-student-engagement-and-student-motivation-
pt1/
Resnick, M.D., Harris, K.M., & Shew, M. (1997).
Protecting adolescents from harm: Findings from the
National Longitudinal Study on Adolescent Health.
Journal of the American Medical Association, 278,
823-832.
Sandholtz, J.H., Ringstaff, C. Student Engagement
(1991): Views from Technology Rich Classrooms.
Apple Computer, Inc. Retrieved June 18, 2014, from
http://www.apple.com/euro/pdfs/acotlibrary/rpt21.pdf
Acknowledgements: thanks to Jennifer Wolf, Karin
Forssell, and Jennifer Carolan for feedback and
guidance along the way.

Measuring student engagement with learning technology

More Related Content

What's hot (20)

Similar to Measuring student engagement with learning technology (20)

Recently uploaded (20)

Measuring student engagement with learning technology