Improving Student Achievement with New Approaches to Data

John Whitmer, Ed.D.
Academic Technology Services
California State University, Office of the Chancellor
WASC ARC Conference
April 11, 2013
Improving Student Achievement with
New Approaches to Data:
Learning Analytics &
the CSU Data Dashboard
slides @ slideshare.net/JohnWhitmer/

Outline
1. Context: California State University &
Graduation Initiative
2. Chico State Learning Analytics Case Study
3. CSU Data Dashboard Project
4. Next Steps
5. Discussion

1. CONTEXT

California State University
http://calstate.edu
 23 campuses
 437,000 FTE students
 44,000 faculty and staff
 Largest, most diverse, &
one of the most
affordable university
systems in the country
 Play a vital role in the
growth & development of
California's communities
and economy

CSU Achievement Gap

.
– Baseline 6-Year Graduation Rate: 46%
– Target 6-Year Graduation Rate: 54%
– Baseline Achievement Gap: 11%
– Target Achievement Gap: 5.5%
2

New Approaches to Using Data
Enable data-driven decision making for
interventions earlier in the student experience by
1. Integrate new data sources & variables
2. Disseminate findings to a broader audience
3. Provide ability to interact with data analysis,
conduct ad-hoc and custom reporting

2. CHICO STATE LEARNING
ANALYTICS CASE STUDY

Economist. (2010, 11/4/2010). Augmented business: Smart systems will disrupt lots of industries, and perhaps the entire economy.
The Economist.
200MB of data emissions annually!

Source: jisc_infonet @ Flickr.com
Source: jisc_infonet @ Flickr.com
Logged into course within 24
hours
Interacts frequently in
discussion boards
Failed first exam
Hasn’t taken college-level
math
No declared major

Case Study: Intro to Religious Studies
• Undergraduate, introductory, high
demand
• Redesigned to hybrid delivery format
through “academy eLearning program”
• Enrollment: 373 students
(54% increase on largest section)
• Highest LMS (Vista) usage
entire campus Fall 2010
(>250k hits)
• Bimodal outcomes compared to
traditional course
• 10% increase on final exam
• 7% & 11% increase in DWF
• Why? Can’t tell with aggregated data
54 F’s

Learner Analytics
“ ... measurement, collection, analysis and
reporting of data about learners and their
contexts, for purposes of understanding and
optimizing learning and the environments in
which it occurs.” (Siemens, 2011)

Pervasive Adoption of Learning Management Systems
Institution-Supported IT
Resources and Tools. Reprinted
from “The ECAR Study of
Undergraduate Students and
Information Technology,” Eden
Dahlstrom, 2012 by the
EDUCAUSE Center for Applied
Research.

Guiding Questions
1. How is student LMS use related to academic
achievement in a single course section?
2. How does that finding compare to the relationship of
achievement with traditional student characteristic
variables?
3. How are these relationships different for
“at-risk” students (URM & Pell-eligible)?
4. What data sources, variables and methods are most
useful to answer these questions?

LMS Use Variables
1. Administrative Activities
(calendar, announcements)
2. Assessment Activities
(quiz, homework, assignments,
grade center)
3. Content Activities
(web hits, PDF, content pages)
4. Engagement Activities
(discussion, mail)
Student Char. Variables
1. Enrollment Status
2. First in Family to Attend
College
3. Gender
4. HS GPA
5. Major-College
6. Pell Eligible
7. URM and Pell-Eligibility
Interaction
8. Under-Represented
Minority
9. URM and Gender
Interaction

Tools Used
App Function
Excel Early data exploration; simple sorting; tables
for print/publication
Tableau Complex data summaries and explorations;
complex charts; presentation charts
Final/formal descriptive data; statistical
analysis; some charts (scatterplots)
Statistical analysis (factor analysis)
Statistical analysis (charts)

Correlation: Student Char. w/Final Grade
Scatterplot of
HS GPA vs. Course
Grade

Predict the trend
 LMS use and final grade is _______ compared to
student characteristics and final grade:
a) 50% smaller
b) 25% smaller
c) the same
d) 200% larger
e) 400% larger

Correlation LMS Use w/Final Grade
Scatterplot of
Assessment Activity
Hits vs. Course
Grade

Chart: LMS & Student Characteristics

Combined Variables Regression Final Grade by
LMS Use & Student Characteristic Variables
LMS
Use
Variables
25%
(r2=0.25)
Explanation of change
in final grade
Student
Characteristic
Variables
+10%
(r2=0.35)
Explanation of change
in final grade
>

Predict the trend
 LMS use and final grade is ______ for “at-risk”*
students compared to not at-risk students?
a) 50% smaller
b) 20% smaller
c) No difference
d) 20% larger
e) 100% larger
Relationship indicates how strongly LMS use is correlated
with final grade; lower value equals less impact
*at-risk = BOTH under-represented minority and Pell-eligible

Predict the trend
 LMS use and final grade is ______ for “at-risk”*
students compared to not at-risk students?
a) 50% smaller
b) 20% smaller
c) No difference
d) 20% larger
e) 100% larger
*at-risk = BOTH under-represented minority and Pell-eligible

Question 3 Results:
Regression by “At Risk” Population Subsamples

At-Risk Students: “Over-Working Gap”
27

Activities by Pell and Grade
Extra effort
in content-
related
activities

Conclusions
1. LMS use is a better predictor of academic
achievement than student characteristics.
– LMS use frequency is a proxy for effort.
2. LMS data requires extensive filtering to be useful;
student variables need pre-screening for missing
data.
3. LMS effectiveness for at-risk students may be
caused by non-technical barriers.
4. Small strength magnitude suggests that better
methods could produce stronger results.

Next Generation Learning Analytics
Graphic Courtesy Sasha Dietrichson, X-Ray Research SRL

Next Steps
Potential for improved LMS analysis methods:
 time series analysis
 social learning
 activity patterns
 discourse content analysis
Group students by broader identity, with unique
variables:
 Continuing student (Current college GPA, URM, etc.)
 First-time freshman (HS GPA, SAT/Act, etc)

3. DATA DASHBOARD PROJECT

THE FRAMEWORK
Advancing by Degrees: A
Framework for Increasing
College Completion by
Offenstein, Moore &
Schulock
Institute for Higher
Education Leadership and
Policy and The Education
Trust (http://bit.ly/10QtMXC)

This research describes
academic patterns (or leading
indicators) that occur early in
the pipeline that can be tracked
and monitored in real time
against milestones on the
graduation route.

Leading indicators statistically
improve predicted
probabilities of completion
over just the use of student
background characteristics

Milestones are measurable
educational achievements
that students reach along
the path to degree
completion.

Milestones Leading Indicators
Year-to-year Retention
Transition to college level coursework
(English and Math)
Earn one year of college level credits
Complete General Education
Complete degree
Remediation
Begin remedial coursework in the first term, if
needed.
Complete needed remediation
Gateway Courses
Complete college-level math and/or English in
the first or second year
Complete a college-success course or other
first-year experience program
Credit Accumulation and Related Academic
Behaviors
Complete high percentage of courses
attempted (low rate of course dropping and/or
failure)
Complete 20-30 credits in the first year
Earn summer credits
Enroll full time
Enroll continuously, without stop-outs
Register on-time for courses
Maintain adequate academic progress

Driving Questions for Dashboard
1. What percentage of students reach each of the
leading indicators?
2. What is the impact of reaching each of the
leading indicators on success rate?
3. Does meeting any of the indicators reduce or
eliminate gaps between student demographic
groups?

PROOF OF CONCEPT

Purpose
 Demonstrate potential value of combined
reporting and statistics
 Evaluate availability and integration of data
 Pilot potential tools in real-world scenario
 NOTE: production system may be dramatically
different from POC, given lessons learned and
scalability

1. Report
Parameters

2. Retention Rates

3. Retention Rates by URM Status

4. Data Export Options

Concern: Male, 2nd
Year Persistence

4. NEXT STEPS

What’s Now … And Next
 Conducting 3 multi-campus pilots
1. mCURL: Moodle Common Usage Reporting & Learning
Analytics: (8 CSU & 2 UC campuses)
2. Blackboard Analytics for Learn (3 campuses)
3. LMS-agnostic campus surveys
 Investigating additional pilot with LMS-agnostic tool to
move beyond “clickometry” into social network analysis,
discourse analysis, etc.
 Raises question for MOOC research: relationship between
student intent/motivation, student characteristics/leading
indicators, MOOC use, and achievement

Data
Dashboard
ERS
Data
CCA
Data
LMS
Data
Other
Data
Sources
Data Dashboard

Feedback? Questions?
John Whitmer
jwhitmer@calstate.edu
 Monograph @
http:www.johnwhitmer.net
 Twitter: johncwhitmer
Desdemona Cardoza
dcardoza@calstate.edu

Improving Student Achievement with New Approaches to Data

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Improving Student Achievement with New Approaches to Data (20)

More from John Whitmer, Ed.D. (20)

Recently uploaded (20)

Improving Student Achievement with New Approaches to Data

Editor's Notes