Best Practices for Evaluating Mobile Learning
1 like320 views
The document discusses best practices for evaluating mobile learning projects, emphasizing the importance of effective evaluation to assess impacts and returns on investment. It presents case studies from various mobile learning initiatives in different regions and offers eight tips for successful evaluation processes. The insights underscore the need for flexible methodologies and meaningful metrics to measure the effectiveness of mobile learning in educational settings.
Best Practices for Evaluating Mobile Learning
- 1. Best Practices for Evaluating Mobile Learning: Real Stories from Around the Globe Dr. Julie A. Evans CEO, Project Tomorrow June 25, 2018 @SpeakUpEd @JulieEvans_PT
- 2. Dr. Julie A. Evans Chief Executive Officer, Project Tomorrow Chief Researcher, Speak Up Research Project Mobile Learning Project Evaluator since 2007 Also, skilled in the evaluation of blended learning and digital content usage in K-12 and higher education Thank you for being here!
- 3. Special Guest: Hiram Lopez-Landin Senior Analyst, Wireless Reach Qualcomm
- 4. Today’s discussion topics ▪ About our work o Project Tomorrow and the Speak Up Research Project o Qualcomm Wireless Reach ▪ Why are evaluations of mobile learning important today? ▪ State of mobile learning today & evaluations for mobile learning ▪ Let’s talk about 3 different mobile learning projects ▪ My top 8 tips and strategies for evaluating mobile learning ▪ Extended learning – links to some additional resources ▪ Your thoughts, insights, questions
- 5. Tweet with us! @JulieEvans_PT @SpeakUpEd @hlopezlandin @QCWirelessReach
- 6. Getting to know you!
- 7. About Project Tomorrow (www.tomorrow.org) ▪ Nonprofit education organization supporting K-12 education since 1996 ▪ Mission is to ensure all of today’s students are well prepared for the future ▪ Programs and research focus on role of digital tools within the education ecosystem – believe in power of STEAM to support student preparation for college and career success o Speak Up Research Project on Digital Learning: collecting & reporting on the authentic feedback of K-12 stakeholders to inform federal, state & local programs and policies
- 8. About the Speak Up Project (www.tomorrow.org/speakup) ▪ Annual research project since 2003 ▪ Uses online surveys + focus groups ▪ Facilitated 100% through schools and districts ▪ We design online surveys to collect feedback from your K-12 Students, Teachers, Parents, Administrators, and Community Members ▪ All K-12 schools – public, private, parochial, charter, virtual - are eligible to participate ▪ Project Tomorrow manages all data collection and reporting for you - 100% free service ▪ Schools get summary report with all locally collected data + state and national data for benchmarks We share national data with federal, state and local policymakers to inform programs and funding
- 9. About the Speak Up Project (www.tomorrow.org/speakup) Survey question topics include: ✓ Use of technology to support learning ✓ School climate for innovation ✓ College and career ready skill development/interest ✓ Leadership challenges ✓ Teachers’ needs for professional learning ✓ How do different stakeholders value digital learning ✓ Emerging trends w/digital tools, content and resources ✓ New classroom models: mobile, blended, flipped ✓ School to home communications ✓ Designing the ultimate school ❖ Activities ❖ Attitudes ❖ Aspirations Since 2003, 5.4 million K-12 stakeholders have submitted a Speak Up survey
- 11. 11 45 PROGRAMS 17 COUNTRIES 150 STAKEHOLDERS 270K PEOPLE Wireless Reach Education Portfolio Impact to Date
- 12. 12 17 9 7 6 2 2 1 1 United States East Asia Southeast Asia South Asia Middle… Latin America Sub-Saharan… Europe Programs by Region Percent of female beneficiaries in all education programs 49% Organizations served in education portfolio570 Wireless Reach Education 10 Years of Accomplishments Devices used in education portfolio11,000 All figures include aggregated data from 2006 to 2017
- 13. 13 Education Strategy Wireless Reach education programs are designed to address the barriers to adoption of wireless technology in the classroom including the need for: • Digital content & assessment • Infrastructure & IT support • Privacy & security
- 14. Why is it important to evaluate our mobile learning projects?
- 15. Why is it important to evaluate our mobile learning projects? Are our investments yielding a return on investment?
- 16. What do you see? Are we all seeing “evaluation” or even “mobile learning” the same way?
- 17. State of Mobile Learning: selected research findings from Speak Up 2017 State of Mobile Learning: selected research findings from Speak Up 2017 Learning on the go – anytime, anywhere
- 18. Survey Audience # of Surveys Submitted K-12 Students 340,927 Teachers & Librarians 34,833 Parents 23,159 Administrators 3,249 Community Members 4,611 About schools and districts: 3,641 districts, 10,619 schools 29% urban, 34% rural, 37% suburban, 68% title 1 eligible National participation in Speak Up 2017: 406,779
- 19. @Project Tomorrow 2017 Students have more personal access to mobile devices than ever before 35% 50% 79% 91% 53% 57% 55% 39% 32% 35% 48% 54% 10% 15% 14% 13%15% 12% K-2 Gr 3-5 Gr 6-8 Gr 9-12 Smartphone Tablet Laptop Digital Reader Smartwatch
- 20. @Project Tomorrow 2017 Teachers and mobile devices within learning Classroom access is changing rapidly In class access 2015 2017 BYOD 17% 23% 1:1 in school assignment 20% 31% 1:1 in school + home assignment 12% 14% Cart access 31% 30% No access 31% 18% Not surprising: teachers want more PD on how to effectively integrate mobile devices within instruction
- 21. @Project Tomorrow 2017 Administrators say mobile learning = positive results ▪ 49% of school principals say they see positive academic results from their 1:1 mobile learning programs Benefits of mobile learning Elementary school principals Middle school principals High school principals Students develop topics more fully 82% 83% 77% Students are more engaged in learning 72% 62% 55% Preparation for college/work 71% 71% 71% Increased access to online content 69% 73% 73% Learning is more personalized 65% 60% 57% Students are collaborating more 57% 58% 58% Improved teacher-student communications 48% 58% 63%
- 22. But what metrics are school and district leaders using to measure outcomes from mobile learning?
- 23. Turn to a neighbor and share: How are you evaluating the success of your mobile learning programs? How satisfied are you with your evaluation process?
- 25. Administrators say these are the top ways they are measuring the impact of their digital initiatives: Metrics % of district administrators nationwide Which ones are you using? Student engagement levels 77% Classroom observations 69% Student feedback 62% Student achievement results 60% Teacher buy-in 54% Teacher feedback 53% Parent feedback 37%
- 26. Two of the 8 essentials: ✓ Purposeful planning for mobile device usage ✓ Measuring project results with meaningful metrics
- 27. ✓ Purposeful planning for mobile device usage ▪ What are your goals? ▪ What outcomes do you want to see? ▪ Why is this important for your school or district? ✓ Measuring project results with meaningful metrics ▪ Aligning metrics with goals ▪ Leveraging appropriate tools ▪ Being open to what you don’t know yet Evaluating mobile learning begins with project planning
- 28. Introduction to 3 mobile learning projects, the metrics they used/are using to evaluate program success & lessons learned that can inform your efforts: ❑ “Making Learning Mobile” – Chicago ❑ “STEAMing Ahead with Mobile Learning – San Diego ❑ “Classroom of the Future” – Turkey Let’s get real!
- 29. “Making Learning Mobile” – Chicago o Three year focused study on the impact of extended Internet access and mobile learning on student achievement o Study group: 5th grade students and their teachers at Falconer Elementary School in Chicago, IL o Students each assigned a Samsung Android tablet with 4G LTE enabled education broadband to use at school and at home Partners: Kajeet, Qualcomm Wireless Reach, Project Tomorrow
- 30. “Making Learning Mobile” – Chicago Objectives of the project was to learn more about: ✓ How did the students and teachers use the tablets and extended Internet connectivity for learning? ✓ What benefits did they derive from those experiences? ✓ What was the impact of the experiences on student learning and overall academic achievement? Evaluation metrics included: • Feedback data from students and teachers – analysis of changes over 3 years • Classroom observations to evaluate depth of usage and impact of home access • Analysis of student achievement results
- 31. “Making Learning Mobile” – Chicago Key findings from the study: 1. Tablet became an indispensable learning tool 2. Tablets and extended connectivity enabled greater communications between students and teachers 3. Benefits to students included development of college/career ready skills and new learning behaviors 4. Teachers’ skills, motivation and readiness for tablet integration impacted effectiveness of usage 5. Purposeful integration within math = higher student achievement and growth
- 32. “Making Learning Mobile” – Chicago Insights from the evaluation of this project: ▪ Build your evaluation methodology and tools to be flexible and open to “surprises” ▪ Understand that the teacher adoption/assimilation/adaption process may be a three year adventure – be reasonable with project goals and the timeline for intended outcomes, and adjust your logical framework accordingly ▪ Reality alert: traditional data collection and analysis tools may not work for mobile learning ▪ With mobile learning, it is essential to collect and analyze both quantitative and qualitative data – feedback matters ▪ Keep eyes wide open for lessons learned – these are gold!
- 33. “STEAMing Ahead with Mobile Learning” – San Diego Partners: San Diego Public Library, e3 Civic High School, Qualcomm Wireless Reach, Project Tomorrow 9th graders were provided with 4GL tablet that included an AR app to learn about the STEAM elements within the Central Library Dome design, engineering and construction
- 34. “STEAMing Ahead with Mobile Learning” – San Diego Objectives of the project was to learn more about: ✓ How a contextually relevant learning experience for students could introduce students to the role of STEAM within construction and design ✓ The role of tablets and 4GL in increasing student engagement in learning ✓ How to effectively increase student interest in STEAM career fields ✓ The ways in which this application could be used within other curricular fields and with the general public Evaluation metrics included: • Feedback data from students and teachers • Observation of the learning activity with “person on the street” interviews • Analysis of changes in students’ interest levels in STEAM career fields
- 35. “STEAMing Ahead with Mobile Learning” – San Diego Key findings from the study: 1. Students said use of AR content increased their engagement in learning 2. Benefits included having opportunity to work collaboratively with classmates on a real world activity; enjoyment level surprised the students! 3. Different learning modalities within the experience appealed to different students: videos, images, text, assessment 4. 40% of the were more interested in a STEAM career field as a result of this learning experience 5. Teachers’ excited about how to use AR in other curricular areas; library staff exploring ways to bring same content to general public
- 36. “STEAMing Ahead with Mobile Learning” – San Diego Insights from the evaluation of this project: ▪ A small project evaluation period demands tighter adherence to project goals. Project goals therefore must be clearly defined with broad buy-in from all stakeholders and partners. ▪ Critical to use the right tools to collect data. Example: to show change in attitudes, leverage pre and post data collection to “see” process of discovery and new ideas ▪ Reality alert: students are not familiar with all new technologies and thus, the awareness/tool learning process must be taken into account within the analysis of impact ▪ With a new technology such as AR, bring teachers into the evaluation development process upfront so that the analysis of the outcomes is based upon a realistic understanding of the environment
- 37. “Classroom of the Future” – Turkey Partners: Turkish Education Association, Qualcomm Wireless Reach ▪ 7th grade students and their teachers from 7 schools have Qualcomm-enabled tablets with wireless connectivity to support learning in school and at home ▪ Initial focus was on English instruction classes but project has expanded to include other subject areas now ▪ Teachers were provided with specialized professional development to support this project
- 38. Objectives of the project was to learn more about: ✓ How the inclusion of mobile devices in the learning environment enhances student learning and engagement particularly in reading and writing ✓ How the mobile learning experience improves students’ perceptions of their technology skills especially as an important global workplace skill ✓ If the teacher PD and the use of the devices in the classroom results in increases in teachers’ capacity, comfort and confidence in implementing mobile learning Evaluation metrics included: • Feedback data from students and teachers – pre and post surveys • Aligning pre and post survey data elements directly with project goals • Linking teacher data with student data for analysis of impact
- 39. Key findings from the study (so far!): 1. Students and teachers agree: the use of the tablets with wireless connectivity both at school and at home are important to support learning. 2. 3/4 of the teachers reported: mobile learning environment resulted in improved student engagement, enhanced quality of their students’ work and students taking greater responsibility for their own learning. 3. The availability of the 1:1 tablets for student use at school and at home increased the students’ overall use of technology for learning - 86% of the students developed stronger technology skills during the school year. 4. With the new tools and training, 88% of the teachers reported greater communications with their students and 76% noted an increase in their overall effectiveness as a teacher.
- 40. Interim insights from the evaluation of this project: ▪ Even as project expands to include new teachers and schools, it is important to keep the objectives and evaluation plan stable so that you can evaluate for growth and maturity alongside the success of the replication. ▪ Analyzing teacher and student data at the class/school level side by side provided valuable insights into the relationship between usage, personal valuation and impact. ▪ Monitoring is not the same as evaluation. But monitoring is important for a valid evaluation. ▪ A good evaluation demands good data. To get good data, you have to ask the right questions.
- 41. Selection of tips and strategies you should think about when evaluating your own mobile learning projects 8
- 42. Selection of tips and strategies to use when evaluating your own mobile learning projects 1. Evaluation is not magic. Write out a plan and do that at the same time you are developing your operational plan for implementation so it aligns to your objectives for your project .
- 43. Selection of tips and strategies to use when evaluating your own mobile learning projects 1. Evaluation is not magic. Write out a plan and do that at the same time you are developing your operational plan for implementation so it aligns to your objectives for your project . 2. The quality of the data will determine the validity of your evaluation – and thus, what you will learn. You should always worry about the validity of your data!
- 44. Selection of tips and strategies to use when evaluating your own mobile learning projects 1. Evaluation is not magic. Write out a plan and do that at the same time you are developing your operational plan for implementation so it aligns to your objectives for your project . 2. The quality of the data will determine the validity of your evaluation – and thus, what you will learn. You should always worry about the validity of your data! 3. Use the right tool for the right objectives in your plan. Some tools work better than others for certain objectives.
- 45. Selection of tips and strategies to use when evaluating your own mobile learning projects 1. Evaluation is not magic. Write out a plan and do that at the same time you are developing your operational plan for implementation so it aligns to your objectives for your project . 2. The quality of the data will determine the validity of your evaluation – and thus, what you will learn. You should always worry about the validity of your data! 3. Use the right tool for the right objectives in your plan. Some tools work better than others for certain objectives. 4. Evaluation is ongoing – full year and year to year
- 46. Selection of tips and strategies to use when evaluating your own mobile learning projects 5. Just because you use a mobile device and you are in education don’t assume that you know how to evaluate a mobile learning project. Evaluation is tricky business to get right!
- 47. Selection of tips and strategies to use when evaluating your own mobile learning projects 5. Just because you use a mobile device and you are in education does not mean that you automatically know how to evaluate a mobile learning project. This is tricky business to get right! 6. Keep your eyes wide open for surprises and unintended outcomes – mobile learning is not “jello” yet. This field is still developing and there is no recipe or formula for success yet.
- 48. Selection of tips and strategies to use when evaluating your own mobile learning projects 5. Just because you use a mobile device and you are in education does not mean that you automatically know how to evaluate a mobile learning project. This is tricky business to get right! 6. Keep your eyes wide open for surprises and unintended outcomes – mobile learning is not “jello” yet. This field is still developing and there is no recipe or formula for success yet. 7. Be realistic about your outcomes especially regarding the timeframes in which you expect to see those outcomes. To get to the big impact outcomes takes a lot of work in the classroom!
- 49. Selection of tips and strategies to use when evaluating your own mobile learning projects 8. Most importantly, an evaluation of a mobile learning project is about so much more than mobile devices. It is really about the transformative learning experiences at your school. Take advantage of that!
- 50. More resources available for extended learning https://www.tomorrow.org/publications/MakingLearningMobile.html ▪ 3 years of evaluation reports from the Making Learning Mobile project https://www.qualcomm.com/media/documents/files/united-states- making-learning-mobile-mlm-project.pdf ▪ Case study on the Qualcomm Wireless Reach site Additional information about the projects featured today: Making Learning Mobile – Chicago
- 51. More resources available for extended learning Additional information about the projects featured today: STEAMing Ahead with Mobile Learning – San Diego https://www.qualcomm.com/media/documents/files/united- states-steaming-ahead-with-mobile-learning.pdf Classroom of the Future – Turkey (not published yet – hard copies provided today for you!)
- 52. More resources available at www.tomorrow.org National Speak Up reports and infographics Targeted and thematic reports Digital learning trends Community engagement Mobile learning – infographic! Games in the classroom Blended learning outcomes Presentations, podcasts and webinars Services: consulting, workshops, evaluation and efficacy studies ✓ We have expertise in the evaluation of mobile implementations, new classroom models and digital content usage New Speak Up 2017 reports released thru Sept 2018
- 53. Plan now for your district to participate in Speak Up 2018 Online surveys for: o K-12 students – individual + group o Parents – English and Spanish o Teachers o Librarians/Media Specialists o School Site & District Administrators o Technology Leaders o Community Members & Communications Officers Surveys open Oct 2018 – Jan 2019 Learn more www.tomorrow.org/speakup 100% free service to all schools and districts – including reports with comparison data
- 55. Best Practices for Evaluating Mobile Learning: Real Stories from Around the Globe Dr. Julie A. Evans CEO, Project Tomorrow June 25, 2018 @SpeakUpEd @JulieEvans_PT